Ive setup that proxmox cluster, down to the sff and and the 2.5gb nics. Ceph works decently at 2.5gb, but there are some foibles. Anything running sqlite will need to be on local disc instead of the ceph data store to sort out write errors. It wont be in HA by default, but you can still power it off and migrate it to the ceph store, and the move it if you need to do long term maintance.

I also have GPU passthrough setup on my jellyfin container, so that isnt in HA either. Still, worth the performance bump, even with an sff gpu.

[–] JasonDJ@lemmy.zip 2 points 5 days ago (1 children)

Ooh good call on jellyfin I'll have to keep that in mind. I hadn't considered that jellyfin would probably want direct access to the igpu for QSV or any other re- encoding. All my nodes are specced the same, hardware wise...I wonder if I'll be able to just pass down the igpu to the worker-node vms and have that available to all of them. I don't need their displays, and all my SFFs have onboard serial in case I actually need to see what's on console.

[–] rainwall@piefed.social 2 points 5 days ago* (last edited 5 days ago)

Maybe? I'm working with a hodgepodge of sffs, so each box is different. I wouldnt expect it to work with direct PCI passthrough even with your setup, but maybe at a different abstraction layer?

If you are going full hyperconverged cluster, look at implementing proxlb too for dynamic load balancing. Its my next project on the cluster.