- two wans
- lots of vlans
- 32vcpu hypervisor with gpu passthrough
- 25gbps san running off the nas
Everything I do is mostly with VMs, the hypervisor is diskless pxe booting off the SAN. I've been cleaning up my lab, it used to be a plank of wood https://discuss.online/post/44690398
There is a nanokvm hanging off the hypervisor (rs232,atx header, hid) which makes remote rebuilds quite painless. I love my inexpensive nanokvm.
What does your homelab look like?
Which hypervisor did you use that can pxe boot? Neat idea
The hypervisor is just nixos running kvm and incus - it doesn't have any opinion about pxe booting; So pxe gets dhcp from the san, which supplies the bootstrap, which is a tpm wrapped luks volume, which supplies the iscsi target, which bootstraps nixos over iscsi.... Once the nixos kernel loads anything goes, so KVM isn't aware of the iscsi nature of the diskless system at all.
I did play with nixos -> xen, but nested virtualization wasn't a thing with xen yet, so i went back to KVM. incus over libvirt so i can have a proper hypervisor API, so i can have projects with different credentials and scopes creating and destroying VMs willy nilly.
How are you finding incus?
I'm intending to move from proxmox to incus, but as it has Home Assistant on, it feels like we'll be going back to the stoneage whilst it's down and hardware costs are too high to get a 2nd box at the moment.
I'm very happy with it; it works well, has good control of different projects, allows limited api access to different vlans; supports clustering and live migration. I haven't found anything I can't do with incus that I can do with libvirt. it is surprisingly mature.
I do have a second machine, so I have juggled VMs around when i played with different hypervisors. You can always take a disk image of your home assistant and restore it in incus.