I'd highly recommend Talos for this if you haven't already decided. It's entirely API driven and immutable. It can be bootstrapped from PXE and truly feels like a local cloud environment.
Selfhosted
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
-
No low-effort posts. This is subjective and will largely be determined by the community member reports.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
Talos is amazing and if you want to start from a fully automated setup (GitOps, Renovate), I highly recommend using https://github.com/onedr0p/cluster-template
This is the way. They have a image generator too
I've been setting up a baremetal Talos-based cluster on 4 SFF machines and using Flux to declaratively deploy things to the cluster. Once you wrap your head around how Flux operates, deploying things becomes fairly straightforward.
For me, the most difficult process was deploying the cluster properly, but the recent-ish video posted on the Tailscale YT channel helped a LOT.
I run k3s on my bare metal cluster and for a lab I think it is excellent. Gets the cluster up and running easily so you can learn about actual Kubernetes stuff.
Personally I run it on fedora coreos, but I would not recommend that for everyone. I would just use whatever distri you like. All the usual suspects will work perfectly.
In the real world, you're more likely to see multiple vms on each server and each vm running a k8s node. So I'd go with that setup.
Once you have that going, play with clustering services within k8s: Kafka, redis, etcd, postgres.
I think I'd install proxmox on all machines because there is a proxmox provider for Terraform. Then, manually create the VMs, and to learn the barebones, use kubeadm to set everything up, and kubectl to manage it. Once comfortable and knowledgeable with that, start messing around with Terraform and Ansible.
PXE is unnecessary unless you're going to be creating a reusable boot image. Just faster to use LiveUSB.
What did you getaid off, and what are you trying to apply to? Maybe help to understand on what you're trying to learn.
Just for your own sanity, just install Talos on the 3 machines, understand how to join them to a cluster, then deploy some stuff around the cluster. Get a feel for the basics before you get into the mess of trying to do it all in VMs.
I'd also check some comparisons on the various flavors of different lube stacks: k3s, microk8s, kubedge...etc. Theres so many now it's hard to track.