Prometheus, Loki and Grafana.

And so so many Prometheus metric exporters.

Observability is such an endless rabbit hole, it’s so easy for me to spend huge amounts of time accomplishing not that much lol. But very enjoyable and cool to see it all come together.

My pro tips: using Kubernetes actually makes this stuff a heck of a lot easier to set up thanks to the common patterns that k8s has - lots of turnkey helm charts out there that make it all so easy and are powerful. Another tip would be to use Prometheus service discovery if you can. Also, Loki/Promtail is actually quite easy to set up - but using LogQL queries can be very tricky. Just be warned, observability is a full time hobby in itself lol

I'll keep it very simple: I don't.

If I'm trying to do something and I notice an issue, then I'll investigate it. But if it's not affecting anything, is it really a problem?

I’ve used monit for maybe 2 decades now. Works great and simple to use.

https://mmonit.com/monit/

https://github.com/awesome-foss/awesome-sysadmin#monitoring

I use netdata (agent only, not the cloud/SaaS stuff)

I use zabbix to monitor everything, agent on each device uses around 30 mb of memory and with the Linux templates it can monitor just about everything on the server.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

4 acronyms in this thread; the most compressed thread commented on today has 14 acronyms.

[Thread #11 for this sub, first seen 19th Jul 2023, 17:40] [FAQ] [Full list] [Contact] [Source code]

I set up custom bash scripts collecting information (df, docker json, smartCTL etc) Either parse existing json info or assemble json strings and push it to Homeassistant REST api (cron) In Homeassistant data is turned into sensors and displayed. HA sends messages of sensors fail.
Info served in HA:

• HDD/SSD (size, smartCTL errors, spin up/down, temperature etc)
• Availability/health of docker services
• CPU usage/RAM/temperature
• Network interface/throughput/speed/connections
• fail2ban jails

Trying to keep my servers as barebones as possible. Additional services/apps put strain on CPU/RAM etc. Found out most of data necessary for monitoring is either available (docker json, smartCTL json) or can be easily caught, e.g.

df -Pht ext4 | tail -n +2 | awk '{ print $1}

It was fun learning and defining what must be monitored or not, and building a custom interface in HA.

I use netdata, it's very good at digesting thousands of metrics to sharing actionable. The cloud portion is proprietary, but you can toggle off the data collection. I did turn on the cloud portion though, I get email notifications when something breaks. Might sound counter to the self hosted mantra, but a self hosted monitoring system isn't very helpful when your own systems go down.

Monit for simple stuff and daemon restart on failure. LibreNMS for SNMP polling, graphing, logging, & alerting.

Might be a bit more complex than what you want, but I love Prometheus + Alertmanager and a nice sexy Grafana dashboard

Nagios. Core, but i've worked with it for years and am kinda masochistic. (Currently tying it into an IDRAC6)

I used monitorix a long time ago. now netdata.

Have used both Zabbix and Prometheus for this. Highly recommend Prometheus over Zabbix if you can afford the time to learn it. It’s a bit weird at first but it’s so much easier to extend and manage then Zabbix in my experience.

You will need to set up Grafana to go along with Prometheus. But, again, it’s so flexible that you will end up being happier with it.

Regarding your edit: people are answering the question you posed in your post title, not necessarily giving you advice about how you should do it.

I'm ever so slowly teaching myself Zabbix, need something full-featured because I also need monitoring for my hosting clients etc