Programming

Do you know about the MAILTO= lines in crontabs? That's exactly what they're there for. (And your script needs to output error messages on failure, of course.) You'll need a local mail forwarder like ssmtp or exa, though.

If it were critical I would set up an "email me on script run failure" wrap around the individual cron jobs. Simple exit code check, potentially with console output as email body is simple and useful.

Put the job in a script that only outputs text on error, and use MAILTO, like many have said. Due to spam filters that might not be an option though.

Next option is ntfy or gotify. make sure the script exits with nonzero status on error, and append || curl -XPOST -d 'cronjob failed' 'https://ntfy.sh/your-random-topic' to the crontab line. Naturally you'll likely want the ntfy/gotify mobile app to receive the notodication. On iOS this might suck, as at least for me the ntfy app didn't update in the background at all.

Next one is an push uptime monitor, self-hosted or managed. I like Uptime Kuma. Make sure the script only exits with 0 status on success, and append && curl 'https://your-monitor-endpoint/identifier to the crontab line. Or use an inverted monitor and use || instead of &&.

The uptime monitor will then notify you if the script hasn't succeeded in the defined interval.

With some combination of these you can be pretty confident you'll notice a failure, but still better to check every once in a while.

I also have a lot of jobs around that notify for each run, despite success or error, but it's a bit too easy to just glance the message and accidentally ignore an error.

I think the real obvious question is, why aren't you monitoring your disk space?

Monitor your logs with a network monitoring tool like Nagios, Icinga, etc. You can set filters for alerts you are genuinely interested in, and email the alerts to your ticketing system or sysads.

Just as in life you shouldn’t say “can’t”, in programming, you shouldn’t say, “started failing silently”.

~~If you made something that fails silently, take a moment to slap yourself silently.~~

Plus one dir a simple alert wrapper. Although for me it's even more simple: the only cron stuff I have is backup and individual service updates and both have an entry on my Homepage Dashboard with a health indicator. Red dot == shit.

The only fancy custom thing I've built is not even cron based: sometimes I missed on my nixos server that I needed a reboot because kernel modules updated as well - so my nushell prompt compares the live environment with the loaded kernel - if versions mismatch my shell (rightfully) flames me to pay more attention and that I need to reboot.

For anything from based I'd use a wrapper - for me it would be a simple curl to the localhost matrix bridge so that I'd get a push message but for others it's Mail or just a ntfy - whatever is easiest for your server to reach you, really. I would not want to check on my automations and the notification service id notice within two hours if it's down because I use it for a lot of other items as well (no mails, appointments, reminders or events in 2 hours??)

I really like ntfy or gotify for stuff like þis. Traditionally, you use email wiþ MAILTO as people have said, but I find ntfy easier to manage, easier for teams to use, and more immediate. Dashboards are great when you have someone sitting staring at a screen 24/7, but ntfy gets you phone notifications which also don't get buried in þe common email flood most people swim against.

If you've got a good amount https://github.com/healthchecks/healthchecks

Everyones' suggestion to use MAILTO is great. Wrappers are another option. One more is just schedule another cron that checks to make sure the first one did what it is supposed to do and notify you if it broke.