Especially server accessible only by SSH....
Programmer Humor
Welcome to Programmer Humor!
This is a place where you can post jokes, memes, humor, etc. related to programming!
For sharing awful code theres also Programming Horror.
Rules
- Keep content in english
- No advertisements
- Posts must be related to programming or programmer topics
I can't be bothered to walk down to the basement, so practically my server is also only accessible by SSH
Especially after age 40 and a knee surgery... I'm tired boss! 😩
I'm 150+km away from my server, with literally everything on it lol
I'm at college right now, which is a 3 hour drive away from my home, where a server of mine is. I just have to ask my parents to turn it back on when the power goes out or it gets borked. I access it solely through RustDesk and Cloudflare Tunnels SSH (it's actually pretty cool, they have a web interface for it).
I have no car, so there's really no way to access it in case something catastrophic happens. I have to rely on hopes, prayers, and the power of a probably outdated Pop!_OS install. Totally doesn't stress me out I'll just say I like to live on the edge :^)
Setup a pikvm as ipmi and you'll have at least another layer of failure required to completely lose connectivity
Currently the server(s) are in my room, which is so messy my dad probably wouldn't even enter it voluntarily. And in the case grub/fstab/crypttab/etc. are messed up, which is probably the most common error, he probably couldn't solve it by himself. Soon everything's gonna live in its own little room in the basement, so it's gonna be accessible easier actually.
Initializing VPC...
Configuring VPC...
Constructing VPC...
Planning VPC...
VPC Configuration...
Step (31/12)...
Spooling up VPC...
VPC Configuration Finished...
Beginning Declaration of VPC...
Declaring Configuration of VPC...
Submitting Paperwork for VPC Registration with IANA...
Redefining Port 22 for official use as our private VPC...
Recompiling OpenSSH to use Port 125...
Resetting all open SSH connections...
Your VPC declaration has been configured!
Initializing Declared VPC...
Never update, never reboot. Clearly the safest method. Tried and true.
Found the debian user!
Never touch a running system
Until you have a inviting hole in your system
Nevertheless, I'm panicking every time I update my sever infrastructure...
When you make a potentially system breaking change and forgot to make a snapshot of the VM beforehand...
There's always backups... Right?
.... Right?
oh there is. from 3 years ago, and some
Someone set up a script to automatically create daily backups to tape. Unfortunately, it's still the first tape that was put in there 3.5 years ago, every backup since that one filled up failed. It might as well have failed silently because everyone who received the email with the error message filtered them to a folder they generally ignored.
Just had to restart our main MySQL instance today. Had to do it at 6am since that’s the lowest traffic point, and boy howdy this resonates.
2 solid minutes of the stack throwing 500 errors until the db was back up.
If you have the bandwidth... it is absolutely worth it to invest in a maintenance mode for your system, just check some flat file on disk for a flag before loading up a router or anything and then, if it's engaged, just send back a static html file with ye olde "under construction" picture.
Bonus points if your static site sends a 503 with a retry after header.
I have more than once typed shutdown instead of reboot when working on a remote machine... always fun
Make an alias for Ehen you type shutdown it dies restart and if you want to shutdown make an alias that goes like
Yesireallywanttoshutdown
Networking, we had a remote office in Europe (I'm in the US) and wanted to reset a phone. Phone was on port 10 of the Cisco switch, port 1 went to the firewall (not my design, already in place).
Helping my coworker, I tell her to shut port 10.
Shut port 1, enter.
Ok... office is offline and on another continent...
Tbh there is nothing more taxing on my mental health than doing maintenance on our production servers.
when it was the wrong server and you're hoping it comes back up before 5 minutes and nagios starts sending alerts
I install molly-guard on important machines for this reason. So fast to do a reboot on the wrong ssh session
If a tree falls in the woods...
I work with IBM i/AS400 servers and those are not exactly the quickest thing to "reboot" (technically an IPL). Especially the old ones. I have access to the HMC/console but even this sometimes takes several minutes (if not dozens) just to show what's going on.
It's always a bit stressful to see the codes passing one after the other and then it stops on one and seems to get stuck there for a while before continuing the IPL process. Maybe it's applying PTFs (updates) or something, and you just have to wait while even the console is blank.
I've been monitoring those servers for years and I'm still sometimes wondering if it hanged during the IPL or if it's just doing its thing, because this part, even with codes, is not very verbose.
Fortunately it's also very stable so it pretty much always comes back a few minutes after you start wondering why the hell it's taking so long.
Y'all need high availability in your lives.
ERROR Insert Coin
That's why you connect an arduino to the motherboards reset pin and load it with a program where it resets the system if it doesn't receive an ACK signal over the usb connection every 10 minutes.
Eventually though the networking and apache stops working after around 150 days so you also have to make a script that resets the system after 30 minutes of not having network.