this post was submitted on 05 Sep 2024
919 points (99.0% liked)

Programmer Humor

25009 readers
1249 users here now

Welcome to Programmer Humor!

This is a place where you can post jokes, memes, humor, etc. related to programming!

For sharing awful code theres also Programming Horror.

Rules

founded 2 years ago
MODERATORS
 
you are viewing a single comment's thread
view the rest of the comments
[–] nick@midwest.social 42 points 10 months ago (2 children)

Just had to restart our main MySQL instance today. Had to do it at 6am since that’s the lowest traffic point, and boy howdy this resonates.

2 solid minutes of the stack throwing 500 errors until the db was back up.

[–] xmunk@sh.itjust.works 20 points 10 months ago (2 children)

If you have the bandwidth... it is absolutely worth it to invest in a maintenance mode for your system, just check some flat file on disk for a flag before loading up a router or anything and then, if it's engaged, just send back a static html file with ye olde "under construction" picture.

[–] dondelelcaro@lemmy.world 14 points 10 months ago

Bonus points if your static site sends a 503 with a retry after header.

[–] nick@midwest.social 4 points 10 months ago* (last edited 10 months ago)

That’s not really… possible at this point. We have thousands of customers (some very large ones, like A——n and G—-e and Wal___t) with tens or hundreds of millions of users, and even at lowest traffic periods do 60k+ queries per second.

This is the same MySQL instance I wrote about a while ago that hit the 16TiB table size limit (due to ext4 file system limitations) and caused a massive outage; worst I’ve been involved in during my 26 year career.

Every day I am shocked at our scale, considering my company is only like 90 engineers.

[–] MystikIncarnate@lemmy.ca 2 points 10 months ago

Is that the same database my user couldn't connect to today?