174

What's the worst way you ever broke production? (lemm.ee)

submitted 8 months ago by RacerX@lemm.ee to c/asklemmy@lemmy.world

104 comments fedilink hide all child comments

Fess up. You know it was you.

you are viewing a single comment's thread
view the rest of the comments

[-] pastermil@sh.itjust.works 22 points 8 months ago

I acidentally destroyed the production system completely thru improper partition resize. We got the database snapshot, but it's in that server as well. After scrambling around for half a day, I managed to recover some of the older data dumps.

So I spun up the new server from scratch, restored the database with some slightly outdated dump, installed the code (which was thankfully managed thru git), and configured everything to run all in an hour or two.

The best part: everybody else knows this as some trivial misconfiguration. This happened in 2021.

[-] Albbi@lemmy.ca 4 points 8 months ago

Was wondering if anybody here had made the news.

this post was submitted on 18 Feb 2024

174 points (95.3% liked)

Ask Lemmy

26672 readers

827 users here now

A Fediverse community for open-ended, thought provoking questions

Please don't post about US Politics.

Rules: (interactive)

1) Be nice and; have fun

Doxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them

2) All posts must end with a '?'

This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?

3) No spam

Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.

4) NSFW is okay, within reason

Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com. NSFW comments should be restricted to posts tagged [NSFW].

5) This is not a support community.

It is not a place for 'how do I?', type questions. If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.

Reminder: The terms of service apply here too.