Ask Lemmy
A Fediverse community for open-ended, thought provoking questions
Rules: (interactive)
1) Be nice and; have fun
Doxxing, trolling, sealioning, racism, toxicity and dog-whistling are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them
2) All posts must end with a '?'
This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?
3) No spam
Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.
4) NSFW is okay, within reason
Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com.
NSFW comments should be restricted to posts tagged [NSFW].
5) This is not a support community.
It is not a place for 'how do I?', type questions.
If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.
6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !uspolitics@lemmy.world
7) No Hit-and-Run questions.
Please don't delete your post for no apparent reason. If you plan on deleting a question later, say so in the post, or if you feel that you have a good reason to remove it, message a mod beforehand. It's not fair to the ones who took their time to answer, and it's not in the spirit of the community.
8) No Bots.
Posts or comments from bots, LLM's, AIs, Neural Networks, Transformers, or Marvin the Paranoid Android are not welcome in AskLemmy. Real humans only please.
Reminder: The terms of service apply here too.
Partnered Communities:
Logo design credit goes to: tubbadu
view the rest of the comments
I’m a system administrator for a relatively big SaaS for a relatively big company.
I was attempting to configure a new inbound email protocol in the TEST environment. I needed to reference how it was configured in PROD so I could copy certain settings into TEST, so I had both up on my screens (you can see where this is going). But first I needed to clear out the old cloned configurations in TEST.
…And not even half a second after I deleted those config files did I realize what I’d done: deleted inbound email configuration on PRODUCTION. In the middle of the day. Killed inbound email for the system for the whole company.
The whole reason I had PROD open was because I didn’t have good documentation of how it was configured (what was cloned into TEST was old and not correct). So now I was seriously fucked. I had to redo the config from scratch live in PROD. Even tried backing out the deletion but that failed.
Had to advise my boss of the fuckup. He was new. Had only been with the company for like 2 weeks. I had to assure him that I am NOT actually a total fuckhead and I usually don’t do stupid shit like this. I felt terrible, because I knew he’d have to give the post-failure analysis review on this to his new bosses, and that would definitely be awkward.
Anyway, he was cool with it. Assured me that he’d done something similar at a previous job. He let me do my thing to fix it, which I did. Inbound email was only down for …well, a long time (3-4 hours), because once I was able to get the config working, I had to re-register all of the inbound email addresses to use it -which had its own sordid story. That took some time.
After the fact, we modified the TEST environment to have a different “theme” (header colors) so it wasn’t identical to PROD on-screen, and I documented the email configuration thoroughly.
I worked in software ops where part of my day was handling incidents and bring anxiety to the room when I came running to a dev team with an incident caused by their recent release. A few times it was caused by someone accidentally running a test scenario in prod. At one point I got enough and said "add a bright orange 'test' banner on your UI for dev environments." Apparently that was a foreign concept to absolutely everybody, and the more I look around the less I see people actually trying to make an effort to distinguish the two to prevent such catastrophic failures. Especially when you're handling tens of gigawatts of energy on a daily basis.