198
submitted 10 months ago by Dio9sys@lemmy.blahaj.zone to c/asklemmy@lemmy.ml
(page 2) 50 comments
sorted by: hot top controversial new old
[-] dan@upvote.au 16 points 10 months ago* (last edited 10 months ago)

I broke the home page of a big tech (FAANG) company.

I added a call to an API created by another team. I did an initial test with 2% of production traffic + 50% of employee traffic, and it worked fine. After a day or two, I rolled out to 100% of users, and it broke the home page. It was broken for around 3 minutes until the deployment oncall found the killswitch I put in the code and turned it off. They noticed the issue quicker than I did.

What I didn't realise was that only some of the methods of this class had Memcache caching. The method I was calling did not. It turns out it was running a database query on a DB with a single shard and only 4 replicas, that wasn't designed for production traffic. As soon as my code rolled out to 100% of users. the DBs immediately fell over from tens of thousands of simultaneous connections.

Always use feature flags for risky work! It would have been broken for a lot longer if I didn't add one and they had to re-deploy the site. The site was continuously pushed all day, but building and deploying could take 45+ mins.

[-] jjjalljs@ttrpg.network 14 points 10 months ago

Always use feature flags for risky work! It would have been broken for a lot longer if I didn’t add one and they had to re-deploy the site. The site was continuously pushed all day, but building and deploying could take 45+ mins

This reminds me of the old saying: everyone has a test environment. Some people are lucky enough to have a separate production environment, too.

load more comments (8 replies)
[-] ace_garp@lemmy.world 16 points 10 months ago

Setting up a GNU Mailman mailing-list with the 'reply-to-all' setting still set to on (as per the defaults).

It was intended to be a distribution list.

The list had 8,000 recipients.

.

I found out what an ohnosecond is, as the blood chilled a little in my veins.

Turned out not too bad, 6 innocuous replies were sent out to everyone. It was fully open for about 25 minutes before I set 'reply-to-all' to off. Could have been much worse.

Will never ever forget to do that with a mailing-list again.

load more comments (1 replies)
[-] ExLisper@linux.community 14 points 10 months ago* (last edited 10 months ago)

I once pushed a git commit with youtube link as the commit message. Nothing terrible, some completely random video. Still, it looked really weird in the commit history. Turns out you can edit this if you have access to the server and I did have access to the server.

One time in the same company I found a random youtube link in the middle of a java class. Yes, it was still compiling. No I didn't commit it.

[-] Rolive@discuss.tchncs.de 9 points 10 months ago

What's wrong with that? I'd put a rickroll in there without regrets.

load more comments (2 replies)
[-] tetris11@lemmy.ml 6 points 10 months ago

Urgh. I sadly do this all the time

Interactive rebase, amend the commit message for your commit, continue the rebase, and force push.

Thank heavens for Magit which simplifies this process.

load more comments (1 replies)
[-] Brkdncr@sh.itjust.works 13 points 10 months ago

You mean like when Janice shat herself at work?

load more comments (2 replies)
[-] shani66@lemmy.comfysnug.space 11 points 10 months ago

Now that i think about my first job was fucking wild.

My buddy was in a forklift taking some stock down and i was spotting, basically just hanging out and making sure no one got in the way. A few minutes after the normal time it'd take he thinks something is wrong and calls me to take a look (from afar) to see how fucked we are; the answer was very, the pallet was barely holding together at all, but i couldn't see a damn thing from my position. Before i could get back to spotting we heard a loud crack and the world went still, i imagine for much longer by him, and not a second later we had hundreds of pounds of foul smelling mulch everywhere.

I had a lot more there too; babysitting an old man that looked on the verge of death with no management anywhere to be found, moving hundreds of pounds at a time by hand, dealing with the best conspiracy theorist ever.

I've been bored everywhere else I've ever worked.

load more comments (4 replies)
[-] sour@kbin.social 9 points 10 months ago* (last edited 10 months ago)

no work related but am overfill sink with water changer because forgot to remove drain cover

is flood

am get in trouble also ._.

load more comments
view more: ‹ prev next ›
this post was submitted on 14 Dec 2023
198 points (98.1% liked)

Asklemmy

43750 readers
1142 users here now

A loosely moderated place to ask open-ended questions

Search asklemmy 🔍

If your post meets the following criteria, it's welcome here!

  1. Open-ended question
  2. Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
  3. Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
  4. Not ad nauseam inducing: please make sure it is a question that would be new to most members
  5. An actual topic of discussion

Looking for support?

Looking for a community?

~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~

founded 5 years ago
MODERATORS