this post was submitted on 11 Mar 2026
106 points (97.3% liked)

Programming

26036 readers
164 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 2 years ago
MODERATORS
 

So my manager today asked me if I could stay later when there's broken things in prod, and then today his star dream employee yolo'ed a full stack change into prod without review. It's fucking massive and implements new API endpoints, touches >20 files. Many of the diffs are too large to render in the browser.

It's almost comical, but something immediately broke.

Most of my day, I'm digging through code to identify bugs created from this shit, just to get a stealth merge midday.

I kind of don't know what to do.

all 48 comments
sorted by: hot top controversial new old
[–] Wrrzag@lemmy.ml 5 points 1 day ago (1 children)

I kind of don't know what to do.

Rollback and go home

[–] Reginald_T_Biter@lemmy.world 6 points 1 day ago

I had a colleague who pushed a load of half cocked broken nonsense to a repo i was owner of. Blatant emoji comments from AI and all. I just reverted it because I was under a load of pressure to get it out and on ASAP.

He messaged me 6 months later asking me if I "deleted his code". I told him I reverted it because he pushed a load of broken bullshit to main and swanned off for 6 months.

He was like, "oh..." then i realised. This guy doesn't even know what git revert is. Had to explain it.

What the actual fuck.

[–] thingsiplay@lemmy.ml 2 points 1 day ago

I say as long as Ai exist, real programmers will always have a job and are not replaceable. YOLO Vibelords are replaceable, but only real programmers have the skill to go through the shit. That's the most secure job ever if you ask me.

[–] VibeSurgeon@piefed.social 14 points 2 days ago

Don't provide bailouts for neither your manager nor your colleague. Highlight as far up the chain as you can the damage their behavior is having on the business.

[–] slazer2au@lemmy.world 94 points 2 days ago (2 children)

I would say no to staying back. The employee that broke it needs to fix his code, at the very least roll the change back.

At the next meeting or post mortem bring up why code can be pushed to prod without review. Your git forge should be able to block merging to prod by a single person.

[–] reabsorbthelight@lemmy.world 30 points 2 days ago

Yeah. I log off at EOD and don't check messages. It's not my families fault he lets his employees vibecode shit into prod. The Vibelord is the main problem, but others do similarly stupid things.

[–] Auster@thebrainbin.org 19 points 2 days ago (1 children)

Adding to that, about the "stealth merge", couldn't it be just the rollback? Don't change what works, as the saying goes.

And expanding on the meeting or post mortem idea, take this time not to fix the code, but to make an essay showing what broke, how it broke, and how it could be avoided in the future. Also preferably not letting your tone getting acid or those hearing from the company could instinctively refuse to listen to you even if your points are solid.

[–] Reginald_T_Biter@lemmy.world 4 points 2 days ago

If only there was some way of reverting the merge commit and redeploying.

[–] Corbin@programming.dev 45 points 2 days ago

You need SRE concepts. First, if you break it then you fix it; in a system where anybody can make a change, it's the changer's responsibility to meet service objectives. Second, if your boss doesn't find that acceptable then they need to appoint a service owner and ensure that only the owner can make changes; if the owner breaks it then the owner fixes it. Third, no more than half of your time should ever be spent fixing things; if something is constantly broken then call a Code Yellow or Code Red, tell your service users that you cannot meet your service levels, and stop working on new features until the service is stable again.

Under no circumstances, ever, should anybody stay late. There should only be normal business hours, which are best-effort, and an on-call rotation which is planned two months in advance. Also, everybody on call should be paid hourly minimum wage on top of salary for their time.

[–] thedeadwalking4242@lemmy.world 34 points 2 days ago (2 children)

Do NOT under any circumstances fix it. The other guy needs to and you need to be clear he broke it and how

[–] Tharkys@lemmy.wtf 15 points 2 days ago

I came here to say exactly this. The offending dev and your manager aren't going to understand the severity of issue until he spends days trying to find the bugs. Not your circus, not your monkeys.

[–] Feyd@programming.dev 22 points 2 days ago

Roll it back and go home.

[–] traches@sh.itjust.works 31 points 2 days ago

Roll it back and go home

[–] Jesus_666@lemmy.world 23 points 2 days ago

Pushing to prod without review and breaking the running application is a resume-generating event in many companies. In many others it's not even possible because of programmatically enforced policies.

If your company's response is not to prevent or dissuade it but to have other people work overtime to fix the mess then that's a major management fail.

Try to educate your boss about best practices. This incident should give your arguments some more weight.

Deployment to prod should not be something a developer can do by themselves; a proper CI/CD system can be configured so that prod can only be deployed to by people with an appropriate role (product owners or lead devs if your company doesn't have POs).

If you don't have such a system, make it an explicit policy: Only Steve the lead dev (or someone specifically appointed by him while he's absent) can push to prod; if anyone else does it they get invited to an uncomfortable meeting with Steve. If they do it again the meeting will be with HR.

But seriously, you should lobby for a proper CI/CD system (if none is present) and for the system to be configured so that a) you can't merge to the main branch without a code review and b) deploying to prod only works from main and with explicit approval by a PO/lead dev. That should stop most of the shenanigans.

[–] owenfromcanada@lemmy.ca 20 points 2 days ago

If your manager won't hear anything negative about Vibelord, polish up your resume. There's nothing you can do to change the culture of your team if the boss isn't on board with it.

Not sure what kind of person your boss is, but if you want to try to win him over, try to figure out what might sway him. If he's technically illiterate, talk to him about best practices (like, I dunno, code reviews) and cite reliable sources and data if you think that would help. Avoid calling out Lord Vibington, the main thing is to put a picture into your boss's head of what this could/should look like. He might not realize that these issues are preventable.

The goal here is to get your boss to take more ownership of the team's culture, and start insisting on preventative measures. Mr Viberator will either have to conform, or there will be increasing friction between him and the boss.

[–] custard_swollower@lemmy.world 15 points 2 days ago* (last edited 2 days ago)

Explain to your manager that if an employee can merge code without reviews, he should expect breakages on regular basis.

Propose that your team can meet to talk about how to make the process better, so no breakages occur.

Mention personal responsibility and knowledge - it’s a good practice that a person who wrote the latest code changes either helps either is responsible for making the service working again, if there are issues - specifically because they might have some idea what’s broken.

If your manager will ignore it, I’d start looking.

[–] slacktoid@lemmy.ml 10 points 2 days ago

Ask to be paid by the hour

[–] not_woody_shaw@lemmy.world 10 points 2 days ago* (last edited 2 days ago)

Changes that big need to be broken down into smaller PR-sized steps. To do that you need an actual plan of how to get from here to there. Bro is swaggering like a senior then coding like a junior.

What to do? Well now you have evidence that AI tools are not yet good enough to write code that doesn't need human review before deploying to prod. Chat with whoever is in charge of Process, and get some team guidelines in place to prevent unreviewed code going to prod, and unreviewable PR's being submitted for review.

[–] 87Six@lemmy.zip 9 points 2 days ago

I'd just talk to the guy, clarify HE needs to stay late not me, then log off. If he's nice about it I'd consider ataying but from your description, he's anything but.

What you gotta do in situations when someone is beingan asshole is spark a meeting with the higher ups so they can see the BS you deal with and how it's none of your business that prod was messed up.

[–] kivarada@programming.dev 8 points 2 days ago* (last edited 2 days ago)

Pray that the whole system breaks sooner than later...

[–] MonkderVierte@lemmy.zip 5 points 2 days ago* (last edited 2 days ago) (2 children)

You render diffs in the browser?

And no, let the yolo fix the things he broke, or he will never learn to be a reasponsible coder.

[–] eager_eagle@lemmy.world 3 points 1 day ago (1 children)

browsers are quite capable of rendering text, even images!

[–] MonkderVierte@lemmy.zip -1 points 1 day ago* (last edited 1 day ago) (1 children)

But they're bad at it. Especially in performance and amount.

[–] eager_eagle@lemmy.world 0 points 22 hours ago* (last edited 22 hours ago) (1 children)

Not true. Browsers are definitely faster handling text than your average terminal emulator without GPU capabilities, they just use more resources.

That said, GitHub diff viewer specifically is a sluggish mess. But that's not because of the browser.

[–] MonkderVierte@lemmy.zip 1 points 20 hours ago (1 children)

What, you think the usual text editor renders all lines just because? No, they load the lines that are currently visible + some more for scrolling. At least the ones that don't choke on a mere 100k lines.

But webbrowser engines just aren't made for this, but for DOM rendering.

[–] eager_eagle@lemmy.world 1 points 20 hours ago* (last edited 20 hours ago) (1 children)

So does a browser. Modern browsers don't even need to receive the whole HTML to start rendering things.

There's a lot of engineering effort put into browsers, much more than in terminal emulators because they need to do much more than just rendering text.

[–] MonkderVierte@lemmy.zip 1 points 20 hours ago

Ok, maybe i have a slight bias, because editors in-browser are usually a system-in-system(-in-system) JS mess.

[–] jtrek@startrek.website 14 points 2 days ago

GitHub and similar tools show PR diffs in the browser.

[–] Kissaki@programming.dev 5 points 2 days ago* (last edited 2 days ago) (1 children)

if I could stay later when there’s broken things in prod

In general, or on this instance?


Do you have team retrospectives? That's where I would bring it up in my team. Raise my concerns, explore and understand what team consensus is around this topic, around risks, quality, etc.

If the team consensus and/or management consensus is YOLO - then I try to protect myself from personal investment and going beyond contractual obligations. Because I already know what will come and how it will negatively affect me personally.

It's possible a honest discussion with management about goals and risks could lead to clarified guidelines, requirements, and goals. If it doesn't, I'd probably be looking for a better job/environment. Because I'll be miserable if colleagues YOLO, no matter how careful I am personally.

[–] reabsorbthelight@lemmy.world 5 points 2 days ago (2 children)

I told my manager that we need to focus on quality. He partially agrees, but part of the problem is that he didn't even know how to get to our company git repo till I showed him. All he knows is the front end.

Unfortunately the Vibelord is his favorite employee

[–] Kissaki@programming.dev 1 points 1 day ago

I obviously don't know your environment, but I don't think a manager needs access or knowledge to git to effectively manage teams, hear and understand team member concerns, and to steer guidelines and guardrails.

[–] Skullgrid@lemmy.world 7 points 2 days ago

Unfortunately the Vibelord is his favorite employee

escalation time

[–] cecilkorik@piefed.ca 4 points 2 days ago (1 children)

When Rome is burning, do as the Romans like Nero do. Fiddle a nice tune, tell Claude to fix it for you, and go home. If they're not willing to give any fucks, why should you?

[–] 13igTyme@piefed.social 3 points 2 days ago (1 children)

Fox AI coding with AI coding. Genius. Also update your resume and start looking elsewhere.

[–] eager_eagle@lemmy.world 1 points 1 day ago

hallucinated problems require hallucinated solutions

or something

[–] ExLisper@lemmy.curiana.net 3 points 2 days ago

This is not how you do development. Like, at all. If changing jobs is an option for you do it. If not, well, not much you can do. You're not going to fix the entire company if management and other devs don't know how it should work and don't want to change.

[–] MNByChoice@midwest.social 2 points 2 days ago* (last edited 2 days ago)

Edit: I should explain first that I do not think you can change your employees or manager. A technical solution will never fix a management problem. Perhaps your manager is getting enough heat to be open to better management controls. You should be in "sanity protecting" mode.

Original: While the posts include a lot of fiction, but "overemployed" crowd have some good advice. Start applying elsewhere, downshift your effort at $job_one, and move to collecting a check.

Or ramp up and take your manager's job or get fired trying. Gather data and allies in the C-suit, and stage a coup. Or unionize (or post pro-union flies in the bathroom).

[–] mesamunefire@piefed.social 1 points 2 days ago

Make ci/cd part of the process. It stops a lot of people who push huge unmaintanable changes.

If it doesent pass the tests and cant build then it stops them from merging.