Ask Lemmy

I worked at a bank at the time. We were moving to a new system and running recons against the old system to check the behaviour was the same. I had to run a manual recon of the old system vs the new 4 times per day. There was a lot of focus from management and users on the new system.

The week leading up to Christmas, I was the one person not on holiday yet, and also the most junior person on the project. I found that week so stressful, as I had to run these recons and quickly decide whether each break was real or not before reporting to the users. Despite having worked on that system, I had very little confidence and didn't have the same intuitive mental model of the system my colleagues had. I had to dig into each break case-by-case, but they seemed much more able to understand what was going on via a few simple queries.

Anyway, I get through the week and left for the holidays on Thursday evening. I'm just grateful that I've gotten through it. Then, around 3pm on the Friday, I see a missed call from the tech lead. I log in, and everything's on fire. I join the incident call, and it turns out that we hadn't processed a single trade in the new system that whole week. I discover that it was thanks to a config change I'd made several weeks before, that had just made it to production. No-one (neither the users, nor I) had realised! But we missed several hundred million pounds worth of payments in that week as a result.

It was so jarring, having been relieved that I made it to the holiday, then joining the incident call and struggling to work out what to do. I completely dissociated and my mind was blank. I remember being on the call and really passively and calmly walking around my room. I kept thinking "I need to do x, I need to do y" but my mind couldn't focus and I was just staring at the screen. At some point I just lay in bed with my laptop while on the call.

There had been a total failure of process: my change had been approved by two people, the nonprod environment was configured differently in a way that didn't expose the bug, the recon failures looked very similar to the false positives, and there were so many false positives that it was impossible to dig into all of them. Meanwhile, we didn't have basic queries monitoring that trades were flowing in, and the users weren't paying much attention either, until they realised that it was broken.

Still, I made a lot of mistakes. I should have just escalated that there were breaks instead of trying to figure it out myself. I shouldn't have been afraid to call the tech lead and bring them out of their holiday. And I shouldn't have been afraid of the confrontation with the users.

Anyway, that experience really messed me up mentally for a long time. I lost so much confidence and became so much more scared of production (not in a healthy way). It really was not the right environment for me.