Asklemmy

Several years ago, when I was more just the unofficial office geek, our email was acting up. Though we had Internet access as normal. At the time, email (Exchange) was hosted on-prem on our server. Anything server related, I'd contact our MSP to handle it. Which usually meant they'd simply reboot the server. Easy enough, but I was kinda afraid and hesitant to touch the server unless the MSP explicitly asked/told me to do something.

I reported it to our MSP, expecting a quick response, but nothing. Not even acknowledgment of the issue. This was already going on for like an hour, so I decided to take matters into my own hands. I went to the server, turned on the monitor...and it was black. Well, shit. Couldn't even do a proper shutdown. So I emailed again, waited a bit, and again no response.

Well, if the server was being unresponsive, I figured a hard shutdown and reboot would be fine. I knew that's what the MSP would (ask me to) do. What difference was them telling me to do it versus just me doing it on my own? I was going to fix email! I was going to be the hero! So I did it.

Server booted up, but after getting past the BIOS and other checks...it went back to black screen again. No Windows login. That's not so terrible, since that was the status quo. Except now, people were also saying Internet all of a sudden stopped working. Oh shit.

Little did I know that the sever was acting as our DNS. So I essentially took down everything: email, Internet, even some server access (network drives, DBs). I was in a cold sweat now since we were pretty much dead in the water. I of course reached out AGAIN to the MSP, but AGAIN nothing. Wtf...

So I told my co-workers and bosses, expecting to get in some trouble for making things worse. Surprisingly, no one cared. A couple people decided to go home and work. Some people took super long lunches or chitchatted. Our receptionist was playing games on her computer. Our CEO had his feet up on his desk and was scrolling Facebook on his phone. Another C-suite decided to call it an early day.

Eventually, at basically the end of the day, the MSP reached out. They sent some remote commands to the server and it all started working again. Apparently, they were dealing with an actual catastrophe elsewhere: one of their clients' offices had burned down so they were focused on BCDR over there all day.

So yeah, I took down our server for half a day. And no one cared, except me.