Sysadmin

I've worked at a company where the entire billing system ran on a Windows Server 2003 machine, running in a Vmware Workstation Player 15 VM on a Windows 8 PC.

The billing software wasn't actually billing software, it was a kind of build your own software toolkit heavily customized. Like, a sort of simple scripting engine. The actual program was no longer available and would only run on older versions of Windows even if it was. There was no installer available for it. If any part of this setup failed they would be unable to process invoices. The data format was totally proprietary with no way to export it to any other platform.

The whole setup was accessed by remote workers using an unprotected RDP connection. No VPN.

Technical Debt:

1. Current company is running one of its main databases on Oracle 13 because they refuse to pay Uncle Larry a dime, but also cannot migrate it (Oracle eBusiness Suite EBS).

2. Last company made copies of all customer servers hard drives and kept them in a cold vault. So, they had to maintain systems dating back to the 1980s to current in order to recover data from drives in cold storage. The local data recovery company loved them too as drives freeze up and don't spin after a while. They went under during Covid citing operational issues.

Was fun having a sun 2 pizzabox, a Sun Ultrasparc 10, an SGI Orion and a series of x86 systems on my desk all the time though.

3. A power company whose name I cannot share insisted on having 5 access and control rooms per power plant (most usually have 2-3 per plant) for redundancy. It drove them to near bankruptcy in the 2010s because of their tech debt trying to keep them all in sync world wide. Coolest part: I setup 20 x 70" Plasma TVs as a single C&C panel for their HQ back in 2007 using 3 Matrox video cards.

Holy.... Are you me? Haha

Solo IT, very outdated, uh, everything. Spent the last few years updating, upgrading, modernizing, etc. We've come a long way! I can conformably take a vacation without sweating out having my stuff at arm's length.

Message me if you want to share more, bc my list is far too long.

I got good news and bad news for you: Welcome to the rest of your career.

Thankfully my company has made a huge push into updating the old stuff over the past 10 years or so. It’s got a long way to go, as there is debt in many areas, but what we have addressed so far is infinitely better in function, user experience and satisfaction, and reduced downtime. It does come with its own financial costs, but sooner or later there’s going to be no one around that knows that tech, and an ever shrinking hardware pool.

Running something on a cobbled together infrastructure can work for a while, but usually when it fails, it does so catastrophically, and there is little recourse but to immediately spend large sums for emergency parts and fixes, rather than spread that cost over time as just standard maintenance expenses.

It’s like driving your car until the engine blows up because you were mad about how much oil changes cost. Sure, you saved $50 per change, but then you blew it all on a $10k engine, so what did you actually save? Nothing! And you probably paid more than if you had just lifecycled it in the first place. It’s amazingly short-sighted.

People can work around a horrifying amount of mess for a dizzying amount of time before it all comes crumbling down due the wrong thing occurring at the right time.

All of these examples are from finance companies, mostly banks. Not all my stories, these include stuff from friends in the field.

I know a place that had no documentation on access revocation for >30 third party systems.

Another with no Identity and Access Management policy until the pandemic. Service accounts with god level access? Go ahead and set an 8 character password with no expiration date, and never change it after 20+ employees who know it leave.

One place with software that sits installed on computers within reach of the public where every client copy includes a password decryption function in a file that you can copy out of the client install and then just call it from whatever program you write. Yeah, you still need read access to the user database's password field, but this was software that employees used to interact with bank accounts. With trivially reversible decryption.

That last software was slated to retire over a decade ago, and last I heard was being kept alive by the finance company paying for source code access and maintaining their own edited version themselves. The last time my friend talked about it a year or two ago, the software was just shedding its reliance on Internet Explorer and shifting to Edge.

Some federal processes and laws still require fax communications for various financial shit behind the scenes.

Do what you can to steer out and away, keep your hands off it/don't perpetuate it, have a threshold for "fuck it, not my problem to fix", have another threshold for "fuck it, let it burn or they won't learn", have a third for "fuck it, I'm running before this eats me", and always always always cover your ass. In writing, hard copy somewhere you control and work doesn't.

Ultimately, remember that companies don't reward heroics. Unless you can quantify your improvements in manager-speak, it won't even register to them. They don't give awards out for burning yourself alive to keep the engines running for another day. They give out penalties when your changes result in temporary setbacks during adjustments to the new normal.

There are many, many, many people in management and elsewhere that do not learn until they've been bit in the ass (if they are capable of learning at all). If you eliminate the friction before they feel it, they won't know you've done anything at all. You want to look good, that's how you move up. Let some things fall. Let some things break, especially when you know the fix is relatively easy and no one wants to take responsibility to ok it before SHTF.

A ton of this job is managing people, at least as much as it is managing complex systems. Not to be sociopathic, never forget the people are people, but start looking at corporate interactions and politics like you might look at a complicated system with no or little documentation.

Indefinitely? No. For 30 years? Yes. So shut up and just duplicate the files on those diskettes if you really need them to be readable.

Yes they can and yes they will. Keep good documentation to cover your own ass. If it's a small business and you're IT, you're probably the only one there. If you make a good enough case for making everyone's life a lot easier, you may have a glimmer of hope to modernize. Otherwise, godspeed on the job hunt.

They cannot limp along indefinitely, but they can do so for decades. That's exactly the problem. It's a common 'turkey problem'. In my experience, whether the company can get out of the swamp depends mainly on two things: whether bosses/decision makers understand anything at all about technology, and whether they are willing (and, hierarchically speaking, able) to have their skin in the game for big, important changes.

I've got really bad news for you, and while it may not be indefinitely....

They finally added the last bit of data to this Session store that broke the whole application. 16MB of data being read/written from store on every http request. 50% of all http request processing was handling the Session middleware.

I hate developers who don't spend the very minimum to understand the environment they work in.

interesting you haven't said anything specific. are you by any chance just a kid that has been told for the first time in their life not to fix what is not broken and is now severely confused about it?

Prepare three envelopes.

Yes. Current dingo CTO at work was grandfathered into the role because he was there long enough for the previous CTO to retire. We are very small. Now, this maroon gleefully incites using Micro$oft's AI as if it was actual mind blowing tech but consistently uses it like a search engine. Oh, and they haven't bothered locking it down. They "trust" Microsoft because "all of our data is there" and "Microsoft is certified to meet compliance" so we don't want to worry about it. lol

They believe that because on paper, because they are tacked on with "security officer" to their name, that somehow they can just wing cyber security and magically poof risk out of existence even when they are not performing any such ISMS functions day to day. They nit pick what they believe will affect the org but there are so may gaps and they just can't justify hiring someone to do the cyber stuff full time. We have no DLP but claim to be CMMC compliant. LOL

Hey man, as long as the paychecks keep coming and they're not the type to sue...