Programmer Humor

is it pronounced beaucoup?

Ooh sparkling archive actually sounds really fancy, I’ll start using that

Verifying the files are there in your backup is only, like 10% of verifying that it's a real, usable backup.

The important question is: can you successfully restore those files from the backup? Can you successfully put them back where they're supposed to be after losing your primary copy?

Backup fire drill tomorrow!

Bold of you to assume people/companies test backups more than once.

Case in point: I once got instructed to "enable EBS snapshots" for customer deployments to meet a new backup requirement. Disaster recovery was a completely different feature we only kind of got to a couple years later and afaik, remains manual to this day.

In the spirit of this thread: no.

Recovering with the backup should put you back to an operational state equivalent to when the backup was taken.

I.e. if you've restored some files, but something is still not working then the backup failed its purpose.

E.g. the timestamps on the files might be important, do they need to be stamped with the time of the backup or the time of the restore?

The next day, the novice's disk crashed. Three days later, the novice was still reinstalling software.

I laugh in NixOS

Adding another link, perhaps not equivalent:

https://freefilesync.org/

I use it to backup media. If the backup roots are your drives, restoring is as easy as copy-paste, merge folders and wait.

Edit: actually, you may be able to just reverse the direction of the sync. I'm not sure.

If the data is present but difficult to restore, it's annoying. You might need to spend a few days fixing stuff.

If the data is gone, it's devastating and can bankrupt a company. On a personal level it's the same as having all your photos destroyed in a fire. And backups not containing the right data are very common.

it’s a question of whether you can restore a botched system with a few commands and in a realistic amount of time.

A few years ago my employer was the victim of randsomware. We're speaking here about a massive network and all sorts of databases and services build on top of those, spanning decades and many different technologies. Basicly several thousand employees and a decade long focus on working digital and automation. Data restoration was not an issue. I haven't heard of anyone losing data.

However, restarting all the services was not as easy. Many of these depended on each other and there were some circular dependencies that have grown organicily over the years. Took about two months to restore core functionality (mostly SAP and email) and many more months to restore all sorts of support services that were required for normal day-to-day work. Two years after the incident the last applications were back online.

Yep. At one place I worked, we did a big off-site disaster recovery exercise every year.

Most of the time it went fine, but there were multiple years where a restore didn't work due to an issue with one or more tapes. Either the data and/or indexes couldn't be read, or the tape physically failed during the restore.

Backups aren't backups unless they're tested.

Absolutely. Used to work at a small MSP. Got ultra unlucky in that we got chosen as the rest case target for a zero day that leveraged our Remote Support tools so our own systems and all of our client systems that were online got hit with ransomware in a very short time frame.

Some clients had local backups to Synology boxes and those worked ok thankfully. However all the rest had backups based on Hyper-V. The other local copy was on a second windows server that also got hit so the local copies didn't help. They did also have a remote copy which wasn't encrypted.

So all good right? Just pull the remote backup copy and apply that.... Yea every time we had ever used the service before had either been single servers that physically died and took disks along on the death or just file level restores.

Those all worked fine. Still sounds like not a problem right? Nope. We found both that a couple of the larger servers had backups that didn't actually have everything in spite of being VM images. No idea how their software even was able to do that.

And the worse part was that their data transfer rate was insanely slow. About 10mbps. Not that per server or par client. Nope that was the max export rate across everything. It would have taken literally months to restore everything at that rate.

I hate to say it but yes we did in fact pay the ransom and the. Had to fight for several days going through getting things decrypted. Then going through months of reinstalling fresh copies and/or putting in new servers. Also changing our entire stack at the same time. Shockingly we handled it well enough we lost no clients. Largely because we were able to prove we couldn't have known ahead of time.

If you read through all that I'll even say the vendors name. It was StorageCraft. I now have a deep hate for them.

Also one more is that with the old Apple HFS+ filesystem based time machine backups it would sometimes report as a valid self checked backup even if it had corruption. It would do this as long as some self check confirmed that it could fix the corruption during a restore. However if you tried directly browsing through the time machine backups it would have files that couldn't be read, unless again you did a full system restore with it.

Nearly lost my wife's semester ending before finding it worked that way.

I can't confirm it but seems it is fully fixed with APFS and might be one of the reasons they spent the effort to make that transition.

I've had an IT Career for about as long as you. I've had 2 memorable restore failures and got real lucky both times.

The first was a ransomware incident, and the onsite backup was not hit, but it was corrupt. Thankfully, the client had been using a 3-2-1 strategy, and the off-site one was fine.

The second was a situation where a failed update rendered a client's RDS unbootable. This time, they didn't have an on-site backup and the off-site one was corrupt. This time I happened to get immensely lucky in that there was no real data on that RDS, so I was able to spin up a fresh one, and install their LOB app and all was good.

We now test that all backups are stable every 6 months.

in my ~15 years of IT, I’ve never seen a backup not restore properly.

I remember Outlook backups failing like nothing else during the restore process 25 years ago.

Which was fucked because it would take 2 weeks to rebuild only to find out it didn't work.

https://youtu.be/tLdRBsuvVKc

Fun video. Many backup options failed iirc.

Cool.
Still doesnt mean you can boot from it