30
Looking for some advice on moving 100TBs of data from the cloud to tape (lemmy.world)
submitted 6 months ago* (last edited 6 months ago) by cm0002@lemmy.world to c/datahoarder@lemmy.ml
14 comments fedilink hide all child comments

With Google Workspace cracking down on storage (Been using them for unlimited storage for years now) I was lucky to get a limit of 300TBs, but now I have to actually watch what gets stored lol

A good portion is uh "Linux ISOs", but the rest is very seldom (In many cases last access was years ago) accessed files that I think would be perfect for tape archival. Things like byte-to-byte drive images and old backups. I figure these would be a good candidate for tape and estimate this portion would be about 100TBs or more

But I've never done tape before, so I'm looking for some purchasing advice and such. I seen from some of my research that I should target picking up an LTO8 drive as it's compatible with LTO9 for when they come down in price.

And then it spiraled from there with discussions on library tape drives that are cheaper but need modifications and all sorts of things

top 14 comments
sorted by: hot top controversial new old
[-] scrubbles@poptalk.scrubbles.tech 9 points 6 months ago* (last edited 6 months ago)

I've done a good chunk, but I haven't found a good source for tape drives. Mine I bought second hand and it overheats and I found that it actually fails it's testing, so be extremely weary on buying them. I did LTO-8, but I've been waiting for years now for them to drop in price. (I probably did the same comparisons as you, TB/$ pricing).

The biggest thing is to make sure you support LTFS. I've tried multiple softwares, even down to tar, but LTFS will make your life simpler. It acts as a simple mount in linux, you just copy and paste.

Tapes themselves are all pretty much the same. I buy them new, never used, just like HDDs, and they're cheap enough that who cares. Note that the compressed sizing will never be reached. Remember that most ISOs are already compressed, and most of your data is already compressed, you can't compress compressed data, so it's pretty much a straight shot.

Tape can be a great option, I keep mine in a safety deposit box, fully encrypted. However getting it all set up is a pain in the ass. If you find a good drive at a good price, let me know.

[-] cm0002@lemmy.world 3 points 6 months ago

Thanks, yea it looks like this is gonna be a long game to acquire lmao

How repairable are these things?

[-] scrubbles@poptalk.scrubbles.tech 3 points 6 months ago

In my experience? Minimal. Although when you buy a single drive for over $1k you're also not super into cracking it open and repairing things. From what I can tell, it's like a blu-ray drive, but more complex, and more delicate. Not to mention since it's tape you technically need a clean room, because dust can fuck up the heads.

If I could source decent drives for less than 10 grand I'd be a huge supporter of it, but us pro-sumers are pretty much left with ebay.

[-] constantokra@lemmy.one 2 points 6 months ago

Do you have any distro or hardware advice? I have an external drive, and I've bought several fiber channel cards for it and none of them seem to work either with my motherboard or distribution.

[-] scrubbles@poptalk.scrubbles.tech 3 points 6 months ago* (last edited 6 months ago)

I used an LSI 8i SAS card, with a standard SAS cable to my internal SAS tape drive. Then I just used plain ubuntu to get it up and running. The LSI cards are my go-to for anything SAS, used them for 10 years and never had a single one fail.

Edit @constantokra@lemmy.one I remembered I had this saved, this saved me a ton of time setting up my drive last time, everything I needed: https://www.linuxquestions.org/questions/linux-hardware-18/lto-tape-drive-linux-experience-4175620090/

[-] constantokra@lemmy.one 3 points 6 months ago

That's a great resource. Thanks for sharing.

[-] constantokra@lemmy.one 3 points 6 months ago

I probably should have gotten a sad drive, but I found a good deal on an external fiber channel one and I didn't realize how difficult fiber channel cards could be.

[-] scrubbles@poptalk.scrubbles.tech 3 points 6 months ago

I'd be interested to hear how it goes, I'm looking at replacing mine. If you get yours to work, please report back. I'm actively looking for a new LTO8 drive

[-] itsnotits@lemmy.world 2 points 6 months ago

actually fails its* testing

[-] peregus@lemmy.world 2 points 6 months ago

100TB??? What the heck did you store in there? A lot of "Linux ISOs", ok, I get it, but how many GB is each? Some may be 10/20GB, but I guess that most of them are about 2/4GB. Some images of old drives, ok (some TB), and then?

[-] TheFrenchGhosty@lemmy.pussthecat.org 4 points 6 months ago

I have ~80TB of linux ISO personally.

It's not hard to get that much, I'm far from having everything I want.

[-] cm0002@lemmy.world 3 points 6 months ago

I've lost count, but upwards of 10,000 Movies/Shows (and ofc each show could have 10, 20 or even 100+ individual files) that only comes to about 100-120TBs

The remaining 150 or so (My usage is 280TBs, I just locked in a limit of 300) is, well, my entire life that isn't deemed so important as to go to my actual personal GDrive. It's not just some images, it has every image of every drive and device I've ever owned just prior to it being put out of service.

[-] ShortN0te@lemmy.ml 0 points 6 months ago

but I guess that most of them are about 2/4GB.

Do you watch your Linux ISOs on a phone or what? Even Netflix has higher quality in HD

[-] peregus@lemmy.world 0 points 6 months ago

Netflix streams 1080 videos at a maximum bitrate of 5Mbps. 5Mbps * 60 seconds * 90 minutes / 8bit / 1024 = 3.3GB

this post was submitted on 16 May 2024
30 points (94.1% liked)

datahoarder

6766 readers
13 users here now

Who are we?

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

We are one. We are legion. And we're trying really hard not to forget.

-- 5-4-3-2-1-bang from this thread

founded 4 years ago
MODERATORS
archivist@lemmy.ml