39

Hello !piracy@lemmy.dbzer0.com

I intend to upload an archive of a website I scraped for the last few weeks. The torrent is about 4 TB large and contains 400000 folders which in turn contain 40 .jpg's on average (plus some metadata).

Should I just create the torrent with the files as they are right now, or should I put the individual folders in archives (or maybe even the entire torrent?)?

Thanks in advance

you are viewing a single comment's thread
view the rest of the comments
[-] bear_with_a_hammer@lemm.ee 7 points 1 year ago* (last edited 1 year ago)

Yes, you can, I would say that it would be good to deduplicate files first.

Also put a txt file with locations and file hashes in torrent, it could be usable if some valuable files were lost in the future due to seeders being inactive and torrent becoming dead, to make .torrent file smaller you would choose a bigger piece size, but in this case users would have to waste additional traffic if they are downloading only files they choose.

It would be nice then to have BEP-47 support for file alignment, but this comes with many padding files locations in torrent's metadata which makes it bigger.

If you have a big nested structure of folders, then every file there would have something like this info in metadata: folder1/folder2/folder3/file.jpg

All of these problems could be solved, if you're choosing format to BitTorrent v2 only while creating the torrent, but keep in mind that it won't work with Transmissions and uTorrents. qBittorrents are working.

You could choose a hybrid, but it makes torrent size a lot bigger.

So overall, bestly you would choose v2 or hybrid (with hybrids sacrificing torrent's size greatly, but providing your users with file hashes, deduplication and reduced bandwidth), or make at least a v1 torrent, optionally in a client that supports BEP-47 after deduplication and adding .txt

this post was submitted on 26 Jul 2023
39 points (95.3% liked)

Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ

53939 readers
330 users here now

⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.

Rules • Full Version

1. Posts must be related to the discussion of digital piracy

2. Don't request invites, trade, sell, or self-promote

3. Don't request or link to specific pirated titles, including DMs

4. Don't submit low-quality posts, be entitled, or harass others



Loot, Pillage, & Plunder


💰 Please help cover server costs.

Ko-FiLiberapay


founded 1 year ago
MODERATORS