datahoarder

I'm in the process of downloading both dataset 9 torrents (45.63 GB + 86.74 GB). I will then compare the filenames in both versions (the 45.63GB version has 201,358 files alone), note any duplicates, and merge all unique files into one folder. I'll upload that as a torrent once it's done so we can get closer to a complete dataset 9 as one file.

• Edit 31Jan2026 816pm EST - Making progress. I finished downloading both dataset 9s (45.6 GB and the 86.74 GB). The 45.6GB set is 200,000 files and the 86GB set is 500,000 files. I have a .csv of the filenames and sizes of all files in the 45.6GB version. I'm creating the same .csv for the 86GB version now.

• Edit 31Jan2026 845pm EST -

  • dataset 9 (45.63 GB) = 201357 files
  • dataset 9 (86.74 GB) = 531257 files

  I did an exact filename combined with an exact file size comparison between the two dataset9 versions. I also did an exact filename combined with a fuzzy file size comparison (tolerance of +/- 1KB) between the two dataset9 versions. There were:

  • 201330 exact matches
  • 201330 fuzzy matches (+/- 1KB)

  Meaning there are 201330 duplicate files between the two dataset9 versions.

  These matches were written to a duplicates file. Then, from each dataset9 version, all files/sizes matching the file and size listed in the duplicates file will be moved to a subfolder. Then I'll merge both parent folders into one enormous folder containing all unique files and a folder of duplicates. Finally, compress it, make a torrent, and upload it.

• Edit 31Jan2026 945pm EST -

  Still moving duplicates into subfolders.

• Edit 31Jan2026 1027pm EST -

  Going off of xodoh74984's comment (https://lemmy.world/post/42440468/21884588), I'm increasing the rigor of my determination of whether the files that share a filename and size between both version of dataset9 are in fact duplicates. This will be identical to rsync --checksum to verify bit-for-bit that the files are the same by calculating their MD5 hash. This will take a while but is the best way.

• Edit 01Feb2026 1227am EST -

  Checksum comparison complete. 73 files found that have the same file name and size but different content. Total number of duplicate files = 201257. Merging both dataset versions now, while keeping one subfolder of the duplicates, so nothing is deleted.

• Edit 01Feb2026 1258am EST -

  Creating the .tar.zst file now. 531285 total files, which includes all unique files between dataset9 (45.6GB) and dataset9 (86.7GB), as well as a subfolder containing the files that were found in both dataset9 versions.

• Edit 01Feb2026 215am EST -

  I was using wayyyy to high a compression value for no reason (ztsd --ultra --22). Restarted the .tar.zst file creation (with ztsd -12) and it's going 100x faster now. Should be finished ~~within the hour~~

• Edit 01Feb2026 311am EST -

  .tar.zst file creation is taking very long. I'm going to let it run overnight - will check back in a few hours. I'm tired boss.

• EDIT 01Feb2026 831am EST -

COMPLETE!

And then I doxxed myself in the torrent. One moment please while I fix that....

Final magnet link is HERE. GO GO GOOOOOO

I'm seeding @ 55 MB/s. I'm also trying to get into the new r/EpsteinPublicDatasets subreddit to share the torrent there.