datahoarder

Hey everyone,

I’m working on archiving a few profiles from Loyalfans, but I’ve hit a wall with their CDN (CloudFront) security and rate-limiting. I’m looking to grab all media (high-res images, GIFs, videos, video thumbnails & audio), but the platform seems particularly hostile to bulk downloading. Has anyone successfully scraped/download a profile on Loyalfans? If YES! then how?

The site uses heavily signed URLs with Expires, Signature, and Key-Pair-Id parameters. These seem to be session-bound or very short-lived.

What I’ve tried so far:

1. Manual "Save As" (Shift + Right Click): Result: Works for the first 10-15 files, then falls apart.
   The Issue: I’m running into what looks like a cache collision or rate limit. After a few downloads, the browser starts saving randomly previously downloaded imagese instead of the new one. It only resolves if I wait 30+ minutes, try again & then continue in this cycle.

2. HAR Extraction & Shell Scripting: Result: Partially successful but extremely finicky. The Issue: I’ve been saving .har files from the network tab, then using grep to grab the CDN links. The problem is that the HAR often picks up thumbnails (_md.jpg, _sm.jpg) or pre-fetched neighbor images. Furthermore, if I don't run the wget/curl script quickly enough, the signatures expire.

3. Selenium-based Python Script: Result: Identical to the manual method. The Issue: Even with headless browsing and random delays, the CDN eventually detects the automated behavior and starts serving 403s or throttles the connection, resulting in the same "duplicate image" cache bug.

4. Vergil9000's Loyalfans Downloader: Link: https://github.com/Vergil9000/LoyalFans Result: Failed completely. I can load a list of profiles I follow, but the actual scraping/downloading logic seems broken or outdated for current site architecture.

Many thanks for taking the time to read my post. Any help would be greatly appreciated ....