PieFed help

Ok good, so we know it's scrapers and not something broken.

What you do next depends how much work you want to put into fighting scrapers. Switching to Private mode is extremely easy and very effective which is why I suggested it first. But makes it impossible for people to get a feel for what your instance is like before joining it, which sucks.

Often a couple of hours of private mode is all it takes for the latest wave of scrapers to give up and leave, so you can switch back later and see what happens.

I am thinking about making it possible for PieFed to automatically switch to private mode when CPU load crosses some kind of threshold. Then automatically switch back again later. Or have a cron job that switches to private mode on a schedule (during quiet times) just to fuck with whatever bots are hanging around.

But if you want to put in more effort there's many options which will help you keep Private mode to a minimum.

• Block HTTP 1.1 connections - real people use http 2.0. Surprisingly effective. In Cloudflare you can block http 1.1 AND where the user agent string contains "Chrome" which will block scrapers while letting federation traffic through - surely some fedi platforms federate using 1.1 but I haven't checked.
• Block old browsers - scrapers use fake user agent strings with a random browser version but 95% of those versions are so old that no human ever uses them. You can do this in nginx but the regex tricky. A chatbot can help with this.
• Watch web server logs and find patterns and manually block them in the nginx config. Repeating IP addresses, user agents strings, etc. I blocked Chrome 133.0.0.0 and Chrome 144.0.0.0 yesterday, for example.
• Anubis. Very difficult to get right.