705
Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.
(i.imgur.com)
this post was submitted on 17 Aug 2025
705 points (99.7% liked)
Technology
468 readers
279 users here now
Share interesting Technology news and links.
Rules:
- No paywalled sites at all.
- News articles has to be recent, not older than 2 weeks (14 days).
- No videos.
- Post only direct links.
To encourage more original sources and keep this space commercial free as much as I could, the following websites are Blacklisted:
- Al Jazeera;
- NBC;
- CNBC;
- Substack;
- Tom's Hardware;
- ZDNet;
- TechSpot;
- Ars Technica;
- Vox Media outlets, with exception for Axios;
- Engadget;
- TechCrunch;
- Gizmodo;
- Futurism;
- PCWorld;
- ComputerWorld;
- Mashable;
- Hackaday;
- WCCFTECH;
- Neowin.
More sites will be added to the blacklist as needed.
Encouraged:
- Archive links in the body of the post.
- Linking to the direct source, instead of linking to an article talking about the source.
Misc:
Relevant Communities:
- Beehaw Technology- Technology Related Discussions.
- lemmy.zip Technology- Hard Tech news.
founded 3 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I just thought that having a client side proof-of-work (or even only a delay) bound to the IP might deter the AI companies to choose to behave instead (because single-visit-per-IP crawlers get too expensive/slow and you can just block normal abusive crawlers). But they already have mind-blowing computing and money ressources and only want your data.
But if there was a simple-to-use integrated solution and every single webpage used this approach?
Believe me, these AI corporations have way too many IPs to make this feasible. I've tried per-IP rate limiting. It doesn't work on these crawlers.
What if we had some protocol by which the proof-of-work is transferable? Then not only would there be a cost to using the website, but also the operator would receive that cost as payment.
It's theoretically viable, but every time that has been tried has failed
There are a lot of practical issues, mainly that it's functionally identical to a crypto miner malware
Are you planning to just outright ban IPv6 (and thus half the world)?
Any IP based restriction is useless with IPv6
Not really true, you can block ranges.
Okay, but how does that help? Or are you suggesting just wholesale banning entire ISPs?
Solution was invented long ago. It's called a captcha.
A little bother for legitimate users, but a good captcha is still hard to bypass even using AI.
And I think for the final user standpoint I prefer to lose 5 seconds in a captcha, than the browser running an unsolicited heavy crypto challenge on my end.
https://blog.cloudflare.com/turnstile-ga/
I hate captchas.
AI is better at solving captchas than you.
I tried, and not really.
I had to scrape a site that have some captcha and no AI was able to consistently solve it.
In order to be able to "crack it" I had to replicate the captcha generation algorithm best I could and train a custom model to solve it. Only then I could crack it open. And I was lucky the captcha generation algorithm wasn't to complex and it was easy to replicate.
This amount of work is a far greater load than Anubis crypto challenges.
Take into account that AI drive ocr drinks from existing examples, if your captcha is novel enough they are going to have a hard time solving it.
It also would drain power, which is the only point of anubis.
There is a difference between you (or me) sitting at home working on this and a team of highly motivated people with unlimited money.
The thing is not that it cannot be done, the thing is that the cost is most likely higher than Anubis.