this post was submitted on 10 Jul 2026
143 points (99.3% liked)
Fuck AI
7633 readers
1480 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I've opted out of search (not just google, most other commercial search) in ~2024. Have not regretted it since, happy to see even commercial entities coming to the conclusion that Google's garbage and not worth it anymore.
Now, if they'd also follow the past set by indies, and block AI crawlers too, and make that the norm, that would be grand.
While that would be great, you can't easily block AI crawlers these days. The vast majority of crawler traffic we've been seeing in the past year has been random bullshit from residential proxies. Less than a third of our crawler traffic self identifies anymore. What's super annoying is that the user agents they use are so ludicrously fake. From my random samplng they frequently claim to be from a Wii using an Opera browser or a Nexus 5. I don't know why those seem so prevalent other than to troll us.
Uh, I beg to differ. I've been blocking most of them for the past year, using essentially three ifs in a trenchcoat.
Bullshit user agents are taken care of by checking headers other than
user-agent: if they say they'reChrome/orFirefox/, check if they sentsec-fetch-mode. Didn't? That's very likely a crawler (and the handful of false positives are easy to make an exception for). For residential proxies, the same applies. For crawlers that piggy-back on Chrome, they usually crawl an URL queue, so if you poison their queue, you can catch those too.At this point, out of ~100 million requests / day, I'm firewalling ~98 million off. Out of the remaining 2 million, ~90% of them gets served garbage to continue poisoning the URL queues. I can serve the rest on a potato, even if some of them are crawlers.
I did have a few people contact me about false positives, but those were very, very few (and also very easy to address). Very little CPU, RAM or bandwidth required, the vast majority of bots caught, negligible false positives. Deploying the solution isn't trivial (yet), but it also isn't hard either.
I don't know that your solutions are viable for a lot of companies, but I can believe that it is effective. It appears your situation has a lot more room for what I'll call "decisive choices" that ours. Our company 100% wants to be indexed on every search it can, so blocking most of the official crawlers is out, though I do limit access to only the places we want them to index for anything I can identify as a bot or bot adjacent. As for the vast majority with the bullshit user-agents, historically, heads tend to roll here when blocking content/requests for legitimate users, so while false positives happen, they need to be kept to a minimum. I've had to roll back several mechanisms that somehow ran afoul of edge case users. So I'm open to trying something similar, I won't be able to do it like you are and will probably have far less success as a result. Not that you don't likely already know this, but I believe that if your solution does become more mainstream, many crawlers would probably adapt to simply make more thorough use of sec-fetch-mode and other headers to more believable as a valid client request.