So? Google will just do it anyway. There’s no consequences for them.
this post was submitted on 10 Jul 2026
143 points (99.3% liked)
Fuck AI
7623 readers
1275 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
founded 2 years ago
MODERATORS
Someone didn't read the article.
I read it nom not seeing your point. Google adds them anyway.
Read. It. Again.
See what "opting out" entails.
Read. It. For comprehension.
I did.
The insults are cute and all but asking me to prove your point for you isn’t the flex you think it is.
No. You apparently didn't.
Go back and re-read it for comprehension. Look specifically at what opting out involves. Then smack yourself while looking into a mirror. I won't be here to enjoy it because I generally don't want to waste any more time on the wilfully illiterate.
Google: Muahaha, we train our AI with the data from the publishers' sites while indexing them. Then we summarize their site so the user never leaves our site and all the ad revenue is ours, brilliant!
Publishers: People aren't coming to our site from Google search because it just summarizes our site for them. So we lose no traffic by completely blocking Google from crawling us.
Google: *Surprised Pikachu face*
AI-generated image by Mark Stenberg via Gemini
The irony of the subject matters being about how bad the impact AI/LLM is
It's the easiest graphic to make too. You can make it with fucking Chrome Dev Tools
Slop is everywhere, sadly. 😟
Finally. Time to end SEO and ban the crawlers and add tar pits and other poisoning https://rnsaffn.com/poison3/
Oh shit, that's just begging to be combined with something like Nepenthes!
I've opted out of search (not just google, most other commercial search) in ~2024. Have not regretted it since, happy to see even commercial entities coming to the conclusion that Google's garbage and not worth it anymore.
Now, if they'd also follow the past set by indies, and block AI crawlers too, and make that the norm, that would be grand.
While that would be great, you can't easily block AI crawlers these days. The vast majority of crawler traffic we've been seeing in the past year has been random bullshit from residential proxies. Less than a third of our crawler traffic self identifies anymore. What's super annoying is that the user agents they use are so ludicrously fake. From my random samplng they frequently claim to be from a Wii using an Opera browser or a Nexus 5. I don't know why those seem so prevalent other than to troll us.
Uh, I beg to differ. I've been blocking most of them for the past year, using essentially three ifs in a trenchcoat.
Bullshit user agents are taken care of by checking headers other than user-agent: if they say they're Chrome/ or Firefox/, check if they sent sec-fetch-mode. Didn't? That's very likely a crawler (and the handful of false positives are easy to make an exception for). For residential proxies, the same applies. For crawlers that piggy-back on Chrome, they usually crawl an URL queue, so if you poison their queue, you can catch those too.
At this point, out of ~100 million requests / day, I'm firewalling ~98 million off. Out of the remaining 2 million, ~90% of them gets served garbage to continue poisoning the URL queues. I can serve the rest on a potato, even if some of them are crawlers.
I did have a few people contact me about false positives, but those were very, very few (and also very easy to address). Very little CPU, RAM or bandwidth required, the vast majority of bots caught, negligible false positives. Deploying the solution isn't trivial (yet), but it also isn't hard either.
I don't know that your solutions are viable for a lot of companies, but I can believe that it is effective. It appears your situation has a lot more room for what I'll call "decisive choices" that ours. Our company 100% wants to be indexed on every search it can, so blocking most of the official crawlers is out, though I do limit access to only the places we want them to index for anything I can identify as a bot or bot adjacent. As for the vast majority with the bullshit user-agents, historically, heads tend to roll here when blocking content/requests for legitimate users, so while false positives happen, they need to be kept to a minimum. I've had to roll back several mechanisms that somehow ran afoul of edge case users. So I'm open to trying something similar, I won't be able to do it like you are and will probably have far less success as a result. Not that you don't likely already know this, but I believe that if your solution does become more mainstream, many crawlers would probably adapt to simply make more thorough use of sec-fetch-mode and other headers to more believable as a valid client request.
Let's see them try to opt out of AI training.
It's not that complicated - block all bot traffic by default, whitelist the bots you can verify are only indexing for search (which is actually beneficial traffic).
The thing that Google's doing to piss off websites is that they use the same bots for indexing and AI trawling, so if you want to show up on their indexes you have to let them slam your website/server to train their AI.
Now websites are telling Google to fuck off because the cost of AI trawling isn't worth being on their search index
It’s not that complicated - block all bot traffic by default,
It's not that easy to identify bot traffic. AI companies have been incredibly persistent and aggressive at pretending to be humans to scrape sites.
it's not worth the cost because once google has your information, users won't need to visit your website.
the question of "who is going to be creating all this high-quality data not only for no money but also for no recognition?'' goes unasked