this post was submitted on 20 May 2026
104 points (99.1% liked)
TechTakes
2594 readers
53 users here now
Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.
This is not debate club. Unless it’s amusing debate.
For actually-good tech, you want our NotAwfulTech community
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Web publishers should honestly just block googlebot at this point. Why should they provide credibility to whatever Google's stochastic parrot hallucinates if Google won't even give them any kickback?
Microsoft deprecated their Bing API back in August, instead telling people to use some Azure AI thing. DDG and the like weren't affected because they have contracts, but I can't imagine they'll be renewed.
Yikes! 😳😬
I think it's possible to allow google search bots but not Gemini bots?
The point is that there's going to be no difference, soon Google search will be just another chatbot interface.
Guess I'll just have to use ddgo, then ¯\_(ツ)_/¯
Unfortunately DuckDuckGo sucks ass at search, even compared to how much the Google search results have degraded over time. I use the no-AI version as my primary search engine, but I have to resort to using Google to find the thing I'm looking for about 1 in 5 times.
DDG has been my main search engine for over a decade but it has degraded as it became basically a reseller of Bing results after Russia started the current phase of the Ukraine war and they stopped partnering with Yandex.
Probably 2 in 5 ddg searches are worthless for me, but trying again on Google rarely if ever helps. Usually, both are returning SEO slop or Product™️ rather than the thing I was looking for.
If sites start blocking googlebot en masse, then googlebot will just start ignoring robots.txt
Can they just put an EULA on the site and then sue Google for unauthorized access?
Not in the US of course, but in the EU or something
Then you can just block the user agent in nginx or whatever you use, like all the other AI scrapers who ignore robots.txt (*cough* Amazon)
Then the user agent string will just quietly become randomised so you can't match it reliably because it turns out that honouring robots.txt was always little more than a "gentleman's handshake".
this is a problem we have had for a while now, i assure you
Until then Qwant and Mogee should have somewhat usable indeces on their own, at least for European users.