A few years ago anything you wanted to search for would generate hundreds of pages of results. From sources all over. If you structured your query well you could dramatically cut down the irrelevant results and then spend a bit of time skimming through the offerings until you found your target.
Searching felt like a hunt, it could take you to far flung corners of the web, and hitting the jackpot on some obscure forum was part of the fun.
Now searching anything feels incredibly frustrating. You know what you are looking for is out there somewhere but it feels impossible getting to it. Basically we have only two main routes now, Google and Bing (and I swear Bing is just copying Google's homework). I've tried a handful of others and they all fall very short.
Something has changed dramatically under the hood.
It seems like the results are very shallow and no amount of prodding will get either search engine to produce anything besides the laziest output. Any subject has a limited pool of sites they lean on and then pretend that that's all of the Internet.
It generalizes very quickly and if you try to counter by putting things in quotations it gives up.
Even in the rare moments I've buckled and used their stupid AI, it takes even more time browbeating the program to just deliver the damned specifics.
What is going on??
How do we get our old search back?
I swear every day of trying to navigate the web feels worse than the day before.
How did we get here?
There's got to be something better, right? Or something we can do to pave a new structure that benefits the denizens of the web better than the lazy slop our tech overlords scrape off the table for us. What can we do?
As others have mentioned, it was engineered to be worse for the capitalists to make profits from ads and the statistics to look better to shareholders. This in turn has made people to create a shit ton of garbage articles and blogs that say a ton while saying nothing, so even if you used a new search engine with the funding of Google but the algorithm of back then it would still be shit probably, because the volume of shit on the internet has now reached a level where finding anything in it is impossible.
The explosion of AI has come right at this moment where search engines have been lobotomized and gives people an option to be able to find things again, which is also another factor contributing to the profits of the AI companies, but even that is limited to what an LLM can average out from the totality of information that it has been fed and has the obvious danger of making stuff up and treating its biased training data as truth.
Generally, I think we're fucked. I really don't know if we're ever going to get back the internet experience that once existed and what it's going to take to get there, but it would probably involve a massive societal change to come before it.
I'd add that traditional search can't reasonably hope to compete with AI in this environment. If you're running a traditional search engine right now, there's a real argument that the best business decision is to accelerate the enshitification to capture as much value as you can from your remaining audience while you still have it.
This is the usual cycle with these sorts of technologies - once a replacement arrives, the natural choice is to shift from growth to exploitation.
This is going to get worse before it gets better.
At this point I think the only way forward is for enough like-minded people to sort of stake their claim on some semi-walled off portion of the internet and greatly limit access from swaths of IP addresses.
Sort of like web rings 2.0. Everyone, or enough people, will have to chip in on costs and provide either money or hardware.
There will have to be some sort of regulatory body with voting members.
To be 'in' the site has to abide by certain advertising restrictions and privacy agreements.
A decent use (finally) for llms would be to audit the private index and remove slop. I think we're just at the beginning of what will be a resource hogging AI arms race.
It's sort of what we've been trying to avoid since the beginning but it seems inevitable at this point.