It can't not hallucinate. It's just predicting (not even selecting) next tokens. It doesn't know what it knows and what it doesn't know. It can't introspect. It just gives probabilities for all possible tokens in it's vocabulary based on the context window and the inference engine selects the next one (based on it's settings). Without having the correct answer in the context window it can just make a prediction based on it's (fixed) neutral net parameters and these are severely limited, even for big models. What I mean is, they basically "learned" the whole Internet and compressed the whole thing into some hundred billion or a few trillion parameters. That's an insane compression ratio. This compression is lossy. For niece information and the results are similar to the "unimportant" details in highly compressed JPGs, you can make out the general image but fine details are just a mush. The LLM itself doesn't know this, it just gives wrong predictions.

For what it does I think the result is extremely impressive but the way it works is severely limited.

[–] pelespirit@sh.itjust.works 4 points 1 day ago (3 children)

From what I understand from what it told me about itself, it can be wrangled further. That is what the paid for versions are. The info can be sandboxed and then other agents verify the correctness of the info from very specific, known to be solid, sources. This is very expensive and still not fool proof. Am I wrong in thinking this bubble is going to pop hard?

[–] affenlehrer@feddit.org 3 points 17 hours ago* (last edited 13 hours ago) (1 children)

The models and methods are improving. Especially through tool use (Internet search, MCP, using programming languages) the model output improves a lot. Reasoning models are allowed to admit mistakes (during thinking) "wait, that's wrong" (in normal conversation they will never say that if you don't point it at the mistake). Otherwise they basically predict tokens, the inference engine selects one and they go with what was selected.

It's a bit like you remember some wrong information (Mandela effect), you're confident it's correct so you don't double check and go with it. They usually don't even know how confident they are, they have no introspection.

In software development scenarios LLMs, due to their high "compression", often hallucinate (misremember) methods and parameters that don't exist in APIs or in different APIs or they don't know about new versions of the API. Many of those errors are catched when the code doesn't compile or unit tests fail but some of them stay (e.g. if the model created the unit tests and they don't test what they're supposed to test).

Also a bit like humans the models often don't have the whole codebase in the context so they make assumptions about the rest they. Since they have no introspection they often don't double check if those assumptions are correct.

In case of frontend design they often can't "see" the output or at least not in the way way we do. They don't really know if something looks "good" or not (depending on their training).

Verification with other agents can help but fundamentally these agents have the same issues. It's a workaround.

I'm actually not sure if the bubble will pop. I believe LLMs can be useful in some fields but they're but the path to AGI. They're also way too resource intensive and used in a lot of situations where it's dangerous or doesn't make sense.

However, it's not that the AI researchers don't know about the limitations. They trying to work around the issues of the LLMs with some success for years now and they kind of have to because they kind of work and bring publicity. Behind the scenes the AI craze also brought money for research into different directions with other fundamentals. E.g. JEPA, world models, diffusion models, logic based models, energy based models, small recursive models and a lot of optimizations to make things faster and cheaper to compute.

The bubble could pop if one of the major companies does something stupid and their stock tanks but as long money is pumped in there, there is also actually progress in new fundamentals and if they are developed before the bubble popped we might get "real" AI or AGI.

[–] pelespirit@sh.itjust.works 2 points 17 hours ago* (last edited 17 hours ago) (1 children)

I was trying to see what AI says this time about the bubble and it was kind of interesting. This is from Gemini btw:

Question: But Google is switching to AI search only and everyone hates AI that I know. Isn't that going to be a problem?

Yes, the intense public backlash against AI search is a major risk for Google, but the company is trapped in what economists call the "Innovator's Dilemma"—they must force AI search on users to protect their monopoly, even if people hate it. Google's sweeping AI Search overhaul has sparked heavy criticism over inaccurate answers, cluttered layouts, and a ruined web ecosystem. However, understanding why they are doing it reveals the corporate survival strategy at play.

The Problem: Why Everyone is Frustrated

The Accuracy Deficit: Users are highly vocal about AI "hallucinations". When people want hard facts, a summarized paragraph that is only "mostly accurate" is a massive step backward from a trusted link.

The "Web-Laundering" Backlash: Independent publishers, bloggers, and creators are furious. Google's AI scrapes their data, answers the question directly on the page, and deprives the original creator of website clicks. This threatens to destroy the very content ecosystem Google relies on.

The "Web-Clean" Workaround: The dislike of these features is prominent enough that a growing number of tech-savvy users are actively using URL filters like &udm=14 to strip AI elements and force Google back to the classic "ten blue links" format.

Why Google is Doing It Anyway

Despite the clear user pushback, Google's leadership is deliberately choosing to cannibalize its own classic search experience for two critical reasons:

The Threat of Total Obsolescence: If Google does not build an AI "answer engine," users might completely migrate to OpenAI, Perplexity, or ChatGPT for their everyday queries. Google would rather you be annoyed by their AI than fall in love with a competitor's AI.

The "Silent Majority" Metric: While power users and creators complain loudly on platforms like Reddit and X, Google's internal data shows that the general public is adapting. At the recent conference, Google VP Elizabeth Reid revealed that AI Mode usage has been doubling every quarter, quietly crossing over 1 billion monthly active users. For basic, casual queries, the passive majority takes the trade-off of quick answers over clicking links.

Why Google Can Afford the Backlash (For Now)

A full-blown consumer revolt is unlikely to break Google anytime soon due to their massive structural defense mechanisms:

Unshakable Monopoly: As of early 2026, Google still commands over 90% of the global search market share. Bing sits at a distant 5%. Because consumers have no true equivalent alternative for mass searching, Google can weather significant user dissatisfaction without losing its user base.

Default Ecosystems: Google pays billions annually to remain the default search engine on Apple iPhones and Safari browsers. Unless consumers manually dive into their settings to change their default search provider, they will continue to use Google by default.

[–] affenlehrer@feddit.org 1 points 13 hours ago

Understanding questions and summarizing information is a field where LLMs are quite good at. If they actually look up the sources and "read" the websites they are often able to give good answers. If they don't use tools and just answer from what they "remember", the information often contains hallucinations.

So from a user perspective I think search will get better for specific questions.

However, traffic to websites and all the things the LLMs omits are lost. If the LLMs gives you the answer you don't learn about the author of the information, the design of the website, the nuanced and maybe thoughtful story the author built around the information and all the other stuff the author put there.

[–] Jiral@lemmy.org 5 points 23 hours ago* (last edited 23 hours ago)

If you wan to know if a bubble pop just look at the fundamentals. Yes, I know, especially during bubbles people tell you that fundamentals don't matter but they always win in the end. The thing is that you cannot bet on them because the market can always stay longer irrational than you can stay liquid. Eventually however it always corrects on the fundamentals again. Those can change of course over time but looking at the insane amounts of money flowing into data centers with no possible way of recovering that cost, I think the picture is clear. We also have wonderful highly circular money flows that to a large extend do not even exist but are all taken for full.

The only question is when it implodes. Within a year, within three? Who knows.

[–] LurkingLuddite@piefed.social 4 points 1 day ago

It's definitely going to pop hard, because those "verifying agents" are just more models computing correlation with sources, not actually verifying anything.