But it is, necessarily.

For example, when we make shit up, we're aware that the shit we made up isn't real. LLMs are structurally incapable of recognizing the distinction between facts they regurgitate and the ones they manufacture from whole cloth.

You didn't have to consume terabytes of text to build a model for how to form sentences like a human, you did that with a few megabytes of overheard conversation before you were even conscious enough to be aware of it.

There's no model of intelligence so over-simplified to the point of giving LLMs partial credit that wouldn't also give equivalent credence to the "intelligence" of search engines.