This continues to boil down into that tired argument that an amalgamation of human behavior is distinct from how humans actually behave, but since no one can actually prove how humans produce thoughts, it follows you can't actually prove that an LLM actually works or doesn't work any different.

So I dont really dig into that argument.