So, I do LARP. And LARP means you roleplay as someone else. To roleplay as someone else, you need to know a few things. For example, I play a pirate, so I need to know what a forecastle is, what a capstan does, who the boatswain is and that a mizzen is the third mast. I am entirely capable of inventing a 10 minute story that involves lots of nautical terms about chasing and boarding a ship. That story will be entirely plausible and will impress anyone who doesn't know anything about sailing a full-rigged ship. It might even be correct, but not on purpose.
And if you put me in a room with someone who DOES know about sailing anything larger than 4 meters, I'm going to fail as soon as they start talking. Because obviously I only know enough to pretend to be a sailor. I can sound like one just fine, but I'm not.
LLMs are larping at being experts. They can't actually do anything, they can only sound convincing. They might even be right, but they won't be right on purpose.
