I’m skeptical. Every single demo I have seen of LLM powered NPCs is garbage.

They would need really good memory. And be very locked down on hallucinations that would be misleading or otherwise nonsense in the context of the world. Like, they can’t reference they are in a game on accident. They can’t suggest you do an action you can’t. They can’t invent characters that are not “real” in the game. They need to be consistent over time. They can’t spoil the things they were prompted to avoid saying.

It’s quite a task, and no demos have depicted anything that feels like a step beyond hand made dialog and scripts. I’m not saying it’s impossible, but I haven’t come across an example that seems to be heading in this direction. They are all just LLMs being prompted at the appropriate times. No additional infrastructure or tuning, etc.

All of the current issues with a chatbot not remembering what you said to it apply here too. And all of the things I listed get even more difficult if you let the user “talk” to the NPCs in any free form way. In other words, if you let the player role play or attempt to exploit the NPC that could potentially break the game and destroy the whole point of it all.

But as a concept, NPCs that can truly understand context and the player instead of just being dialog trees, sounds awesome. I’m just very skeptical due to what I know about current LLMs. And in part due to the overinflated hype that demos that raced to show off this concept carried.

Adam Connover and Dan Olson discuss one of Nvidia's touted AI-in-games products which was similar to what you were saying. You talk on mic to the NPC, the NPC feeds your speech to its LLM, it replies back to give you a quest or something.

And then they come to the obvious conclusion: players would abuse the shit out of it.

That whole podcast is worth watching, they discuss a lot more than just games and AI.