In the last second of the video the player looks at a wall and then looks away, and they've transported to somewhere entirely different.

This is showing us how toddlers see the world. The model currently lacks object permanence. Everything outside its current field of view stops existing. When asked to redraw, it has to start from scratch. Everything in its world is ephemeral, floating around haphazardly. It has no hard ground to fall on and rise up from.

This model is interacting with its world as an 8-18 month toddler would. Instead of pointing it at Doom, I'd like to know what it does with an actual camera.

Point that generative model back at itself and give it access to a real-world against which to compare its predictions.