Robots Dreaming in Comic Books: The Genie 3 World Model

Okay so I saw this beta test video the other day and I am literally still thinking about it. Someone uploaded a photo of a static, old-school comic book page. Like just a 2D stylized drawing of a city street.

Oh I know exactly the demo you're talking about. The extrusion one.

Yes! They just stepped into it. Like it instantly extruded into this 3D explorable space and they were walking around inside the comic book. And it kept that weird dream logic and the exact aesthetic vibe of the 2D art, seamlessly. It was insane.

Yeah that is Google DeepMind's Genie 3. And the crazy thing there is, it is not just generating a video of walking through a comic. You are actually playing it. It's a persistent, interactive environment.

Right, which is so fundamentally different from what we were freaking out about last year with Sora or Veo. Because those are passive video clips, right? You type a prompt, it spits out a ten-second video, and you just watch it.

Exactly. Sora is essentially a movie. Genie 3 is a video game. It is what we call a foundation world model. It closes the action-perception loop. And the lineage of this is actually fascinating if you look back at the original February 2024 Genie paper by Jake Bruce and Ashley Edwards.

That was the 2D platformer one, right? The 11-billion parameter model.

Yes, but the breakthrough in that original paper wasn't that it could make a little 2D jumping game. The breakthrough was unsupervised action discovery. They trained it entirely on passive internet videos. Over two hundred thousand hours of unlabelled 2D gameplay videos.

Wait, unlabelled? You mean they didn't feed it the gamepad logs?

No gamepad logs, no game engine code, nothing. Just the raw pixels of people playing games on the internet. And by watching how frames transition over time, the model naturally inferred what actions were possible and necessary to move from frame A to frame B.

Okay but how does an AI know what the A button does if it has never seen a controller? That makes no sense to me.

So they used something called a Latent Action Model, or LAM. Between any two frames of a training video, the LAM basically deduces a continuous, abstract mathematical representation of whatever caused that visual transition. It hallucinates a controller, basically. And then during inference, they map those latent actions to human controls like your WASD keys or your mouse, which lets you drive the simulation.

That is wild. So it just watched enough Mario to realize, oh, when the character goes up in an arc, a button was probably pushed to make that happen.

Precisely. Now, to do that efficiently, they had to use a Spatiotemporal Video Tokenizer, utilizing techniques like VQ-VAE and MaskGIT. That compresses the raw video frames into discrete tokens, stripping away the noise so the model can focus purely on the structural and spatial logic of the scene.

Okay so that was the 2D version. But Genie 3, which Jack Parker-Holder and Shlomi Fruchter rolled out, that is full 3D, photorealistic, 720p running at 24 frames per second. How did they jump from 2D pixel art to that?

A massive scale-up in data and a shift in architecture. For Genie 3, they fused the original autoregressive dynamics with highly efficient latent diffusion models. But the real trick to getting it to a real-time 24 FPS was what they call Stage III Few-Step Distillation. They adapted the bidirectional world model into a fast autoregressive system so the rendering pipeline wouldn't lag.

I mean, 24 FPS is basically cinematic framerate. But what about the worlds themselves? Like, if I upload a picture of my backyard, it can make it 3D, but does it know how physics work in my backyard?

Well, yes and no. For Genie 3, they initialized it from powerful video foundation priors, but more interestingly, they grounded it using massive datasets of real-world spatial data. Including Google Street View.

Oh wow. Google Street View?

Yeah, millions of panoramic transitions. That anchors the physics and the spatial layouts in actual reality. So without being explicitly programmed with a physics engine, Genie 3 demonstrates this deep emergent understanding of physical properties. It natively handles aerodynamics, like if you throw a paper airplane in the simulation, it banks and glides. It does dynamic lighting, and complex water physics with interactive surface wakes and splashes.

And you can inject prompts while you're playing, right? I read you can literally type a prompt to change the weather on the fly while you are walking around.

You can. You can spawn a character, change it from day to night, or model a completely counterfactual scenario instantly. Which brings us to the really surprising part of this whole project.

Yeah, because everyone online is talking about this like it's a threat to Unreal Engine or Unity. Like game developers are terrified.

Right, and that is a total misread of what DeepMind actually wants to do with this. The researchers don't view Genie 3 as a game engine. They view it as a synthetic training curriculum for Embodied AI. Robotics.

Wait, what? How does a video game AI help robots?

Think about DeepMind's SIMA agents. If you are training a physical robot to navigate a house, you don't want it crashing into your walls in the real world. You put it inside Genie 3. Future robots can literally learn in their dreams, running millions of trial-and-error tasks inside these infinite, generated simulations.

Oh man. Learning in their dreams. That is deeply sci-fi but it makes total sense.

And it gets crazier. Waymo recently adapted Genie 3 into the Waymo World Model for autonomous driving. Because Genie 3 understands global physics, Waymo uses it to simulate exceedingly rare, dangerous edge cases.

Oh, so instead of just driving a million miles and hoping you encounter a weird situation, you just force the model to hallucinate one?

Exactly. They can prompt the model to generate a scenario with a tornado, or a wrong-way driver, or an elephant standing in the middle of a highway. And Genie 3 generates the high-fidelity LiDAR and camera data for that fake scenario to train the cars safely.

An elephant on the highway. I love that. So the car thinks it's actually seeing an elephant on LiDAR, but it's just Genie 3 spinning a yarn.

Precisely. But we should also ground this a bit, because as amazing as it is, early-access testers have pointed out some major limitations. It is not about to replace traditional game engines anytime soon.

Yeah, I was going to ask about that. What breaks it?

Combinatorial game logic is a big one. They upgraded it with this Special Memory System that gives it object permanence for over three minutes of exploration, which is great, so it doesn't degrade into hallucinations immediately. But it struggles with long-horizon logic tasks. Like, collect a key, take it to a door on the other side of a map, and unlock it.

Ah, so the environment stays consistent, but the rules of the game sort of fall apart in its head?

Right. It forgets the state of the key relative to the door. And complex physics can get really weird too. Basic physics work, but intuitive classical physics break down. If you try to stack a complex tower of blocks, they might start morphing into each other.

Like a dream. Where things just sort of melt if you look at them too closely.

Exactly. And multi-agent interactions, like a one-on-one combat scenario, frequently break down completely. Veteran game developers have rightly pointed out that the low framerate, laggy response, and the lack of discrete object placement mean this is an incredible research prototype, but still years away from consumer game development.

I mean, fair enough. It can't run a precise competitive shooter yet. But just the fact that it can watch a million hours of unlabelled video, deduce the concept of a controller, and then let me walk around inside a 2D comic book sketch while simulating water physics... I think my brain is still stuck on that.

It is a massive leap forward, especially for robotics and autonomous systems. It is literally teaching machines how the physical world works by having them watch us play in it.

Well, next time my self-driving car successfully avoids an elephant, I will know who to thank. That's all the time we have for today, thanks for breaking this down, Maya.

Anytime, Alex.