Stanford's Wetzstein Turns Internet Video Into Robot Control Policies

斯坦福大学Wetzstein将互联网视频转化为机器人控制策略 生成式视频世界模型通过先预测视觉结果再利用逆动力学提取物理运动指令，克服了带标注机器人数据的稀缺问题。 Gordon Wetzstein概述了生成式世界模型从交互式视频合成向具身物理AI及机器人领域的演进历程。该讲座对比了学习型神经模型与传统物理引擎，展示了扩散Transformer、流匹配和外部3D几何记忆如何克服上下文窗口限制与幻觉漂移。Wetzstein还介绍了用于纠正自回归展开退化的BAGGER等技术，并详细阐述了将视频预测与逆动力学解耦如何攻克机器人学习中的海量数据瓶颈。 世界模型代表了物理AI的下一个前沿，正从传统图形渲染管线转向动作条件控制的生成式视频引擎，以实现视觉环境的即时模拟。 (0:10) 机器人领域存在巨大的数据差距——约2000小时的动作标注数据对比20亿小时的未标注互联网视频——这可以通过将策略学习拆分为生成式未来视频预测与随后的逆动力学两个步骤来弥合。 (38:20) 通过将持久的3D空间表征（如深度点云或微型地图）与上下文窗口中短暂的像素外观解耦，可以在不引入过高的二次方计算扩展成本的情况下，缓解自回归漂移和幻觉。 (25:49) 斯坦福大学的研究人员提出了BAGGER（反向聚合）技术，通过反转退化序列来生成自监督恢复演示，从而纠正视频展开过程中的分布偏移和暴露偏差。 (33:26)

Gordon Wetzstein argues that learned generative world models represent the next frontier for physical AI by bridging the sim-to-real gap, though deploying them reliably requires solving compounding autoregressive drift and context window limitations. Learned World Models Versus Traditional Physics Simulators (3:17) Traditional simulation splits physics engines from graphics pipelines, relying on explicit 3D states such as mass, friction, and object positions. While interpretable, editable, and lightweight, traditional simulators struggle with complex dynamics like fluid flows and suffer from a pronounced sim-to-real gap. In contrast, learned world models—such as Google DeepMind's Genie 3—use diffusion or flow matching to synthesize photorealistic frames on the fly directly from past history and control inputs without maintaining an explicit 3D mesh. Context Bottlenecks and Methods for Mitigating Drift (19:23) Because video diffusion transformers process sequences with quadratic attention costs during training, token limits restrict temporal memory. Early interactive setups like Oasis lose physical permanence over camera turns, as observed in Minecraft rollouts where looking up at the sky erases the ground from the model's limited context. Furthermore, open-loop autoregression introduces exposure bias, where slight generation errors compound into severe visual and spatial distortions over extended rollouts. Architectural Memory Structures and Spatial Anchoring (23:02) To counter memory decay without unsustainable compute costs, researchers look to tiered memory architectures. Techniques like FramePack compress older history spatially and temporally into unified compound frames, stabilizing character identities. For 3D spatial consistency, systems back-project predicted depth maps into global 3D point clouds or condition on top-down 2D floor plans. In multiplayer tests using the Doom engine, conditioning distributed player streams on a shared central map maintained spatial coherence across 30-minute sessions. Overcoming the Robotics Imitation Learning Data Bottleneck (38:20) Standard robotic imitation learning faces severe data scarcity: while the internet contains roughly two billion hours of video, Wetzstein notes only about 2,000 hours are action-labeled. To exploit internet-scale video, his lab formulates policy control as causal video prediction followed by inverse dynamics. A web-scale video model first visualizes the successful physical trajectory—demonstrated in decanting tasks and tracking hidden objects in the shell game—after which an inverse dynamics model extracts the low-level motor commands with minimal labeled training data. Hierarchical Verification and Real-Time Inference Challenges (48:50) Because generative models inherently lack internal truth grounding, autonomous deployment depends on closed-loop execution over short planning horizons. High-level vision-language models can supervise semantic task decomposition and trajectory safety, while the world model plans immediate motor primitives re-anchored on sensory feeds. However, pre-training foundation models remains compute-intensive, and large 14-billion-parameter diffusion models face steep latency and hardware cost hurdles that currently prevent responsive consumer deployment and production robotics adoption. While video world models provide a promising alternative to classical physics simulation, commercial deployment requires resolving high inference latency, expanding effective context windows, and grounding purely visual predictions with tactile, auditory, and proprioceptive sensor feedback.

斯坦福大学副教授 Gordon Wetzstein 指出，人工智能的下一前沿是物理 AI，其核心引擎正从基于显式几何的传统仿真器转向基于扩散机制的数据驱动型视频世界模型。世界模型与世界动作模型（WAM）有望克服物理仿真的现实差距并赋能具身智能，但其实用化仍受制于长上下文自回归漂移、算力开销以及动作标注数据的极度匮乏。 生成式世界模型的技术转向与工业落地 (1:17) Wetzstein 指出，以 Google DeepMind 的 Genie 3 为代表的现代世界模型本质上是实时、基于动作条件输入的自回归因果视频生成器，彻底打破了传统计算机图形学依赖 3D 网格渲染和物理状态引擎的分离式管线。传统物理仿真器具有高度的可解释性和编辑性，但难以在实时条件下真实模拟流体动力学等复杂外观与物理规律，存在显著的“仿真向真实迁移差距”（sim-to-real gap）；生成式世界模型则基于隐式隐空间，通过无监督学习海量网络视频直接生成逼真的像素流与自然动态。在产业层面，自动驾驶企业如 Tesla 借助数百万车主的真实视频生成罕见边缘案例（如郊区突现大象），Waymo 结合多视角相机与 LiDAR 点云进行安全数据增强。与此同时，行业资本大规模涌入空间智能，例如 World Labs 以 80 亿美元被 AMD 收购，Yann LeCun 的初创团队筹集 10 亿美元，NVIDIA 则大力推进 Cosmos 世界模型平台。当前工业界采用世界模型的最核心动机仍是长尾场景的数据增强与策略预训练。 扩散架构、自回归漂移与显式空间记忆 (19:23) 现代视频世界模型主要采用 Diffusion Transformer（DiT）或流匹配（Flow Matching）架构，利用 VAE 将高分辨率像素压缩至低维隐空间，并通过 $L_2$ 速度场损失进行去噪生成。然而，当这些模型以开环（Open-Loop）模式自回归生成长视频时，会面临类似大语言模型“暴露偏差”的严重自回归漂移问题：由于模型在推理时不断将自身生成的带有微小瑕疵的帧存入上下文缓存，微小误差会呈指数级放大，引发颜色饱和度失真、几何崩塌与运动冻结。受人类大脑分层记忆机制启发，Wetzstein 实验室提出了 FramePack 技术，通过在时间与空间上对远期历史帧进行下采样与显式打包，在极低计算开销下维持了角色的长期一致性。 针对无显式 3D 状态导致的一转头即丢失场景结构的问题（如在 Minecraft 模拟器 Oasis 中视线上移后低下头即生成全新地形），研究团队将生成的深度图实时反投影并融合成持久化的全局 3D 点云地图。当智能体重新回到已知区域时，系统根据当前虚拟相机位姿渲染点云视角作为空间先验，牢牢锁定空间几何结构。这一机制被进一步扩展到经典射击游戏 Doom 的多智能体交互引擎中，通过集中式 2D 顶层小地图挤出的三维深度图来同步多玩家空间状态，各客户端可独立生成渲染流而互不漂移。为了解决开环生成的分布偏移，团队还开发了 BAGGER 框架，将劣化视频片段反向播放并配合文本提示微调模型，使网络自动学会从错误退化状态逐步恢复为清晰画面。 逆动力学解耦与具身智能的数据瓶颈破局 (37:00) 在将世界模型拓展为具身决策的“世界动作模型”（WAM）时，机器人领域面临极度严峻的数据荒。Wetzstein 援引与 Google Gemini 的对比数据指出，目前全球互联网仅能提供约 2,000 小时的机器人动作标注视频，而无标注的一般视频高达 20 亿小时；若将二者按比例可视化，标注数据仅如一个小红点，而未标注网络视频构成了直径达 60 米的巨大圆盘。主流的模仿学习（如 Diffusion Policy）直接学习从观察到控制动作的映射，但因缺少海量监督信号而难以产生如语言大模型般的大规模涌现能力。 为了打破这一瓶颈，团队提出了“视频生成加逆动力学”的两阶段解耦策略：首先利用基于海量网络视频预训练的因果世界模型预测任务未来的视觉观察结果（即机器人在脑海中“想象”未来画面），随后由仅需极少动作标注数据的逆动力学模型（Inverse Dynamics Model）将这一视觉轨迹反解为底层的机械臂末端或关节电机控制指令。该框架已被成功部署于多种复杂的工业长上下文任务，例如汽车零部件拆箱倒料（双臂协作拆开重型板条箱、倾倒滚珠轴承并分拣蜡纸与塑料包装）、农业中转箱清洗拆解，以及无需外部追踪模块即可通过长期上下文追踪被扣在杯中快速洗牌物体的“三杯猜球”游戏。 前沿挑战：推理时延、层级规划与模型闭源趋势 (49:57) 面向实际落地，视频世界模型与世界智能体仍存在多项尚未解决的工程与理论阻碍。首先是极端昂贵的算力和推理延迟：视频自回归注意力在序列长度上面临极高计算开销，目前的十亿级或 140 亿参数 DiT 模型即使经过蒸馏也难以满足消费级实时交互游戏的性能与经济成本要求。其次，纯粹的视频模型缺乏对物理真实性的内在判断，必须依赖由粗到细的层级规划架构：高层视觉语言模型（VLM）负责跨数分钟或数小时的语义任务拆解，底层视频世界模型则只负责生成数秒甚至一秒以内的短时运动轨迹，并在物理执行后由多模态检验器进行闭环验证。 Wetzstein 同时观察到，尽管开源社区已从早期的 AnimateDiff 演进至 NVIDIA Cosmos 和 Wan2.1，但前沿商业实验室正变得日益封闭，完全开源权重的基座模型逐步减少。他直言，目前的现有基础模型仍未成熟到可以直接支持训练面向终端客户的商业机器人，未来的物理落地不仅需要大幅提升推理与表征效率，还必须将触觉、音频和本体感知信号深度融合进世界模型的物理锚定管线中。 Wetzstein 认为，视频世界模型提供了解除物理仿真迁移差距并吸收全网海量视频知识的潜在路径，但它固有生成幻觉的特性要求系统必须依靠物理传感器的持续闭环重锚定。当前该领域仍受限于极其高昂的预训练算力开销、推理延迟、VAE 空间物理表征损失以及长上下文处理效率；构建真正可靠的物理 AI 实体，仍有待在模型高效蒸馏、动作解耦与多模态传感融合等方面取得进一步突破。

[0:10] Gordon Wetzstein: The next frontier is physical AI where we bring a lot of these capabilities into the physical world and world models are expected to play a major role in that. [3:17] Gordon Wetzstein: The world model has actually many disadvantages compared to traditional simulators. It doesn't have an explicit 3D state. [16:53] Gordon Wetzstein: A video world model is a real-time action-conditioned causal or autoregressive video generator. [44:23] Gordon Wetzstein: No matter how long your context is you want it to be longer for many reasons.

[0:10] Um, it's great to see you all here. I [0:12] think LLMs and Agentic AI has had a [0:15] profound impact on all of our lives and [0:18] will have continue to do so. The next [0:20] frontier is physical AI where we bring a [0:23] lot of these capabilities into the [0:24] physical world and world models are [0:26] expected to play a major role in that. [0:28] So that's what I'd like to talk about [0:31] today. [0:32] Here's an example of a world model. This [0:35] is Google DeepMind's model called [0:37] Genie3. Uh you might think that this is [0:40] a video game, but it's actually a video [0:42] generator. It's a generative AI model [0:45] that generates all of these frames in [0:48] real time on the fly and it's [0:50] conditioned on actions. In this case, [0:52] the actions are these arrow keys that [0:54] you can see on the lower left. So, it's [0:56] kind of like a game engine, but it's [0:58] unlike any game engine that you know [1:00] because there's no 3D representation of [1:02] a scene. It's just a generated video [1:04] that is conditioned on a past. I'm going [1:06] to spend quite a bit of time in this [1:08] talk on just reviewing the fundamentals [1:10] of these models to help people with [1:12] diverse backgrounds just understand [1:14] what's going on in the space. [1:17] Genie is one of many models over the [1:20] last year or two. uh a variety of [1:23] different companies have taken on um [1:26] this world model idea. Um automotive [1:29] companies like Whimo and Tesla have [1:31] publicly talked about it in blog [1:32] articles so you can read up or public [1:34] talks. The main examples that they use [1:37] to motivate the their use of world [1:39] models is for data augmentation. For [1:42] example, Tesla has been talking about [1:44] using this video world model to generate [1:47] events that they didn't observe. So for [1:50] data augmentation they want to create [1:53] rare events. Uh this is what you see in [1:56] way's blog post as well. So the rare [1:58] event here is seeing an elephant on the [2:00] street. It's not something that they've [2:02] observed in that environment that but [2:03] they can generate that event and that [2:06] makes the training of their policies [2:07] more robust. Okay. So data augmentation [2:10] is a big thing and a world model can be [2:12] just videos but it can also be including [2:14] other modalities like you can see in [2:16] Whimo. There's point clouds. there's [2:18] multiple different camera observations [2:19] into different directions and so on and [2:22] so forth. So these models are trained on [2:25] well at least in in the case of Tesla on [2:27] a large amount of video data that is [2:29] captured by the your Tesla. You know all [2:32] the Teslas that are driving around [2:34] record data all the time that's fed back [2:36] into uh the the headquarters and being [2:39] fed into this video model. There's many [2:42] other examples. Uh, World Labs, um, Pepe [2:45] Le startup, in case you missed it, was [2:47] just acquired for eight billion dollars [2:49] two days ago by AMD. Emmy Labs is young [2:52] lun startup. They raised a billion [2:54] dollars as well. Nvidia is putting a lot [2:56] of money into Cosmos, that's their world [2:59] model uh, for the age of physical AI. [3:03] And there's like literally dozens of [3:05] startups right now raising a lot of [3:07] capital um, you know, in this space. So [3:10] it's a hot topic and I hope to shed a [3:13] little bit of light on what's behind [3:14] some of that at least on the technology [3:17] side. So a world model you can think [3:20] about as a natural progression from a [3:22] traditional simulator to something that [3:24] is learned from data. And I've been [3:26] working in the field of graphics, [3:28] computer graphics, computer vision for [3:29] about 20 years. And what we've been [3:31] doing in most of this time is we have a [3:34] state dynamics engine. So that's a [3:36] physics engine, but we keep track of the [3:37] state. And the state could be the mass [3:39] of an object, the friction of the floor, [3:42] you know, the velocity of an object [3:43] flying through the air. So these are all [3:45] states. And then you run a state [3:47] dynamics engine like a physics engine [3:49] that advances the state over time. And [3:51] then you have a graphics engine that [3:52] sort of projects the whole world into [3:54] pixels that you can render. So that [3:56] describes your video game, uh, visual [3:59] effects industry, and many other things. [4:01] So now we learn all this from data. And [4:04] it's good to think about, you know, what [4:08] these are good at because they might be [4:10] complimentary to some extent. [4:12] Traditional simulators have an explicit [4:15] representation of the 3D world. They [4:17] have a 3D model of the environment and [4:19] also of the state. So all the things [4:21] that I just mentioned, mass, friction, [4:23] and so on and so forth. But also, if you [4:24] have massive multiplayer game with [4:26] millions of players online, you know, [4:28] where are each of these players and so [4:30] on? These are all states that need to be [4:31] stored somewhere. Um, traditional [4:34] simulators are interpretable, editable, [4:37] they're compact to some extent. Usually, [4:39] they offer real-time frame rates. But [4:42] there is a simto toreal gap. And the sim [4:44] toreal gap means that whatever output [4:46] you produce with a traditional simulator [4:49] is at least at real-time frame rates is [4:51] typically not photorealistic. It doesn't [4:53] mimic exactly the complexity of the real [4:56] world. And that is true for both the [4:58] appearance and also for the dynamics. So [5:00] just because you have a rigid body [5:03] physics engine doesn't mean you can [5:05] simulate complex fluid flows through [5:07] Navier Stokes equations in real time. [5:10] Okay. [5:11] The world model has actually many [5:13] disadvantages compared to traditional [5:15] simulators. It doesn't have an explicit [5:17] 3D state. Typically it doesn't have a [5:20] state representation by default. Uh so [5:22] we don't we can't really access that. We [5:24] can't edit it. Not be interpretable and [5:27] editable. uh real time is possible and [5:30] there are quite a few approaches that do [5:31] that in real time but it is not trivial [5:34] to achieve because you have a very large [5:35] model that has to run at these frame [5:38] rates but it the promise is that you get [5:41] photorealistic quality and also [5:43] dynamics. So two different things the [5:45] appearance the pixels look [5:47] photorealistic in very complicated [5:48] environments and you get this realistic [5:53] behavior of objects and people. So [5:56] applications of simulators either [5:58] learned or traditional include training [6:00] and evaluation for physical AI. So you [6:02] can put a robot in there and it can be [6:04] trained be a self-driving car also, [6:06] right? It can interact with this world [6:08] in any way it wants and it observes [6:10] realistic or at least plausible [6:12] responses visually. So you can train [6:14] your robot in a simulated environment. [6:17] Uh you can also play it as a human. So [6:19] that could be a game engine or it can be [6:21] a training environment for real people. [6:23] Say you train astronauts to you know do [6:26] maintenance on the international space [6:28] station. It's very difficult or costly [6:29] or even impossible to train people on [6:31] the real thing. So training people in [6:33] simulation oftentimes is a good idea. [6:35] And then as I was saying before you can [6:37] use it as a data generator to train [6:39] models uh based on generated data. You [6:42] can balance your uh pre-training data [6:45] distribution and other things. All [6:47] right. So now I want to take a step back [6:49] and just walk you through some of the [6:51] fundamental ideas behind diffusion [6:54] models or flow matching models. I'm [6:55] going to use these terms [6:56] interchangeably. I'm just going to say [6:58] diffusion model, but it can be any [7:00] variant of a diffusion model. So [7:02] diffusion model is the type of [7:03] generative model that is fueling most of [7:05] these things. It's not an LLM. It's not [7:07] an auto reggressive next token [7:09] predictor. It is a diffusion model. And [7:12] what is a diffusion model? a diffusion [7:14] model. The goal is to generate [7:17] new images. I'm going to use the example [7:18] of an image and then I'll get back to [7:20] the video case in a second. So the idea [7:22] is to generate new images from a given [7:25] distribution of images. Okay, so maybe I [7:28] have n samples of real images. Can I [7:32] learn a model that cannot only sample [7:34] and interpolate between them but create [7:36] entirely new images that have a [7:39] compositional nature that can mix and [7:41] match different components in a [7:42] photorealistic way. So the early [7:44] examples that people showed with Dolly [7:46] and open early models were a astronaut [7:49] riding a horse on the moon. So the model [7:52] has seen horse many horses many [7:54] astronauts and many photos of the moon [7:56] but it's never seen these three concepts [7:58] together. So that was sort of like the [7:59] early excitement that we saw just a few [8:01] years ago. Now the thing is from various [8:04] reasons it's really difficult to sample [8:07] this distribution. It's I don't want to [8:09] say impossible but it's very very [8:11] costly. So we can't sample this [8:12] distribution directly. We can't just say [8:13] oh give me some new image and whoops [8:16] it's there. Okay. So this is a whole [8:19] lecture on why this is difficult and [8:21] Stefano Armon actually did a great job [8:23] in our previous seminar series and going [8:25] through all the details. So I I can [8:26] refer you to that but it's just [8:28] challenger. So what we're going to do [8:29] instead is we're going to take a [8:30] different distribution that is very easy [8:32] to sample from. And that distribution is [8:35] random Gaussian noise. We can easily [8:37] generate a sample of noise. Uh and it's [8:40] not going to be very useful but we can [8:42] sample it efficiently. And then what [8:44] we're going to learn is we're going to [8:45] learn the flow from the noise [8:48] distribution to the data distribution. [8:50] Okay. So that's the part that we're [8:51] going to learn. And the idea then at [8:54] inference time is that we sample noise. [8:57] We follow this learn trajectory that [8:58] takes us from the noise sample to a data [9:01] sample and that allows us to then [9:03] generate a new sample. So we indirectly [9:05] sample this distribution. [9:08] Now what is the scale of the data that [9:11] people use for training these models? [9:13] Well in an academic setting we use still [9:15] imageet. That's like just over a million [9:18] images with text captions or categories. [9:21] And that is very small scale, very low [9:24] resolution. It's great for academic [9:25] projects, but that's not what fuels [9:27] state-of-the-art image generators. A few [9:30] years ago, like we saw people using Lion [9:32] 5B or at least a subset of this. This is [9:34] billions of images. Okay, I don't want [9:36] to say it's all the five billion images. [9:38] They probably use a subset. And these [9:40] are open source models like stable [9:41] diffusions that came out a few years [9:43] ago. So, it's unclear what exact data [9:47] your latest and greatest image [9:49] generator, Nano Banana for Google or [9:51] OpenAI's image gen or whatever uses, but [9:55] I I'm sure that it's billions of very [9:57] nice filtered images that are carefully [10:00] curated. Okay? And so, the data is a big [10:04] part of the recipe for success here. And [10:06] these are billions of images. So, then [10:08] the question is, what have you not seen [10:10] in these billions of images? who's [10:12] probably seen pretty much anything that [10:13] people can come up with, right? [10:16] All right. So, now how do you train this [10:17] model? Um, we said we want to learn the [10:20] mapping from noise to data, but we're [10:23] given the data, the clean data, and it's [10:25] easy to add noise to clean data. We just [10:27] take an image like the one with the [10:28] butterfly over there. We're going to [10:30] call this X knot or X0. And then we can [10:33] just add noise. And we call this the [10:35] forward diffusion process. It's just [10:37] adding noise to an image. uh in fact [10:39] there are many different formulations [10:41] for this forward diffusion process but [10:43] the one that is very commonly used [10:45] nowadays is just a linear interpolation [10:47] between the noise and the data okay so [10:49] that lets me generate a noisy image with [10:52] an arbitrary noise level that is here [10:53] denoted as top [10:56] so what we're then going to learn is the [10:58] reverse the reverse diffusion so [11:01] starting from a noisy sample we want to [11:04] estimate what's called the velocity and [11:07] this is the rate of change along the [11:09] trajectory from noised data. Um, and [11:12] this is a large neural network that [11:14] we're going to train. The loss is [11:15] trivial. It's an L2 loss. We're going to [11:17] train this velocity field that takes us [11:19] from any noisy sample to somewhere that [11:22] is less noisy. Okay, so I hope this [11:24] makes a little bit of sense. So you [11:26] start with a noise and you iteratively [11:28] den noiseise the image. You always get [11:30] something that's slightly less noisy. [11:32] You're basically training a den noiser [11:33] to some extent a den the noising network [11:35] but it doesn't [11:37] necessarily predict the noise or the [11:39] clean image directly. It usually [11:42] predicts the velocity that points [11:43] towards you know that that that shows [11:46] you the rate of change along the path. [11:47] So these are some nuances and you can [11:49] forget about it but it's pointing in the [11:51] right direction. Um oftent times we [11:54] can't work with pixels directly at least [11:57] not for high resolution. So, we're going [11:58] to work in a latent space. And what that [12:01] means is that we have a variational [12:02] autoenccoder, a VAE, that takes a full [12:06] resolution pixel image, and turns it [12:08] into a much lower resolution latent. And [12:10] then all this diffusion process that I [12:12] just described usually happens in this [12:13] latent space and not on the pixels [12:15] directly. So, I'm going to skip that [12:17] detail, but that's what's usually [12:19] happening in practice. [12:22] Okay. And now I already showed you that [12:24] we're going to have this neural network [12:27] and it [clears throat] takes as input a [12:28] noisy frame. It gives you as the output [12:31] a slightly less noisy frame and we're [12:33] going to apply this over and over again [12:35] until we get a clean image conditioned [12:37] on the diffusion step. So the noise [12:39] level if you want and most modern large [12:42] scale image generators use a neural [12:45] network architecture that we know as [12:47] transformer. So this is the almost the [12:49] same as what we use for large language [12:50] models. a pretty standard scalable [12:52] network architecture. It's going to do [12:54] this denoising. But the interesting [12:56] thing is that we want this to be a [12:58] conditional process. So we want to not [13:01] just unconditionally sample from the [13:03] distribution, but we want to specify [13:05] something about the image that we want [13:06] to generate. We just don't want to [13:08] generate random images. So what we're [13:10] going to do is we and then we want to do [13:12] video. Okay. So the conditioning signal [13:15] is fairly straightforward to understand. [13:17] We want to describe the image in text. [13:20] For example, I want the astronaut riding [13:22] a horse on the moon. That is my text [13:24] description of the image. So that's the [13:26] conditioning signal. And I feed this [13:29] conditioning signal into the network by [13:31] first sending it through a text encoder [13:33] there. There's a different neural [13:35] network that just gives me some [13:36] embeddings for the text. And that's [13:38] actually then fed through cross [13:40] attention or some other mechanism into [13:42] the diffusion transformer [13:45] video generation especially when it's [13:47] auto reggressive. So next frame [13:49] predictor, I also have to condition this [13:52] model on the past couple of frames that [13:53] I've already generated so that it's [13:55] consistent with what it's already [13:56] generated. I can't just generate each [13:58] frame independently. I need to generate [14:00] it knowing what I've already generated [14:02] in the past. So this is the same idea as [14:04] an LLMs where I predict the next token [14:07] based on the previous tokens that I had. [14:09] Okay. And so this idea of context frames [14:12] or context is very important. In this [14:14] case, it's images, the last couple of [14:16] images that happened in the past. And [14:18] these are typically conditioned through [14:19] the KV cache. So the KV cache idea is [14:22] something very important for auto [14:24] reggressive or causal video generation [14:26] model as well. Now when I talk about [14:29] video world [clears throat] models, this [14:31] is just a world model that is [14:34] pre-trained on video data, webcale video [14:36] data. So same idea as the images. I'm [14:39] going to take, you know, hundreds and [14:41] thousands and millions of years of [14:43] videos of random stuff from the internet [14:45] and I'm going to train my next frame [14:48] predictor uh using it this diffusion [14:50] model. There are many different stages [14:52] in training such a model, mid-training, [14:55] post- training and so on and so forth. [14:56] I'm not going to go through all these [14:57] different stages, but at pre-training, [14:59] it's trained on videos. You leave out a [15:01] couple of frames and taking the previous [15:03] frames, you learn to predict those [15:05] frames using the diffusion process I [15:06] just described. And then at inference [15:09] time, as I was mentioning, [15:11] the model is conditioned on a number of [15:13] different things. It could be control [15:16] signals such as the identity of a person [15:18] that you want to see in this model based [15:20] on a photo, a map of the environment [15:23] that you want to navigate in, the motion [15:26] of a person that's maybe already [15:28] pre-recorded for motion capture or [15:30] that's also generated. The context [15:33] frames are the last couple of frames [15:35] that I already generated. And then the [15:37] actions. So the actions could be [15:39] navigation actions like these arrow keys [15:41] that we saw in Genie earlier. But it [15:43] could really be anything. Could be a [15:45] text prompt. It could be a tracked [15:47] signal from a human person. So you could [15:50] track your brain waves with EEG, your [15:53] heartbeat, your gaze position with an [15:56] eye tracker or really anything and use [15:58] that to condition the model. Or your [15:59] tracked hands. Think about VR. you put [16:02] on a VR headset and the whole world is [16:04] generated, not rendered from a [16:07] game engine. So then you want to be able [16:09] to touch things in that world. So you [16:11] need to track your real hand and [16:12] condition the video generator on your [16:14] tracked hands or on your head pose. [16:16] Right? [16:18] All right. And then you generate the [16:19] next frame or the next couple of frames. [16:23] That's a world model. Now people use the [16:25] term world action model or wham. That's [16:28] a big term right now. And a wham will [16:32] then also predict not only the future [16:35] frames but the future actions that an [16:38] agent would be taking. So the robot [16:41] could imagine what the future looks like [16:43] and then derive the actions from that. [16:44] And I'll show you some examples of that [16:46] later. This is not the focus of my talk. [16:48] So I I'm just going to touch upon this [16:50] at the end a little bit and show you [16:51] some applications. [16:53] So that brings me to my definition. I'm [16:54] halfway through the talk and we're just [16:56] defining what we're actually talking [16:57] about. A video world model is a real [17:00] time action conditioned causal or auto [17:03] reggressive video generator. [17:06] And there are many problems and I'll [17:07] show you some cool research that we've [17:09] published in the last little while on [17:11] long context. You want the context [17:13] window to be longer and I'll show you [17:15] what happens when it it's not uh [17:18] effective and intuitive control [17:20] mechanisms. Uh drift that's also known [17:23] as exposure bias. I'll explain to you [17:25] what all of these things are in a [17:27] second. there are many open problems. So [17:28] this is a great uh area to work at. Uh [17:32] just a few clarifying terms, a couple of [17:34] different settings in which these world [17:35] models are being used. The one that I [17:38] have described and that I've shown with [17:40] Gen3 is what I call an openloop setting. [17:43] So you have a couple of generated frames [17:46] and you generate the next frame and then [17:47] the next frame moves into the context. [17:50] So whatever's in the context frames is [17:52] actually previously generated. So you're [17:54] going to have to deal with error [17:55] aggregation because every small error [17:57] that you make will compound over time. [17:59] Okay, this is actually very similar to [18:02] LLMs. In the world of LLMs, this is [18:04] known as exposure bias. So this is just [18:06] a problem in this open loop setting. If [18:09] you run such a model on a robot or on [18:12] some self-driving car, you're going to [18:13] have real cameras there. So you don't [18:16] need to use the generated images [18:19] necessarily of the future of what the [18:21] robot will likely do in the future. you [18:23] can actually replace it with the [18:24] egocentric view of the robot captured by [18:27] the camera. So you don't have to deal [18:28] with drift and that kind of stuff. So [18:31] this is a closed loop setup. You have a [18:33] robot, it looks at its own observations [18:35] physically. It generates the future of [18:37] what it has to do in order to satisfy a [18:39] task and then it performs these actions, [18:42] but then it captures new data and it's [18:43] going to use that in context. And so [18:46] that's much easier because you're [18:47] grounded. The real frames help you stay [18:50] grounded. But that's sort of like a [18:52] hybrid solution also where you have an [18:54] open loop setup like in a computer game [18:57] uh and it can just drift away. But if [18:59] you have a map for example or some kind [19:01] of other anchoring signal that is [19:03] persistent over time that can also help [19:05] you stay grounded. So as we move forward [19:08] through the talk I'll show you a couple [19:09] of example projects and they're going to [19:11] be in one or another of these [19:12] categories. So just keep that in mind. [19:15] So the first problem I want to talk [19:16] about is long context and I'll first [19:19] show you what happens when you don't [19:21] have a long context. So here's one of [19:23] the very first uh models video world [19:26] models that was done. It was called [19:28] Oasis was done by an undergrad here at [19:30] Stanford a couple years ago. It's [19:32] marketed by Deart but Julian started his [19:34] PhD [19:36] wrote this thing and it plays Minecraft. [19:38] It's all trained on Minecraft data and [19:40] the reason is that you can get a lot of [19:41] data from this game engine. And you can [19:44] see the actions on the bottom. The [19:45] actions are move, jump, run, invert, use [19:48] items, and so on and so forth. So many [19:50] of you have maybe played this game, but [19:52] there's many different actions that you [19:54] can take. And the video world model [19:55] allows you to take these actions in this [19:57] auto reggressively generated world. Now, [20:00] what happens when the user looks forward [20:04] and then looks up to the sky [20:07] and then looks back down? Well, the [20:09] world has completely changed. There's no [20:11] consistency, right? because we don't [20:13] have a 3D representation of the state. [20:16] Um, it looks plausible, looks like a [20:18] Minecraft world, but it's not what the [20:20] user has seen before. And the problem is [20:23] pretty straightforward. The video model [20:24] cannot do that because the context [20:26] window is too short. Say it remembers [20:29] the last four frames or something like [20:31] looks up and then looks back down. Now [20:34] the sky is in the context, but whatever [20:35] happened before is just outside of the [20:37] context. So the model simply cannot [20:39] remember. It doesn't know what was [20:40] there. it will sample a new you know [20:43] sample from the distribution of [20:44] Minecraft images which will look [20:45] plausible but it has nothing to do what [20:47] with what it's seen before okay so why [20:51] don't we just train on longer context [20:53] windows because that's very expensive as [20:56] I mentioned before these video models [20:58] are based on transformer architectures [21:01] and if you're operating in this token [21:03] limited budget the number of tokens is [21:06] going to be the limiting factor so in [21:07] the self attention mechanism of the [21:09] transformer we have the softmax that [21:12] that includes the queries, keys and [21:14] values and the long story short now as [21:18] you increase the number of tokens or the [21:20] context that goes in uh the [21:23] computational complexity grows [21:24] quadratically during training time and [21:27] even during inference time it's going to [21:28] grow linearly. So the more tokens you [21:30] put in it's going to get quadratically [21:33] more computationally expensive to do [21:35] that. So there's an upper bound on what [21:36] you can actually do. [21:39] Now that might not be a huge problem for [21:41] text. Say a single document is a [21:43] thousand tokens or something like that. [21:46] For video models, this scales pretty [21:48] quickly. So even for a pretty low [21:50] resolution video model that operates at [21:51] 512 x 512 resolution with 32 frames per [21:55] second or 10-second video clip, even [21:57] with compression in the VAE and using a [21:59] latent space, you're going to be up to [22:01] hundreds of thousands of tokens within [22:03] just a few seconds. Okay, so that's very [22:05] challenging. Now, state-of-the-art [22:07] frontier models will have a context [22:10] length of a million tokens or maybe I [22:12] think the latest models have even two [22:13] million tokens, right? So, you see that [22:15] in your little chatbt window. Okay, [22:17] that's great. But that gives you maybe [22:18] one second longer context like we need [22:21] context lengths that are well minutes, [22:24] hours, years, whatever. So, okay, [22:27] question [22:37] Great question. Slightly outside the [22:39] scope of my talk. I will be happy to [22:41] hang around answer questions for as long [22:43] as you want after the talk and it's a [22:44] very important question. Uh typically [22:47] you're going to have a VLM and a [22:49] diffusion model in these omni models. [22:51] The question is how do they communicate [22:53] with one another? There's a great [22:54] question. I'm just going to focus on [22:55] just the diffusion model part for the [22:57] rest of this talk. Okay. [22:59] Okay. So long context very challenging. [23:02] So humans are great at remembering [23:05] things. At least some of us are good at [23:07] remembering some things. We can remember [23:09] things from a long time ago. So you know [23:12] humans can remember a lot of things. And [23:14] then what I often like to do is I like [23:16] to go and knock on Brian's door or other [23:19] people who know a lot more about the [23:20] human brain and perception than I and [23:23] think about okay how do humans do this? [23:25] How did they how did evolution come up [23:27] with a solution for this problem? And [23:30] interestingly, [23:32] human memory is sort of like a mixture [23:34] of expert systems. There are different [23:36] systems that are really good at [23:37] different things in memory. For example, [23:40] the visual memory operates in the visual [23:42] cortex and in other parts, spatial [23:44] memory operates in different parts of [23:46] the brain. [23:48] There's also different time scales at [23:50] which different parts of the memory [23:52] system operate. There's the working [23:54] memory that operates in the time scale [23:55] of seconds, short and long-term memory [23:58] that operates in the time scale of know [24:00] minutes, hours, days or even decades. [24:03] And then these are, you know, subdivided [24:05] into explicit, implicit memories, [24:07] semantic and episodic memories. There [24:09] many different types of, you know, [24:11] systems that are good at remembering [24:13] something specific. And so now what we [24:16] could do is sort of explore a couple of [24:18] these concepts and think about how we [24:20] can bake that into a neural network [24:21] architecture. U maybe not to mimic [24:24] exactly what humans do but to find an an [24:27] analogy perhaps that might help us with [24:29] our problems. And so this brings me to [24:32] our first uh project that was a very [24:34] simple idea let and he came up with this [24:37] idea of frame pack and that is why don't [24:39] we take why do we have to remember every [24:41] frame in the history at full resolution [24:44] and sample at every frame in time. Why [24:46] don't we just you know subsample in [24:48] space and time the the past and pack it [24:51] all in an efficient structure that the [24:53] CUDA kernel can actually process really [24:55] efficiently. And that was sort of like [24:57] the the idea behind frame pack where he [24:59] said that things that happened farther [25:01] in the past will likely be we can [25:04] remember them at a lower spatial [25:06] resolution and we can also subsample the [25:08] frames. So we can just sort of [25:09] downsample spatially and temporally over [25:11] time and pack it all into one higher [25:13] resolution image that can be efficiently [25:15] processed by the diffusion transformer. [25:18] And this was a long time ago in the year [25:21] 2025. A lot of things happened very [25:24] quickly in this field. And at that time [25:26] even short video generators were [25:28] completely changing the facial identity [25:30] of people or characters that were [25:33] generated. And using this technique of [25:35] context packing uh Lumi was able to show [25:37] that you know we can get really nice [25:40] short-term consistency for these types [25:42] of characters. So it was a really clever [25:44] and simple idea for you know extending [25:47] the context length. [25:49] We looked into a couple other ideas as [25:51] well. uh for example this idea of [25:53] explicit memory and spatial memory was [25:55] explored in a paper that we also [25:57] published with my postto at Europs last [26:00] year and the idea there was that we [26:02] built a next frame predictor um but [26:05] instead of just working on RGB we also [26:08] want to have a spatial memory something [26:10] that remembers the spatial structure [26:13] that likely doesn't change over time so [26:15] remembering everything and how it moves [26:16] is hard because you need to remember [26:18] everything in 4D space and time but if [26:21] you just remember the 3D nature of the [26:23] scene. So that you know if I go to the [26:25] the Hoover Tower, I come over here and I [26:27] go back to the Hoover Tower, at least [26:29] the Hoover Tower still looks the same. I [26:30] don't expect every leaf to still be the [26:32] same. And in fact, I wouldn't even be [26:33] able to recognize if leaves have changed [26:35] or people that walk by. They don't have [26:37] to be remembered, but the spatial [26:38] structures have to be remembered. So the [26:41] way we implemented that in this project [26:42] was we not only generated RGB for each [26:46] frame, but also a depth map that can be [26:48] just extracted with other models. This [26:50] depth map was then fused into a globally [26:52] consistent point cloud. And so we're [26:55] basically building up a 3D map being [26:57] represented by a 3D point cloud over [26:59] time. So we don't know the environment [27:01] when we start out, but as the agent [27:02] walks around the room, we're just going [27:04] to keep building up this 3D map. And [27:06] then we're going to render a point cloud [27:07] rendering of this map from whatever [27:09] position we're at and use that to [27:11] condition the model. So when we revisit [27:13] a place that we visited before, it's [27:15] already specified at least the spatial [27:17] structure of that. But when we go to a [27:19] new place, it's not conditioned on that [27:21] because it's just going to be generated. [27:24] We also remember a couple key frames and [27:26] there was a pretty simple idea that you [27:28] know is now adopted by actually a lot of [27:30] different systems like this idea of [27:31] using a map building up the map along [27:33] the way is this idea of this anchored [27:35] open loop generation that I mentioned [27:37] earlier. [27:39] All right, let's talk about multiple [27:41] agents operating in this world. So [27:43] hopefully I've convinced you by now that [27:45] diffusion models are real-time game [27:46] engines. And what I'd like to talk about [27:49] next is like how do you turn that into a [27:51] multiplayer experience. So this is [27:54] footage from the game Doom, you know, [27:57] sort of like 1990s when I was a teenager [28:00] and played computer games. These are the [28:02] types of games that we played. And the [28:06] the re I'm not embracing violence or [28:08] first-person shooters in this talk, but [28:11] it is very easy to generate a lot of [28:13] data using the Doom game engine that is [28:15] freely available online. So the only [28:17] reason why we use this is because it was [28:19] the easiest way to generate a lot of [28:21] data. Okay. So we basically built a [28:24] multiplayer game engine and we what we [28:26] did is we used this idea of a map. Most [28:28] computer games have a map already, a [28:30] mini map. Usually that's 2D. uh we can [28:32] design the maps manually or generate [28:35] them. And then what we do is we [28:37] condition the video model on this map. [28:39] And the way we do that is we extrude all [28:41] the walls up straight up. We position [28:44] the player in on the map and we render a [28:48] depth map of the map or disparity map [28:51] that is used to condition our next frame [28:53] predictor along with the context frames [28:55] and turning noise into frames as I said [28:58] before and also with these actions. So I [29:00] want to be able to navigate with my [29:02] agent. So now whatever actions I'm [29:05] taking in this case it's basically [29:06] navigation and shooting go into this [29:08] model and I generate the next frame. But [29:11] the it's going to be consistent over [29:13] time based on the usage of this map. And [29:15] then we need to update the that state as [29:17] well. So we do that with a separate [29:20] neural network here. Uh I can talk about [29:22] the details later but we need to update [29:23] the globally consistent state for all [29:25] players and that is also stored on the [29:27] map. [29:29] Okay. So when a single player walks [29:31] through a level kind of looks like this, [29:33] the geometry of the scene is always [29:35] going to be consistent. You can always [29:36] go back to places, but the texture isn't [29:39] necessarily going to be the same because [29:40] that's only coming from the context [29:42] window. Okay? Only the global 3D [29:45] structure, not the appearance, not the [29:47] texture will be actually consistent. But [29:49] you know what? I bet you that you [29:51] wouldn't even notice if the tile on the [29:53] wall is slightly less brown than the [29:54] last time you saw it. [29:56] Okay, so now for multiple players, we [29:59] basically just store the location and [30:01] orientation of each player on the map as [30:03] well as their state, which is dead or [30:05] alive. And then that is all conditioned [30:08] into for each player. So each player can [30:10] actually play the game completely [30:11] separately. The only thing that needs to [30:13] be shared is the map. [30:16] And so here we have a video of a couple [30:18] lab members playing this game for I [30:19] think 30 minutes straight. There's no [30:21] drift. It works really well. It was [30:24] hosted on a server. So this is like [30:26] rendered actually on a couple pretty [30:28] beefy GPUs [30:30] and it was a lot of fun. We even hosted [30:32] this publicly online for about a week [30:34] through modal. So thanks Modal AI for [30:37] letting us host this and we had [30:39] thousands of players actually play this [30:41] game online. So using a map is a great [30:44] way of sharing state across multiple [30:46] players. [30:47] Let me talk about controllability [30:49] because that's an important aspect of [30:51] this. [30:52] We've worked on a whole bunch of [30:54] different ways of controlling these [30:55] video models. I just want to do a quick [30:57] fast forward of things that are [30:58] important to consider without [31:00] necessarily going into the technical [31:02] details. So one thing that you might [31:03] want to control is the camera pose. So [31:06] this is actually a imagetovideo [31:09] generation idea where you have the image [31:12] only the single image goes in and then [31:14] you generate this video but you control [31:16] the camera pose. So you want to say [31:17] exactly where you're looking and where [31:19] you're going and that's called camera [31:22] control and we sort of published this [31:24] idea with a paper called camera control [31:26] one and two. Uh also last year this is a [31:29] pretty standard feature now in a lot of [31:31] systems like this. [31:33] Uh more recently we worked on hand [31:35] control. So if you use this for VR where [31:38] you generate the entire experience which [31:41] I call generated reality uh you want to [31:44] condition on both the tracked head pose [31:46] the camera conditioning but also on the [31:48] hands in order to enable hand object [31:50] manipulation and interaction in the [31:53] generated world. So here you can see my [31:55] my master student basically is this is a [31:59] generated world. He can use these props. [32:00] He can pet the dog. the dog can react to [32:03] his hand or he can do many different [32:05] things, play a game with his hands. So [32:07] that's a pretty standard feature that [32:09] you'd want to have in a VR setting and [32:10] typically use controllers or tracked [32:12] hands for that, right? But you can also [32:14] integrate that into a video model. And [32:17] then another feature that we published [32:18] in a paper at CVPR earlier this year, we [32:21] call it bullet time actually space-time [32:23] control. So here we're seeing a couple [32:25] of examples where we saw the input [32:27] videos first. So these are just random [32:28] videos, but then we figured out a way [32:30] for not only controlling the camera, but [32:32] also stopping the time. So we can have [32:34] independent control knobs of the the [32:37] world time inside the scene and the [32:39] camera. So I don't know how many of you [32:41] remember the movie The Matrix, but there [32:43] was this famous scene that is called the [32:45] bullet time where the camera sort of [32:48] moves around Neil as he's shooting and [32:50] dodging all these bullets, right? So we [32:52] can do this actually with real videos. [32:54] you can be uh in that matrix movie now [32:57] or your dog or your cat or whatever. Um [32:59] but this requires independent control [33:01] over the global time in the scene and [33:03] the camera itself which is usually [33:04] coupled. [33:06] So I'm I'm mentioning a couple of these [33:08] space-time control and camera control [33:10] features here because these are all [33:12] features that we've seen in some of the [33:13] very popular video generation models uh [33:15] that are commercially available such as [33:17] WorldLaps's Adl. So there's space-time [33:20] control, there's camera control, there's [33:22] a couple of these things and these all [33:23] based on ideas like that. [33:26] All right. So last topic that I want to [33:29] cover that we've been doing in the lab [33:30] is drift. So as you're autogressively [33:33] rolling out this video generator, you [33:36] know, bad things happen because you [33:37] always make small mistakes when you [33:39] hallucinate something, you generate [33:40] something. So if you just keep rolling [33:42] out this auto reggressive video [33:44] generation, uh strange things happen as [33:47] you can see here. So the motion of the [33:49] scene gets weird. Colors get saturated. [33:52] Objects start looking really strangely. [33:56] And the reason for that is simple. [33:58] During training, the model is always [34:01] trained on clean frames that are coming [34:03] from some real video in the context. But [34:06] during inference, the past frames are [34:08] actually generated and they become [34:09] progressively worse because they they [34:11] are not actually from the same [34:12] distribution as what you've trained on. [34:14] So this is known as exposure bias also [34:17] in LLMs. Um so the result is that it [34:20] accumulates errors. And so in robotics [34:23] there's a very simple method that has [34:25] been around for a long time called data [34:27] set aggregation or dagger. And that [34:30] works as follows. Let's say you have a [34:31] self-driving car and the car drives on [34:33] the street and every once in a while it [34:35] drifts over. Uh what are you going to [34:37] do? You have a remote hello operation [34:39] person dial in and say, "Oh, something [34:41] bad has happened. I'm gonna correct uh [34:44] what's happening here. I'm gonna steer [34:47] the car back into the middle of the [34:48] lane. So, you have these expert [34:50] demonstrations that help you recover [34:52] from errors. And so, we would like to do [34:55] the same thing. This is well established [34:56] method works like a charm. Um we would [34:59] like to do the same in video generation. [35:01] The question is just where do you get [35:02] these expert demonstrations? You can't [35:03] just have people manually correct bad [35:05] videos. So we came up with a super [35:07] simple idea which is we post-train the [35:10] model on you know degrading versions of [35:13] itself. So we just roll out the video [35:15] model you see these degradations for [35:17] example here by you know color [35:19] saturation in in the in the video and we [35:22] play them in reverse [35:24] um with a text prompt that says this is [35:27] played in reverse. So you wouldn't see a [35:28] fish swimming backwards. So we don't you [35:30] know that we fix that by the text [35:31] prompts but basically you get the [35:33] corrective errors by just playing the [35:35] videos in reverse we call this bagger or [35:37] backwards aggregation [35:40] and when we train the model this way [35:42] multiple rounds it actually learns to [35:43] correct itself because it's learn [35:45] because the degradation actually changes [35:47] every time you train the model and every [35:49] model is different uh and so on and so [35:51] forth. So you avoid color saturation you [35:54] avoid strange motion it works pretty [35:57] well. So here's a couple other methods [35:58] in case you're actually into this. So [36:00] there's different previous work history [36:02] guidance for example. You see color [36:05] saturating quite a bit on the very left [36:07] self-forcing the the colors always look [36:10] good because this uses a distribution [36:12] matching loss. So it makes sure that the [36:13] images always look good but then the [36:15] motion is affected and the motion is [36:16] very static and you see actually see [36:18] degradation in the motion. It's mostly [36:20] static whereas bagger actually allows us [36:22] to get good appearance and motion. [36:25] All right. I should say that many of [36:26] these academic projects are trained here [36:28] on Marlo. So we're really grateful to [36:30] have this. This is the interruption [36:31] where I'm going to do some [36:32] advertisement. Marlo is an amazing [36:34] resource here on campus. Uh it we need [36:37] it to do research like this. Uh we can't [36:40] do it without there's far too few too [36:44] few GPUs that we have right now. I think [36:46] this is a shared resource across all of [36:48] campus. And uh yeah, thank you to Marlo [36:51] and the team for helping us uh with [36:53] this. uh Craig and many others are in [36:55] the room. So yeah, we can't do this work [36:58] without Marlo. [37:00] All right, but now I'm going to talk a [37:01] little bit about uh these video action [37:03] models. So how do we use this for [37:05] robotics? And this is not work done here [37:07] in the lab. This is actually work done [37:08] in a startup that I'm affiliated with [37:10] called ROA. AI and this is sort of the [37:13] world agents part. So assuming that we [37:15] know everything that we've just heard [37:16] about video models, you know, how do we [37:18] use it to drive robots? [37:20] Okay, so what's driving a robot is a [37:23] policy and a policy typically closes the [37:27] perception action loop. So you perceive [37:29] your environment as the robot and then [37:31] you need to figure out what are the next [37:33] actions that you need to take in order [37:34] to complete a task. So that's the [37:37] observation to action loop. Now there's [37:39] multiple ways you can do this. You can [37:41] do this with a traditional simulator as [37:43] I mentioned earlier and a traditional [37:45] simulator is great because it gives you [37:47] lots of data for free. But the problem [37:49] is really the sim toreal gap. The [37:51] diversity of the data will never match [37:53] the complexity of the real world. And [37:55] that's part of the reason why a lot of [37:56] robots that are trained in a simulated [37:58] environment don't necessarily function [38:00] well in the real environment. It's lack [38:03] of diversity. So now what if you had a [38:05] perfect world model that acts as a [38:07] simulator? Well, you're going to get [38:09] lots of data. You don't have a simpal [38:11] gap in principle at least. But there is [38:14] no perfect world model quite yet that [38:16] isn't able to simulate everything you [38:18] want with the condition on every action [38:20] you can take. So it's going to be very [38:21] easy for the robot to break this. So [38:23] these world models aren't quite ready [38:24] yet. And maybe there's a hybrid world [38:26] where we can combine traditional [38:28] simulators and generation. That's [38:30] probably a good thing. And many other [38:31] ideas are out there but it's not quite [38:33] there yet. [38:34] So most people sort of you know just [38:37] train and why not train your policy on [38:40] all the data that was used to train the [38:41] video model in the first place because [38:42] it takes a lot of time to train the [38:43] video model and then if your only goal [38:45] is to train a policy after with that you [38:47] know why not just use the original data [38:49] in the first place. So most people use [38:51] this you know imitation or supervised [38:53] learning strategy where we have a lot of [38:55] videos with labeled actions and we're [38:57] going to train our robot policies on [38:58] this. This is sort of diffusion policy [39:00] for example. was also a diffusion model [39:02] that takes the last couple of [39:03] observations and the last couple of [39:04] actions of a robot in but the output is [39:06] not frames it's actions okay that's sort [39:09] of what most people in robotics do the [39:11] problem is that availability of data so [39:14] what's enabled the world to scale AI [39:17] systems is scalable network [39:18] architectures like transformer data and [39:22] a lot of GPUs but the data hasn't really [39:25] been there for robotics [39:27] I just this is actually from a couple [39:29] months ago I asked Gemini to tell me how [39:32] many action label video tokens are there [39:34] on the internet guess. And it gave me [39:37] something like this uh you know maybe [39:39] 2,000 hours or something like that. And [39:41] I asked it to give me a sense of how [39:43] many video tokens are there online on [39:44] all of the internet right this total [39:46] approximation without action labels and [39:49] said oh well we have two billion hours [39:51] of random videos online. So this [39:54] supervised learning approach basically [39:56] limits ourself to training on this [39:58] little red dot. And our idea was well [40:01] why don't we on the blue circle instead [40:04] and the blue circle here is not drawn to [40:05] scale. If I were to draw it to scale [40:07] with a little red dot it would have a [40:09] diameter of 60 meters. So just to give [40:11] you a sense of the order of magnitude we [40:13] have a lot of random videos. These could [40:15] be egocentric videos of people wearing [40:17] cameras filming themselves you know [40:18] cooking in the kitchen or something like [40:20] that right but it's not robot specific [40:22] data. [40:23] Okay. So, you know, it's a problem to [40:27] work on smallcale data because I think [40:31] the the motto of Silicon Valley open AI [40:33] and so on since 2020 at least was that [40:37] we need to scale. And the reason is that [40:39] we see emergent behaviors at a scale [40:42] that we just don't see otherwise. And [40:44] these emergent behaviors could be chain [40:46] of thought, reasoning, they could be in [40:48] context learning, lots of cool stuff [40:50] that you use every day with CHPT. They [40:53] only appear if you train a large enough [40:54] model on a large enough amount of data. [40:56] And that was sort of, you know, found in [40:58] the scaling laws for new language model [41:00] paper from 2020. So we need to sort of [41:02] think about robotics in the same way. [41:04] There's two ways of thinking about it. [41:05] either we make the blue dot larger and a [41:07] lot of companies are out there right now [41:09] capturing you know a lot of data for [41:11] robots or we just train on all the data [41:13] that we already have that isn't labeled [41:15] on on actions. Okay. So our idea was we [41:19] reformulate the problem as a video [41:21] generation problem where we say hey our [41:23] policy should match map observations [41:26] into future observations and then we can [41:29] derive the actions from that because [41:30] then we can train on all the unlabelled [41:32] videos and so mapping future [41:34] observations into actions that's an [41:36] inverse dynamics problem. If I visualize [41:39] myself doing a task in the future like [41:41] my goal is to grasp this object I can [41:44] just visualize mentally what what's [41:46] going to happen. my arm is going to move [41:47] there and I'm my hand's going to grasp [41:49] it. Okay, if I had the video of that, I [41:51] can it's actually trivial to get the [41:53] actions the endector pose and so on or [41:55] the hand or the robot arm from that [41:57] video. And we can train the video [42:01] on internet scale data, web scale data. [42:04] So we break it up into two sub problems. [42:06] The video generation problem, causal [42:08] video generation in real time action [42:10] condition and then the inverse dynamics [42:12] problem. And it turns out that the [42:13] inverse dynamics problem is very data [42:15] efficient. You don't actually need much [42:17] data to train an inverse dynamics model, [42:19] but you need a lot of data for this [42:20] video generation. And so you can read up [42:22] all the details on our blog post causal [42:24] video models or data efficient robot [42:26] policies that we put out earlier this [42:28] year. I mean, we started working on [42:29] video-based uh robotics about two years [42:32] ago and now everybody is doing this, [42:35] right? Like this is sort of what Cosmos [42:37] 3 all the world models out there all [42:39] video gener. [42:45] What we're doing is we're building [42:47] robots for industrial tasks and [42:49] logistics. So, they're doing really dumb [42:52] things like uh dumping out heavy objects [42:55] out of a box. And you can see them in [42:57] action here. Uh on the upper left is a [43:00] what's called a decanting task. Big [43:02] heavy box comes in uh in a factory. The [43:05] robot arms will auto autonomously open [43:07] the box. They will dump out these ball [43:09] bearings into a tote. They will sort [43:12] plastic parts and the wax paper that's [43:14] inside the box into the recycling bins [43:17] and so on and so forth. Really dumb [43:18] task. Nobody wants to do this. Uh but [43:20] now the robot can do this and this is a [43:21] customer defined task from a large [43:23] automotive manufacturer. Uh or in [43:26] agricultural uh facilities oftent times [43:28] agricultural products are actually being [43:30] carried around in these boxes. Somebody [43:32] needs to clean the boxes and break them [43:33] down after it's done. So now the robot [43:35] can do this. Here's a cool demo that was [43:40] sort of the cup game. It shows that we [43:41] have really long context actually. So, [43:43] can you solve this cup game without any [43:45] perception or tracking or anything like [43:47] that? So, you know, the person hides an [43:49] object under the cup, shovels them [43:51] around, and only if we have a long [43:53] enough context can you actually solve [43:54] this task. On the lower left, you see [43:56] the the video that's generated. The [43:58] robot imagines itself doing the task [44:00] successfully and then it executes the [44:02] action successfully. [44:04] Or returns processing. Say you order [44:07] garments on Amazon and you don't like [44:10] them, you return them. Somebody needs to [44:11] inspect these garments and pack them [44:13] back up to make sure that they're usable [44:15] again. So a robot can do this, but it's [44:18] going to be much more efficient in doing [44:19] it if it remembers what it's already [44:20] inspected before. [44:23] All right. So again, this is not work [44:24] that's done here, but it directly builds [44:26] on video world models where we use the [44:29] video model as the policy. Basically, [44:32] there's a lot of open challenges in this [44:34] space. Um it it it is getting pretty [44:37] crowded. So we have been working on this [44:38] for a few years already. So I can't say [44:40] hey this is you know the wild west right [44:43] now because there there's a lot of works [44:45] out there already and these are pretty [44:46] large models. So it's you need quite a [44:48] few GPUs to actually do anything useful [44:50] in this space but there are a lot of [44:52] open problems. So long context we need [44:55] longer context. This is a never- ending [44:57] question. No matter how long your [44:59] context is you want it to be longer for [45:00] many reasons. You want to remember 3D [45:02] structure. want to remember things that [45:04] you've done. It's good for extrapolating [45:07] dynamics. Um you need it for in context [45:10] learning. You need it for many different [45:11] things. Uh planning on reasoning. You [45:14] know VLM integration. There was a [45:16] question earlier. You know what what is [45:17] there a VLM? Uh yes you want the VLM to [45:20] be in there. The question is how do you [45:21] combine these two in the best way? Uh [45:24] maybe you want to do reasoning on the [45:25] diffusion model scale not necessarily in [45:28] the VLM. Most people are using reasoning [45:30] in the VLM right now. efficiency. These [45:33] models are large. You have many [45:34] diffusion steps. So there's a lot of [45:36] work on few step distillation. How can [45:39] you generate the images with one step or [45:41] two steps instead of 30 or 50? Uh token [45:44] efficient generation topics that a lot [45:46] of folks in my group are working on like [45:48] Brian over there and George. Um how do [45:51] you just, you know, use the tokens from [45:53] the context that actually matter? You [45:55] don't need to look at everything. Maybe [45:57] you just figure out what's important for [45:58] the tasks that you're trying to do and [46:00] focus on those. Uh so that's good [46:02] quantization and pruning model [46:03] architectures lots of stuff good stuff [46:05] out there. Uh representation learning [46:07] right now the VAE that we're working [46:09] with aren't really not great. Uh but [46:11] it's not clear how to get a scalable uh [46:14] you know approach for representation [46:16] learning that is not meant for [46:18] discriminative tasks but for generative [46:20] tasks right so that's an open question. [46:23] uh agentic world models. We've seen GPT6 [46:25] now using tools and closing the loop and [46:28] many cool examples like 3D scene [46:29] reconstruction using Blender stuff like [46:31] that. So how do we use GPT6 and aentic [46:34] AI as verifiers or in some kind of an [46:37] iterative loop in this in this world and [46:41] then world action models again it's not [46:42] actually entirely clear what the best [46:44] way of generating actions and [46:46] observations are or do you always need [46:48] observations you know can we just drop [46:50] them at inference time perhaps lots of [46:52] open questions make things more [46:53] efficient and and better [46:56] all right that brings me to the end of [46:58] the talk uh I think I'm a little bit [46:59] over time because we started a couple [47:01] minutes [47:02] Uh so I direct what's now called the [47:04] physical and spatial AI lab here at [47:07] Stanford. Um we work on all kinds of [47:10] things related to optics, imaging, XR, [47:13] LAR but mostly on the AI part driving it [47:16] but work on fundamentals and [47:17] applications and I just wanted to thank [47:20] all my students and posttos who've done [47:22] all the amazing work that I've shown you [47:23] today. Without them I wouldn't be able [47:25] to do anything. So thank you so much and [47:27] thanks to Marlo because without the GPUs [47:29] we also wouldn't be able to do anything. [47:30] Um thanks. [47:39] Okay. The question is how do we think [47:41] about time in a world model? I mean [47:43] typically the the naive way is just to [47:46] think about frame index as time. So when [47:48] a frame is five frames in the past it [47:51] has a index of minus five or something [47:53] like that. That's how the positional [47:54] embeddings are sort of created in the [47:56] tokens. That's how the model sees time. [47:59] it it knows what's spatially closed and [48:01] temporally closed based on those frame [48:03] on the frame index. But that's [48:05] insufficient if you actually want to [48:06] have control over the global time or [48:08] independent control over camera pose [48:10] time and frame index. So that that was [48:12] sort of a topic we explored in this [48:15] bullet time project where we said oh the [48:17] scene has to have a global time that we [48:19] need to be able to control independently [48:21] of the frame index for example. So uh [48:24] mostly it boils down to okay let's [48:26] define what we want to control. Let's [48:28] find an efficient control mechanisms. [48:30] How do we get this control signal into [48:32] the architecture? But then the biggest [48:34] question is usually how do we get the [48:35] training data to actually train a model [48:37] on that? Because it needs to have a lot [48:38] of examples where all of these variables [48:41] you want to control are observed [48:43] independently being changed and that [48:45] usually it boils down it's a data [48:47] problem. [48:50] Yeah. Yeah, that's a great question. So [48:52] how does the model know it's wrong? So [48:54] in the closed loop setup, you [48:57] imagine the future, but you imagine it a [49:00] little bit into the future, right? [49:01] You're you're planning out maybe a [49:03] second or you know a few frames, a chunk [49:06] into the future. You perform that action [49:08] and then you can have a verifier which [49:10] could be a VLM or perception model or [49:13] something else that sort of observes [49:14] what actually happened and is this in [49:16] line with what I actually want to [49:17] achieve. You can also roll out the you [49:20] can generate many plausible futures in [49:22] simulation first and then pick the one [49:24] that you actually want for example. So [49:27] there's I don't think there's a [49:28] principle way of telling whether an [49:32] physically executed action from the [49:33] model will actually be successful, safe [49:37] uh and even viable. It will just do it [49:42] right. But I think you know safety is a [49:44] big thing and a VLM at the highest level [49:47] you can have a VLM act as a verifier but [49:49] there are many more complicated [49:51] uh and better methods to doing that. [49:57] Oh yeah yeah so the question is like how [49:59] good are the base models and also like [50:01] maybe let's let me add a a nuance to [50:04] this. Pre-training is really hard. It [50:06] takes a lot of resources. So you need [50:09] you know it's incredibly hard. I can't I [50:12] can't tell you how hard it is because [50:14] you need to get the architecture right. [50:15] You need to get a lot of data. You need [50:17] to get the labels for the data and then [50:19] you have to have an enormous amount of [50:20] resources to train these models. And [50:22] once you've trained it, you can still [50:24] post train and do some mid training, but [50:26] that is more like massaging it towards [50:29] what you want. If you want a real-time [50:31] model, for example, because you're [50:32] working at a game engine company or [50:35] something like that, you know, taking a [50:37] very large 14 billion parameter [50:40] diffusion transformer, it's going to be [50:42] very hard, right? You like you can [50:44] distill the hell out of it and it's [50:45] still going to be slow and it needs a [50:47] beefy GPU. And that brought us down to [50:50] cost, latency, and cost. If you just [50:52] want to play a game, I mean, you're not [50:54] going to pay by the token, right? Like [50:55] you just want to play the game and it's [50:57] going to be have to be responsive. So [50:59] retraining is really hard. And I think [51:01] there are a few more open weight up [51:03] models out there. They're pretty good. I [51:04] mean, they've gotten a lot better. Like [51:06] two years ago, a year ago, we had [51:08] animate diff from animate div to Cosmos [51:11] 3 and one 2.2 and other state-of-the-art [51:15] models. I mean, we've gotten very far, [51:17] but I think there's always going to be [51:18] just a few models, and people seem to [51:20] have gotten a little bit more secretive [51:22] about their models. So, we've seen less [51:25] pure models to be open sourced also. Um, [51:29] they're big and they're getting [51:32] good. I don't think they're quite there [51:33] yet. That's what I was saying. Like I I [51:35] don't think I would train my robot that [51:37] I want to sell to a customer on any of [51:38] these existing models, right? It's not [51:41] quite there yet, but it's getting [51:43] closer. [51:45] Yeah. Okay. So, how's the point globally [51:47] consistent point cloud conditioned into [51:49] this what I called an anchored open loop [51:52] generator? Right. So, first of all, [51:55] you're right. The point cloud grows as [51:57] you're exploring the world. You get more [51:58] and more points and you want to merge [52:00] them. We did it pretty naively. We just [52:02] filter out the dynamic objects and then [52:04] we merged it into the point cloud. But [52:05] if you roll this out for an hour or a [52:08] year, you're going to get a lot of [52:09] points, right? And they need to be [52:10] stored somewhere. Now, so that's [52:12] something that we did not solve. Um, the [52:16] way we put this into the context is [52:18] actually we render the point cloud from [52:19] the perspective of the user and that [52:21] gives us an image of the point cloud [52:23] that has a bunch of splattered points. [52:25] That's very fast to do and it actually [52:27] doesn't require that much memory because [52:28] it's just an extra channel or three in [52:31] the context window and that's even [52:32] sparse. So it's pretty compressible if [52:35] you want and so [52:37] the context isn't necessarily the [52:39] problem but I think that the growing map [52:42] would probably be a problem. This is a [52:43] 3D map. People have explored Gaussians [52:46] and meshes and all kinds of other things [52:48] to represent these maps. Or you could do [52:49] it in 2D also that that's going to give [52:51] you a lot of savings and then just [52:52] extrude stuff. Yeah. So there's a lot of [52:55] tricks and a lot of papers have been [52:56] published on this topic for the last [52:58] year or so. [53:00] All right. Uh so hallucination is a is a [53:02] big problem. I mean the thing is a [53:03] generative model will always [53:04] hallucinate. That's what it's trained to [53:06] do, right? So when people say, "Oh, we [53:08] don't want the models to hallucinate." [53:10] That's what it's trained to do. Any [53:11] generative model will hallucinate. It'll [53:13] hallucinate something, but hopefully [53:15] it's plausible enough and close enough [53:16] to what you want that you know it does [53:19] the right thing. So I would say in the [53:22] context of robotics which is probably [53:24] the area that you're most interested in. [53:26] I mean I think staying grounded on the [53:27] physical observations that's sort of the [53:30] key. So you look at the observations [53:32] from your robot whether it's a drone or [53:34] a robot could be a wrist camera doing [53:37] manipulation or a headorn camera or [53:40] something else. You always generate just [53:42] a little bit into the future uh enough [53:45] to perform the next action and then [53:47] you're gonna roll this out in a loop. [53:49] Right? So you're not going to do [53:51] planning for like minutes or hours into [53:54] the future. I think that's best done by [53:55] the VLM. It's like task planning and you [53:58] know breaking up the high level tasks [54:00] into multiple steps and multiple [54:01] actions. But the video model is really [54:03] good at motion planning and planning the [54:04] next physical robot trajectory that's [54:07] going to succeed in grasing this object. [54:10] But what comes after it doesn't even [54:11] know that, right? So I think staying [54:14] grounded on physical observations is is [54:16] a very important. And it's not just [54:18] visual observations. It could be tactile [54:20] sensors or audio or other signals too. [54:22] So I didn't talk about this but that can [54:24] be useful.