World Labs Unveils Atlas 3D AI Model as Co-Founders Target Robotics

World Labs推出Atlas 3D AI模型，联合创始人瞄准机器人领域 World Labs认为，预测全新相机视角为空间智能提供了基础原语，不仅能替代密集的3D扫描，还可为机器人与设计实现实时模拟。 World Labs联合创始人推出了以“新视角预测”为核心的基础世界模型Atlas，该模型将生成式合成与3D重建整合至单一架构中。Atlas突破了传统的next-token或下一帧生成模式，能够接收稀疏的现实世界图像与相机轨迹，从而构建具备真实空间依据的确定性3D漫游与模拟。该模型大幅减轻了创作者的采集负担，为机器人训练提供了高保真度的环境随机化，标志着迈向完全空间智能的关键一步。团队最终认为，预测全新视角对于3D物理世界建模的基础性作用，正如next-token预测对于语言模型一样关键。 World Labs 正式发布新一代世界模型 Atlas，将“新视角预测”（New View Prediction）确立为核心基元，首次在单一原生多模态架构中融合了视觉生成与 3D 几何重建。 (0:51) 不同于传统视频生成模型的随机幻觉，Atlas 将输入帧直接与显式 3D 相机位姿绑定，把原本需要数百张照片的密集扫描捕捉降维至 3 到 25 张稀疏图像，并能自动补全未拍摄到的遮挡死角。 (3:51) 团队将 3D 重建重构为“长上下文窗口”生成问题；早期测试模型在经典测试场景中精准脑补出花园餐桌下被遮挡的足球，成为证实该架构可行性的关键突破时刻。 (16:55) 结合对 Scenix 的收购，Atlas 计划加速机器人领域的 Real-to-Sim 与 Sim-to-Real 闭环，通过构建具备动力学推理能力的神经模拟器来缓解物理世界动作训练的数据短缺问题。 (29:10) 李飞飞与团队认为，正如 Next-Token 预测驱动了大语言模型，新视角预测源于生物应对运动的视觉进化，是理解与模拟物理世界且具备 AI 完备性的核心基元。 (42:42)

## Background 本期视频由著名风险投资机构 a16z 于 2026 年 9 月 4 日发布在 YouTube 频道，记录了 a16z 普通合伙人 Martin Casado 与 World Labs 联合创始人李飞飞（Fei-Fei Li）、Justin Johnson 以及 Ben Mildenhall 的深度对话。视频聚焦 World Labs 全新发布的世界模型 Atlas，围绕空间智能的探索，探讨了如何通过“新视角预测”将生成与 3D 重建统一在同一个模型中，以及该技术对创意产业、物理模拟和机器人具身智能所带来的深远变革。 ## TL;DR World Labs 团队深度解析了新一代世界模型 Atlas 的底层技术范式——以“新视角预测（New View Prediction）”替代传统的下一帧或下一个 Token 预测，首次在统一多模态架构中融合了 3D 几何重建与生成式先验。模型仅凭极稀疏的几张照片或普通手机视频，即可高保真复现具备严格物理空间一致性的场景漫游与子弹时间效果，将数据采集门槛降低了数十倍。团队不仅复盘了架构从纯静态向原生动态演进的历程与坚信 Scaling Law 的顿悟时刻，更阐明了世界模型在赋能持久性 3D 创意资产沉淀、充当神经模拟器以及突破机器人数据与 Real-to-Sim 瓶颈上的核心价值。 ## Atmosphere & style 整场对话充满极高密度的硬核前沿技术探讨与真诚的学术创业激情。主持人 Martin Casado 提问犀利且直击行业痛点，而李飞飞、Justin 与 Ben 三位顶尖学者兼创始人的配合默契自然，兼具严谨的理论推演与极富感染力的创业故事。尤其是团队追忆在初夏清晨看到“花园桌下逼真足球”时的顿悟瞬间，以及李飞飞借生物演化移动性赋予眼睛的深刻隐喻，让整期节目既有硅谷前沿突破的兴奋感与笃定自信，又洋溢着对空间智能与物理世界本质探索的哲学诗意。 ## Bulletin - [00:00](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=0s) World Labs 推出下一代世界模型 Atlas，将核心预测基元从传统视频模型的“下一帧预测”重构为“新视角预测”（New View Prediction），在 3D 场景重建效率上实现 50 到 100 倍的数据需求骤降。 - [05:16](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=316s) Atlas 打破了传统计算机视觉中“3D 密集重建”与“生成模型”长达半个世纪的学科割裂，在预训练阶段将 3D 相机位姿、深度图与 RGB 像素熔合为一等公民原生多模态架构。 - [10:47](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=647s) 在系统架构层面，团队解耦了上一代 Marble 绑定的 3D 高斯泼溅单一输出格式，支持多达 64 张以上长上下文图像吞吐，将三维重建本质转化为极长上下文的自回归生成任务。 - [29:02](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1742s) 李飞飞指出机器人领域当前的首要瓶颈是数据而非算力，Atlas 作为生成式基础设施将加速 Real-to-Sim 仿真与领域随机化（Domain Randomization），未来世界模型可直接从神经模拟器演化为动作规划器。 - [40:34](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2434s) 团队将“新视角预测”提升至等同于大语言模型“下一个 Token 预测”的 AI 完备性地位，视其为生物演化中因空间移动而必须攻克、通向真正空间智能的第一性底层基元。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=0s) 大语言模型的基石是“下一个 Token 预测”，视频模型的基石是“下一帧预测”，而 Atlas 的核心是基于空间上下文与物理接地的“新视角预测”（New View Prediction）。 - [01:05](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=65s) Atlas 实现了类似电影《黑客帝国》“子弹时间”的生成能力，仅需三台普通 iPhone 拍摄动态画面，无需绿幕、演播室或内外参精密标定即可重构出时空冻结的穿梭视角。 - [02:51](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=171s) 与传统视频模型黑盒盲猜、抽卡式的“老虎机效应”不同，Atlas 输入的每一帧图像都绑定了精确的 3D 相机位姿，具备物理空间接地与高确定性的几何保真度。 - [05:16](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=316s) Atlas 终结了学术界半个世纪以来在视点估计、几何重建与像素生成上的割裂，首创将 3D 相机位姿和深度图作为一等公民融入端到端世界模型。 - [08:15](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=495s) 空间智能（Spatial Intelligence）绝不仅限于生成像素，而必须深入到空间的生成、空间内推理、可编辑性交互以及具身物理规划四个维度。 - [10:47](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=647s) 上一代产品 Marble 将输出模态严格绑定在 3D 高斯泼溅上导致扩展受阻；Atlas 重构了逻辑，将各模态解耦，确立新视角预测为核心基元。 - [13:32](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=812s) 作为 NeRF 奠基人，Ben Mildenhall 指出传统密集 3D 重建要求每个缝隙有 3-4 个角度覆盖且极其耗时；Atlas 借助强大的数据先验，将输入视角需求从数百张压缩至两三张。 - [16:50](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1010s) 在斯坦福四合院演示中，模型仅凭地面仰拍照片就自动补全了空中俯瞰的全景细节，展现出几何三角测量与生成式先验补全未知盲区的深度结合。 - [18:44](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1124s) 三维重建在本质上等价于拥有超长上下文的生成任务；Atlas 突破了单图限制，支持同时处理 64 张以上视点图像的长上下文窗口。 - [21:27](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1287s) 缩放定律（Scaling Law）在 3D 世界模型中依然高度有效，测试中摄像机穿过花园桌下方并真实呈现足球视差的顿悟时刻，坚定了团队全线押注 Atlas 的决心。 - [25:23](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1523s) 创意工作流的最大痛点在于缺乏持久性 3D 状态（persistent 3D state），导致视角一换物体漂移；世界模型提供一致的物理基准，能将传统 3D 建模中繁重的修改效率提升数倍。 - [29:02](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1742s) 机器人技术目前最严峻的卡点是真实世界交互数据，Atlas 通过高效环境稠密重建与领域随机化（Domain Randomization），极大打通了 Real-to-Sim 与 Sim-to-Real 闭环。 - [32:48](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1968s) 机器人策略输出的是置身物理世界、必须应对各种边缘失效模式的自主智能体，生成式神经仿真器（Neural Simulator）将成为具身智能不可或缺的试错试验场。 - [34:31](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2071s) 当世界模型充分理解物理环境对动作的动态反馈后，底层的神经模拟器在数学逻辑上可直接对偶演化为动作规划器（Planner）。 - [36:53](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2213s) 为了构建极高精度的静态几何，反直觉的有效训练路径是向模型输入海量动态现实数据，促使其在预训练中理解潜在动力学并将动态变化自然因式分解剥离。 - [37:55](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2275s) 世界模型的下一阶段竞争在于“可编辑性”（Editability），核心工程挑战是在不降低顶尖生成画质的前提下，赋予用户对 3D 状态、物体身份与时间轴的深度控制。 - [42:24](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2544s) 从演化生物学看，动物拥有眼睛而植物没有正是因为动物需要移动；在空间运动中理解和预测新视角是生存的核心，也是空间智能时代最根本的 AI 完备基元。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=0s) 空间智能新范式：从“预测下一个 Token”到“新视角预测” 在节目的开篇，World Labs 团队抛出了对空间智能（Spatial Intelligence）的愿景：实现真正具备空间上下文感知与物理接地（spatially contextualized and grounded）的像素生成，正是其最新世界模型 Atlas 所迈出的关键一步。Justin Johnson 提炼了这一代际跨越的本质区别：如果说大语言模型（LLM）的基石是“下一个 Token 预测”（next token prediction），现有视频模型的基石是“下一帧预测”（next frame prediction），那么 Atlas 的根本逻辑则是“新视角预测”（new view prediction）。 这一范式转换带来了生产力上的断代式飞跃，展现出 50 到 100 倍的效率提升。正如访谈开场讨论的缩影：电影《黑客帝国》（The Matrix）中经典的尼奥下落避弹镜头，原本需要在绿幕摄影棚中架设数百台精密校准的摄像机；而 Atlas 仅凭三台普通手机拍摄的视频，无需绿幕、摄影棚或复杂的校准流程，就能重构出同等惊艳的冻结时间视角。团队在模型缩放（scaling）的过程中也验证了这一技术路线的确定性——模型参数每扩大一次、训练时间每延长一轮，其表现都会出现显著跃升。 ### [01:05](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=65s) Atlas 的三大核心能力与用三台 iPhone 复刻“子弹时间” Martin Casado 谈及 Atlas 发布后在业内引发的轰动，并邀请 Justin Johnson 系统阐述这款新模型的定位。Justin 解释道，Atlas 是 World Labs 推出的下一代世界模型，底层具备三大核心能力支柱：**生成（Generate）**、**重建（Reconstruct）** 与 **模拟（Simulate）**。在具体功能层面，Atlas 支持高度精准的相机条件生成（camera-conditioned generation），用户只需输入一张静态图像并给定相机运动轨迹，即可引导模型从任意设定的视角生成连续画面；同时，它在稀疏 3D 重建（sparse 3D reconstruction）上表现惊人，能够接收从单张到多达 100 张现实世界的不同视角图像，输出穿越该空间的平滑漫游视频或显式的 3D 空间重建；最后，在模拟维度上，Atlas 不仅展现了强大的机器人仿真能力，更凭一套“子弹时间”（bullet time）演示刷爆网络。 为了通俗解释“子弹时间”，Justin 唤起了流行文化中最具标志性的一幕：《黑客帝国》第一部中尼奥仰身避子弹的慢动作旋转镜头。电影工业当年为了捕捉这一瞬间，需要在环形绿幕影棚中排布数百台昂贵设备，进行极度繁琐的逐台校准。而 Atlas 彻底颠覆了这种重工业流程： - **极简采集**：只需三脚架架设三台普通 iPhone，在日常环境下拍摄动态画面（例如投篮入框、草莓落入牛奶碗）。 - **零额外成本**：不需要绿幕、专业演播室，也省去了耗时数天的相机内外参严苛标定。 - **动态时空重组**：从三段不同角度的普通视频中重新构图，模型能“冻结时间”，让虚拟相机在飞溅的牛奶液滴之间穿梭飞跃，输出极致细腻的多重视角动态重现。 ### [02:51](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=171s) 空间上下文与告别“老虎机式”生成的确定性控制 谈及 Atlas 的输入输出本质，Justin 将其最核心的基语（primitive）定义为“新视角预测”。用户向模型输入某个场景的一组视角或一段描述，这些信息会在模型内部被整合成一个隐式的“空间上下文”（spatial context），完整定义了当前所讨论的物理世界；随后，用户可以在该时空坐标系中任意放置一台虚拟相机，Atlas 便能准确理解并渲染出从该视角看过去的真实世界形态。 ```mermaid flowchart TD accTitle: Atlas 与传统模型预测逻辑对比 accDescr: 对比展示大语言模型、传统视频模型与 Atlas 空间世界模型在核心预测基语上的演进差异 subgraph LLM["语言模型 LLM"] A["文本序列 Context"] --> B["Next Token Prediction<br/>预测下一个 Token"] end subgraph Video["传统视频模型 Video Models"] C["时间轴初始帧"] --> D["Next Frame Prediction<br/>沿时间轴预测下一帧"] end subgraph AtlasModel["Atlas 世界模型 World Labs"] E["多视角/描述输入"] --> F["构建隐式 Spatial Context"] F --> G["指定 3D 相机位姿与时间点"] G --> H["New View Prediction<br/>精准预测新视角与几何结构"] end ``` 针对市面上琳琅满目、皆自称“世界模型”且具备视角变换功能的视频扩散模型，Ben Mildenhall 进一步抽丝剥茧，阐明了 Atlas 的根本差异。过去许多视频模型依靠单图生成或“首末帧插值”成名，近期也有模型支持 20 至 50 张图像的所谓“全参考”（omni reference）。但那些模型对待图像就像“黑盒”，仅凭文本提示词去盲猜画面的关联，导致用户往往陷入反复重抽抽卡的“老虎机效应”（slot machine effect）。 相比之下，Atlas 的关键突破在于： - **物理空间接地（Spatial Grounding）**：输入的每一帧图像都原生绑定了精确的三维相机位姿（3D camera pose）。 - **极高精度的确定性复现**：如果在房间的四个角落各拍一张照片输入 Atlas，模型会精确还原房间内的一切，绝对不会像传统视频模型那样去随意胡编角落之间的摆设与几何关系。 - **精确导向的创意编排**：即便面对不同 AI 生成的梦幻图片或异地素材，创作者也可以像电影导演一样在三维空间中精确布光、布景，严格规划运镜路线，实现完全确定性的镜头穿梭。 ### [05:16](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=316s) 原生多模态架构与计算机视觉半个世纪的统一 当 Martin Casado 追问 Atlas 究竟只是传统视频模型的暴力缩放（scale-up），还是彻底重构了底层架构时，Justin 与李飞飞给出了斩钉截铁的答案：这是一套在底层就将“3D 重建”与“内容生成”熔合在一起的全新原生架构。 Justin 详细剖析了模型结构的设计革新。以往在计算机视觉领域，3D 重建和生成模型完全是两套平行的子学科：前者依赖复杂的几何约束与专用算法来精确复刻真实物体，而后者依赖文本扩散模型天马行空地创造不存在的虚构场景。Atlas 在预训练阶段就将两者融为一体，打造了真正的原生多模态结构： - **全要素原生输入**：模型在预训练时不仅吃进文本、图像与视频，还将 3D 相机位姿作为底层一等公民（first-class citizen）的条件变量直接注入网络。 - **3D 几何表征的引入**：在具体实现上，模型将相机位姿与 RGB 色彩信息以及深度图（depth maps）相互绑定，让深度图直接表达三维物理结构。 李飞飞随后动情地回顾了计算机视觉长达半个多世纪的发展历史。她感叹道，在视觉领域深耕数十年来，见证过数不清以“3D 重建”或“新视角合成”（novel view synthesis）为题的博士毕业论文。传统的计算机视觉学术会议（如 CVPR、ICCV）总是被割裂成泾渭分明的几个 Track：一块专门做像素生成，一块专注图像识别，另一块专注 3D 几何重建。 而 Atlas 第一次以极为优雅的方式，通过在视点估计（viewpoint estimation）与新视角生成上的统一锚定，终结了长达半个世纪的学术割裂，真正将像素级的“空间重建”与“想象生成”统摄在同一个端到端世界模型框架之中。 ### [08:15](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=495s) 空间智能的本质与新视角预测的基石地位 当主持人 Martin Casado 将话题拉回 World Labs 创立之初的愿景时，他指出，普通大众通常把大语言模型理解为“下一个 Token 预测”（Next-Token Prediction），而在 World Labs 的语境下，这项任务变成了“新视角预测”（New-View Prediction）——即模型在接收到单个或一组已有视角后，能够推断并预测出在空间与时间中处于全新视角的画面。李飞飞（Fei-Fei Li）随即系统阐释了“空间智能”（Spatial Intelligence）的内涵：空间智能绝不仅仅是生成像素，而是要赋能我们在空间中进行生成、空间内推理、编辑与交互。 李飞飞强调，即便最终的终极目标是引入时间维度的 4D 世界，仅就 3D 而言，这也是最底层的基本盘。要实现渲染、仿真模拟以及具身智能的动作规划，首要必须解决对几何、结构与物理法则的理解。Atlas 之所以成为关键一步，在于它在生成每一个画面时都能推断并生成最核心的几何信息——视点与相机姿态（Camera Pose）。市面上已经存在海量（"gazillions"）能生成像素的模型，但生成“真正具备空间上下文、并在物理空间中扎根（grounded）”的像素才是最艰难的飞跃。 ```mermaid flowchart TD accTitle: 从新视角预测到空间智能的演进路径 accDescr: 展示了从输入视角、相机姿态联合估计，过渡到 3D 与 4D 空间理解、仿真与具身交互的层级架构。 subgraph InputStage ["感知与预测层"] A["输入视角 (RGB 图像/视频)"] --> B["Atlas 新视角预测模型"] B --> C["联合估计与生成: 像素 + 相机姿态 (Camera Pose)"] end subgraph SpatialStage ["空间结构与几何层"] C --> D["3D 几何与物理结构表征"] D --> E["4D 动态与时间演化"] end subgraph ActionStage ["空间智能应用层"] E --> F["高保真仿真与编辑"] E --> G["空间推理与具身动作规划"] end ``` - **空间智能的四大核心能力**：空间生成、空间内推理、交互式编辑、具身动作规划与模拟。 - **几何锚点**：Atlas 每次推断出的“相机姿态（Camera Pose）”构成了下游所有涌现行为（Emergent Behaviors）与物理一致性的几何基准。 - **未来路线图**：后续将继续引入时间维度（第 4 维）以解锁真实的物体动态，以及更高保真度的空间物理细分与仿真。 --- ### [10:47](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=647s) 从 Marble 到 Atlas：打破高斯泼溅的输出瓶颈 World Labs 成立已有两年半时间。针对为何没有一步到位推出 Atlas，Justin Johnson 幽默地回应称自己的团队需要“大量的算力芯片（GPUs）”来进行 Scaling。他详细复盘了上一代模型 Marble 与最新模型 Atlas 之间的底层架构范式转变。去年发布的 Marble 模型能够接收图像、视频或文本提示词，并生成 3D 世界，但 Marble 的输出模态完全绑定在 3D 高斯泼溅（3D Gaussian Splatting）上。虽然高斯泼溅渲染速度极快、能轻松跑在移动设备和 VR 头显上，且容易对接游戏引擎与物理引擎，但将其作为唯一的端到端输出格式，最终成为了限制模型扩展的严重瓶颈。 在 Atlas 的设计中，团队重构了底层逻辑，提前对不同模态进行了分流解耦（Bifurcate Modalities），使各模态在统一架构中协同运转。Atlas 不再把“生成一个高斯泼溅世界”当做核心基元（Primitive），而是将“新视角预测”确立为第一性基元。它既可以直接输出连续的 RGB 帧，也可以预测显式的 3D 几何；在用户需要时，依然可以后处理生成高品质的高斯泼溅资产，却不再受限于其固化的表征限制。团队经历了大量的“心血与泪水（blood, sweat and tears）”才权衡清楚不同表征的优劣，并通过一系列由小到大的实验逐步建立了对该架构具备可扩展性（Scalability）的笃定信念——毕竟在他们启程之时，业内根本不存在所谓的“空间智能 Scaling Law”。 --- ### [13:32](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=812s) 从“密集重构”到“稀疏先验”：NeRF 奠基人的范式倒转 作为 NeRF（神经辐射场）的发明人，Ben Mildenhall 回顾了自己整个职业生涯在“从图像中重建 3D”领域的技术积累。早期团队一直在争论：究竟该通过合成多个新视角再重建 3D，还是直接从输入生成 3D 格式？Ben 指出，传统 3D 密集重构（Dense Reconstruction）长期以来在商业化落地上极其艰难，因为其逻辑类似于医学或科学成像：它要求任何你想看清的缝隙，都必须有至少 3 到 4 个不同角度的高清机位覆盖。 Ben 形象地以录制现场举例：在麦克风底部、桌面下方、墙角缝隙、乃至绿植叶片的层叠之间，为了不漏掉死角，专业人员需要极其枯燥而耗时地满屋子扫描；即使经验丰富的人也需要耗费数分钟，普通用户拿手机扫描一个多房间环境甚至要耗时一到两小时，拍摄两三百张照片。而 Atlas 的核心价值，就是将所需的输入视角从两三百张直接砍到两三张（实现 50 到 100 倍的数据需求骤降）。通过将强大的数据驱动先验（Data-driven Priors）与视角预测相结合，以往那些完全不符合重构标准的随手随拍视频、网上搜集的老旧片段，甚至被随意扔掉 95% 冗余照片的废弃数据，如今都能被直接唤醒并“复活”为连贯且完整的三维世界。 ### [16:50](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1010s) 重建与生成的融合：从斯坦福四合院到盲区脑补 李飞飞指出了团队官方网站展示中一个极易被外界低估的演示——斯坦福大学主四合院（Stanford Quad）的重建。Ben Mildenhall 当时仅凭站在地面上拍摄的 3 到 25 张普通照片，Atlas 便成功重建出了整座四合院的全貌。更为惊艳的是，展示视频采用的是俯瞰全景的空中飞越视角（aerial view），而输入给模型的却全都是从地面仰拍或平视的照片。这表明视频中看到的绝大多数高空细节全都是模型“生成”出来的，但这种生成又严格遵循了三维“重建”的几何法则。李飞飞由衷感叹，这种生成与物理定律结合的效果“堪称神奇”。 Justin Johnson 随即深入剖析了生成（Generation）与重建（Reconstruction）之间为何必须产生根本性的融合。在 Ben Mildenhall 所深耕的传统重建范式中，之所以需要海量的密集视点，是因为算法必须通过多张图像中的对应像素进行空间三角测量（triangulation）。但这种方法的致命缺陷在于：任何未被输入视角直接捕捉到的像素，在三维重建结果中都会变成一个空洞。然而现实世界中，即便让 Ben 这样的顶级专家拿着单反相机在房间里拍上数百张高密度照片，也绝不可能拍全每一个角落——麦克风底部、桌面下方、椅子腿之间的缝隙永远都会存在盲区。要填补这些不可避免的信息缺口，就必须依赖模型的生成能力去合理“脑补”未知区域。 ```mermaid flowchart TD accTitle: 三维重建与生成的协同流程 accDescr: 展示传统几何重建与生成式模型在三维场景补全中的互补关系 subgraph Input["多视角输入"] A["稀疏现实照片 (3~25 张)"] end subgraph Processing["Atlas 核心机制"] B["几何三角测量<br/>(Triangulation)"] C["生成式先验预测<br/>(Generative Inpainting)"] end subgraph Output["完整 3D 场景"] D["已知视角物理对齐"] E["盲区与遮挡补全<br/>(阴影/缝隙/空中视角)"] F["全场景自由视角漫游"] end A --> B A --> C B --> D C --> E D --> F E --> F ``` - 传统三维重建的刚性瓶颈在于“所见即所得”，未被摄入的死角无法还原。 - Atlas 将几何约束与生成先验深度绑定：先锚定可见结构，再推测补全未捕获的空间。 ### [18:44](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1124s) 长上下文窗口：把重建视为超长上下文的生成任务 Justin Johnson 借用大语言模型（LLM）的发展历程做了一个极具洞察力的类比。过去几年，LLM 领域掀起了轰轰烈烈的“上下文窗口大战”（Context Wars），上下文从最初的 128、256、512 token 一路飙升到上百万，开发者在写代码时已经本能地依赖超长上下文带来的推理提升。然而在图像与视频生成领域，此前从未有人以如此体系化的方式推进上下文长度的扩展——没有任何团队去尝试把一段长达一小时的视频塞进模型，并做类似于“大海捞针”（needle in a haystack）式的第 37 分钟关键帧精准检索。 在 Justin 看来，三维重建本质上就是“输入了极长上下文的生成过程”。只要构建出能够平滑扩展上下文容量的架构，就能在纯生成与精确重建之间搭建起一座连续的桥梁。对比团队早期的产品 Marble，其架构存在根本性瓶颈，最多只能塞入几张图像作为条件；而 Atlas 实现了质的飞跃，可以直接吞吐多达 64 张图像甚至更多。Ben Mildenhall 提到，以前对一栋多房间住宅进行完整的三维漫游重建需要拍摄 2,000 张照片，而现在借助 Atlas，只需输入 30 到 40 张稀疏照片，生成的室内穿梭视角在质量上几乎毫无二致。 - 大语言模型的上下文战争在三维模型中以“输入视点数量的跃升”重现。 - 早期产品 Marble 仅支持数张视点输入，而 Atlas 支持数十张输入并实现高保真连续推演。 - 极少量的稀疏输入（30~40 张）即可达到过去数千张密集采集照片的重建效果。 ### [20:06](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1206s) 自回归机理与 3D 一致性的突破 Atlas 的运行逻辑本质上是一个自回归（autoregressive）生成引擎。用户输入任意数量的初始图像（甚至可以精简到单张照片），然后操控虚拟摄像机，指定在空间特定坐标输出新的观察视角。由于模型具备自回归特性，用户可以逐步、交互式地将新生成的视角重新追加回上下文窗口中，从而排布出一段高密度的连续航拍或漫游路径。 主持人 Martin Casado 表达了强烈的震撼与不解。在一般认知中，扩散模型（Diffusion Models）往往在视觉观感上极度逼真，但在空间几何上缺乏刚性准确度，常常前后穿帮。然而 Atlas 却能在仅有几张离散图片的情况下，在大幅度外推（extrapolate）新视角的同时严格保持真实的三维几何一致性，使得整个房间在空间尺度上完全契合。面对 Martin 关于“这究竟是靠纯数据堆砌还是模型理解了三维”的疑问，Justin 确认这在很大程度上来源于对“缩放定律”（Scaling Law）在三维场景下依然生效的笃定。 ### [21:27](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1287s) 缩放定律的坚定信念与“花园桌下”的顿悟时刻 当 Martin 询问团队在立项之初是否确信这套方案能够成功时，Justin Johnson 坦言自己当时“相当笃定”。李飞飞补充道，三位联合创始人从一开始就对两项核心假设拥有无可动摇的信念：其一是缩放定律（Scaling Law），其二则是“下一视点预测”（Next Viewpoint Prediction）。虽然宏观信念坚定，但具体的模型架构选择与数据配比却充满了工程挑战。李飞飞回顾，Justin 的团队经历了一段漫长而焦灼的探索，从“不知道还要熬多久才能见效”，到“隐约看到了生命迹象”，再到最终惊呼“天哪，这真的行得通”。 Justin 坦承，尽管他坚信方向正确，但完全没想到在第一轮预训练周期（pre-training cycle）中就能以如此快的速度达到这种质量高度。在深度学习中，采用全新架构和全新范式的首个训练循环直接取得惊艳成果是极为罕见的，他原本做好了反复试错几个周期的心理准备。面对 Martin 关于“当前架构是否已经撞墙、需要新的突破”的提问，Justin 坚定地反驳道：“不，我们才刚刚开始。”目前唯一的瓶颈是训练算力（training compute）。在研发过程中，团队沿着缩放阶梯训练了一系列模型，每次增加模型参数、延长训练时间并增加算力芯片，模型表现都会出现肉眼可见的巨大飞跃；博客展示的成果仅仅是受制于发布截止日期所能训练出的当前规模。 李飞飞随后分享了一段极具感染力的创业内幕。在初夏的某一天，团队还在测试一个规模远小于当前 Atlas 的小模型。Ben 将经典的 NeRF 评测场景——一张户外的“花园桌”（garden table）照片输入模型，并尝试生成视点。 第二天清晨，李飞飞和 Justin 的 Slack 同时收到了 Ben 发来的视频。那是摄像机平滑穿过花园桌下方时渲染出来的画面，在桌子底下，端端正正地躺着一个逼真的足球。 这颗完全符合空间遮挡与视差规律的足球彻底引爆了团队。当天早晨，三位联合创始人碰面，彼此目光对视，“仅仅用了五秒钟，我们就达成了共识：就是它了，我们必须全力以赴把它做出来。在此之前，整个计算机视觉领域从未有人见过这样的结果。” - 核心技术信念确立于两大基石：缩放定律与下一视点预测。 - 架构潜力远未见顶，当前性能上限主要受限于训练算力而非理论天花板。 - 初夏的“花园桌底穿梭”演示成为团队决定全线押注 Atlas 路线的关键转折点。 ### [24:42](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1482s) 创意工作流的演进：从 Marble 到 Atlas 的实际应用 Martin 将话题引向具体的落地场景。World Labs 此前积累了大量的创意领域用户，涵盖 2D 图像一致性生成、电影影视制作、3D 建模及游戏资产开发等领域。Martin 询问 Atlas 将如何拓展这些现有场景，并为后续深入探讨机器人技术铺平道路。 Ben Mildenhall 解释道，用户过去在使用上一代产品 Marble 时的方式，恰好印证了“新视角预测”这一核心能力的必要性。很多创意从业者的经典工作流是：先向 Marble 输入一张参考原画，生成一份以 3D 高斯泼溅（Gaussian Splatting）为表现形式的完整三维场景，然后再从中截取若干个全新角度的画面进行二次创作。Atlas 的出现不仅全面升级了这一流程的几何精度与视角自由度，更从底层为创作者提供了近乎无限制的虚拟空间探索能力。 ### [25:23](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1523s) 3D 持久性状态与创意工作流的革新 在新视点合成（novel view synthesis）领域，过去学术界往往将其视为基于密集视角采样的重建问题，而将其转化为生成式视点合成（generative view synthesis）则是一个全新的范式。在实际的创意生产管线中，几乎没有人会仅仅依赖单一的模型（如仅使用单个图像或视频生成模型）来完成整个任务。创作者的典型工作流通常是多阶段的：首先从各种图像模型中挑选图片构建故事板（storyboard）和情绪板（mood board），再把它们作为关键帧输入到不同的视频工具中生成动态片段，最后进行剪辑与后期修切。在此过程中，创作者面临的最大痛点之一就是无法保持 3D 一致性——反复提示图像模型从不同角度展示同一个房间时，物体总会莫名其妙地漂移或变形。 ```mermaid flowchart TD accTitle: 传统创意管线与世界模型工作流对比 accDescr: 对比传统碎片化多模型生成管线与基于世界模型具备持久3D状态的新型管线 subgraph Legacy["传统多阶段创意管线（碎片化且状态不稳定）"] A["图像模型生成情绪板 / 故事板"] --> B["拼接入视频工具生成关键帧"] B --> C["后期剪辑与修片"] C -. "痛点：视角切换时物体漂移、缺乏真实3D几何一致性".-> A end subgraph WorldModel["基于持久性 3D 状态的新型管线（Atlas / Marble）"] D["输入多模态数据 / 姿态图像"] --> E["具备空间上下文的世界模型"] E --> F["精确控制视角、道具与场景元素"] F --> G["多场景扩展：影视布景、展台搭建、建筑与工业制造"] end ``` 这种对“持久性 3D 状态（persistent 3D state）”的渴望，实际上植根于人类数十年来的工作习惯。无论是在真实世界中搭建舞台、摆放道具，还是在虚拟世界中制作电影镜头、营销广告或游戏场景，人类的空间推理本质上都依赖于状态的留存与随时间的演化。人们不希望每次交互都面对“生成一个、丢弃一个、仅保留文本提示词”的短暂虚拟产物，而是希望积累资产、构建一个可沉淀的真实世界模型。通过引入空间上下文机制（spatial context mechanism），模型赋予用户更高的精确度和控制力。 这种能力不仅局限于纯视觉创意领域，更能无缝扩展到建筑、室内设计以及各类实体制造行业。现实世界中绝大多数需要被物理制造出来的东西——小到展会展台的搭建，大到复杂的建筑构造——都必须经历极其繁琐的虚拟设计阶段。当前整个流程中最耗时费力的瓶颈就在于 3D 建模软件：软件界面与交互逻辑历经数十年几乎没有本质变化，当创意总监或建筑师给出随手勾勒的草图或口头修改意见时，设计师往往需要花费整整一周的时间在 3D 软件中手动重新映射并调整模型，这一过程占用了多达 95% 的工作量。AI 如果能让 3D 建模像搭乐高、做陶艺或用铅笔素描一样直观自然，将为创意设计与工业制造释放出惊人的生产力。 - **工作流碎片化现状**：创作者被迫在多种单模态工具之间来回跳转，依赖拼凑图像与视频，极度缺乏统一的空间基准。 - **持久性 3D 状态价值**：保持场景道具、结构和光影在多视角、多时间跨度下的稳定性，终结“视角一换、物体乱变”的模型缺陷。 - **传统 3D 软件的效率鸿沟**：将口头反馈和粗略草图还原到 3D 模型中往往需要消耗一周以上的修正时间，是传统虚拟设计中最繁重的一环。 --- ### [29:02](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1742s) 破解机器人学的核心瓶颈：数据与 Real-to-Sim 仿真 当话题从创意设计延伸到机器人学时，Martin Casado 提出疑问：既然 Atlas 最初展现出的是对创意设计和建筑场景的助力，那么它与 World Labs 收购的机器人公司究竟有怎样的底层映射？李飞飞（Fei-Fei Li）明确指出，Atlas 是机器人全栈布局中至关重要的一块拼图。被收购的团队此前名为 Scenix，其核心技术聚焦于实现“真实到仿真（Real-to-Sim）”以及“仿真到真实（Sim-to-Real）”的双向打通。 ```mermaid flowchart LR accTitle: 机器人训练中的 Real-to-Sim-to-Real 闭环 accDescr: 展示从现实数据采集、稠密重建、环境随机化、策略训练到实机部署的闭环流程 subgraph Reality["真实世界采集"] A["工业现场 / 日常场景<br/>（如线缆整理、洗碗）"] --> B["多角度采集图像 / 视频"] end subgraph SimLoop["Sim 仿真环境（Atlas 加速层）"] B --> C["高保真 3D 稠密环境重建"] C --> D["领域随机化 Domain Randomization<br/>（线缆弯曲度、箱体尺寸/颜色/开盖状态）"] D --> E["大规模机器人策略训练与评估"] end subgraph Deploy["真实世界部署"] E --> F["部署至真实机械臂 / 机器人执行任务"] F -. "遇到新长尾场景与失败数据".-> A end ``` 在典型的工业机器人训练任务中（例如让机械臂在工厂中完成复杂的线缆整理与插接），工程师必须首先获取海量数据来训练动作策略（policy），在仿真环境中评估其表现，最后才能将策略部署到真实机械臂上。然而，过去使用传统稠密重建（dense reconstruction）来还原作业环境的过程异常痛苦、耗时且效率低下，严重拖慢了 Real-to-Sim 环节的迭代速度。Atlas 正是作为下一代生成式基础设施，彻底颠覆并加速了这套环境重建流程。 李飞飞进一步强调，放眼整个机器人领域，当前最根本的核心瓶颈并非算力，而是数据——“未来某一天可能是芯片，但就目前而言，绝对是数据”（"One day it'll be chips but for now it's data"）。机器人要在现实物理世界中作业，真实数据的采集极其困难。更关键的是，为了让策略具备泛化性，必须进行“领域随机化（Domain Randomization）”：以线缆作业或洗碗为例，不能只训练线缆朝一个固定方向弯折，必须随机化线缆的弯曲形态、随机改变箱体的大小、颜色、盒盖位置以及整个场景的光照与几何配置。如果单靠互联网抓取或实物搭建，根本无法获得足够的多样性数据，必须依赖高保真的 Real-to-Sim 技术生成长尾变体。Atlas 作为多模态全能模型（omni/multimodal model），其未来的技术路线图还将直接接入动态物理数据，从根本上弥合环境模拟与机器人动作规划（action planning）之间的鸿沟。 - **Scenix 核心能力**：专攻 Real-to-Sim 与 Sim-to-Real 的闭环系统，解决机械臂在工业装配等场景下的策略落地难题。 - **数据是头号瓶颈**：实体场景数据采集成本高昂，且难以穷尽所有可能的环境变化。 - **领域随机化（Domain Randomization）**：在数字孪生空间中批量生成不同材质、形态、姿态和遮挡情况下的物理推演场景，使策略获得应对未知干扰的鲁棒性。 --- ### [32:48](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1968s) 策略学习与神经仿真器：机器人与物理交互的本质区别 Justin Johnson 深入拆解了机器人策略训练与其他生成式 AI（代码、图像、文本、视频）之间的底层范式差异。其他领域的生成模型本质上是在输出“静态工件（artifact）”——无论是写出一组代码库、画出一张画还是生成一段视频，互联网上都已经存在数以亿计现成的成品数据可供模型消化学习。但机器人策略截然不同，它生成的不是静态工件，而是一个需要置身于物理世界、主动采取动作并努力实现特定目标的“智能体（agent）”。 在现实世界中，物理环境往往不会按照预期的方式给予反馈，各种未曾设想的物理碰撞、摩擦力变化、滑移或干扰都会随时发生。因此，机器人策略在训练阶段必须提前暴露给所有可能发生的失败模式与意外情况（"exposed to every possible thing that could go wrong"）。正是这种对长尾边缘情况的苛刻要求，使得仿真（simulation）成为了机器人训练中不可或缺的唯一通路。 Justin 分析指出，构建这种仿真环境目前存在两条不同的技术路径： 1. **经典物理引擎仿真**：人类工程师凭借想象力预设各类可能出现的故障情形，并在物理引擎中手动编写显式代码进行建模。尽管现代编程智能体（coding agents）正在极大地加速这一过程，但它依然受限于人类预先定义的规则框架。 2. **基于数据驱动的神经仿真器（Neural Simulator）**：直接训练一个能够深刻理解物理世界规律的神经网络世界模型，由模型自主推演环境对机器人动作的动态响应，甚至展现出偶发的、意料之外的物理反应。 通过在大规模物理数据上训练这种高保真神经仿真器，世界模型将演变为下一代具身智能的核心摇篮，允许机器人在无需物理实体介入的前提下完成亿万次虚拟试错与交互演练。 - **静态工件 vs 动态交互智能体**：代码与音视频生成只需对齐既有数据分布，而机器人策略必须具备应对物理世界随机扰动的闭环决策能力。 - **失败用例覆盖**：机器人在部署前必须在仿真器内经历极大规模的极限工况测试，这是物理世界实拍数据无法低成本供给的。 - **从规则引擎到神经仿真**：结合生成式世界模型的神经模拟技术，正在打破传统基于手工编写物理参数的仿真天花板。 ### [34:31](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2071s) 从神经模拟器到规划器：世界模型的通用智能命题 将神经模拟器（learned neural simulator）作为训练机器人控制策略（robotic policies）的仿真试验场，是 Atlas 极其诱人的演进方向，但团队的视野绝不止步于此。Justin Johnson 阐述了一个更为深远的逻辑跃迁：一旦模型学会了充当物理世界的神经模拟器，其内部隐空间实际上就已经构建起对客观规律的“大脑直觉”——它懂得世界面对不同外力交互时会作何反应。既然如此，“模拟器本身为什么不能直接成为规划器（planner）呢？”这正是 World Labs 对世界模型及其通用性（generality）的核心论断：一个底层模型必须具备生成世界、仿真环境以及推演多视角展现的核心能力；而“理解世界在受到某种动作后会如何演变”，本质上与“为了达成某种预期的世界状态而反推自己应当采取何种动作”属于硬币的两面。 ### [35:22](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2122s) 动态性质疑与破解：从 Marble 的完全静态到 Atlas 的原生支持 Martin Casado 率先对 Atlas 的发布表达了热烈祝贺，称其为“今年整个行业最具分量的模型发布”，行业反馈几乎是一片赞誉。不过他也抛出了一位资深同行发来的尖锐反馈：“模型确实令人惊叹，但它需要包含更多动态（dynamics）。”在 Casado 看来，无论对于机器人落地还是广泛的通用场景，一个能真正动起来的世界才是最理想的。 面对这一质疑，Justin Johnson 与团队相视一笑，甚至调侃道模型其实已经拥有了“婴儿期的动态（baby dynamics）”。Justin 透露了一个此前在官方博文中未曾浓墨重彩强调的技术事实：World Labs 的上一代世界模型 Marble 在架构和训练逻辑上都是彻底静态的，面对动态物理完全无能为力。但在 Marble 之后，团队立即将这一痛点列为最高优先级攻关，Atlas 从底层的网络架构到输入训练数据，早已在底层原生支持了动态物理特征。只要细看官方演示视频，就能发现一些细微的动态细节： - 场景中水面泛起的层层涟漪与波浪； - 俯瞰航拍鸟瞰视角中，道路上平稳穿行蠕动的一辆辆小汽车。 ```mermaid flowchart TD accTitle: Marble 与 Atlas 的技术代际演进对比 accDescr: 展示了从完全静态的 Marble 到原生支持动态与时空因式分解的 Atlas 的技术进化路径。 subgraph Previous["上一代模型：Marble"] M1["纯静态模型架构"] --> M3["受限于纯静态训练数据，极难规模化扩展"] M2["时间维度完全冻结"] --> M3 end subgraph Current["新一代模型：Atlas"] A1["原生支持时空动态架构"] --> A3["大规模混合预训练：静态 + 动态海量数据吞吐"] A2["隐式表征时空物理"] --> A3 A3 --> A4["后训练对齐：定向分解动态干扰，输出极致几何重建"] end Previous -.->|"架构重构与范式升级"| Current ``` ### [36:53](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2213s) 重构与动态的辩证：为什么获得极致静态需要喂给模型动态数据 Casado 敏锐地提出了技术直觉上的矛盾：如果算法的目标是从多个视角精准重构静态 3D 空间，动态元素难道不会成为严重的干扰项与噪点吗？Justin 直截了当地指出，这正是过往纯静态思路的认知盲区。在底层 3D 重建中，理论上确实希望冻结时间以获取绝对精准的静态对应视角，然而现实中“绝对静态的现实数据”极其罕见，根本无法支撑现代大模型级别的数据 Scaling 需求。 World Labs 得出的突破性认知是：即便最终想要的是纯粹的静态几何输出，最有效的训练路径恰恰是向模型暴露尽可能丰富的动态现实，让模型在海量静态与动态混杂的数据中自行领悟规律，学会把动态变化“因式分解（factor out）”剥离出去。Atlas 在预训练阶段已经吞吐了极其庞大的动态物理数据，其内部权重蕴含着深厚的潜在动力学理解（latent dynamics）；而在该版本发布的后训练（post-training）阶段，团队特意将重心收敛在空间视角的移动与高精度静态重建上，故意压低了时间维度的变化。这也意味着，动态能力早已深植于模型骨髓，未来版本的释放只是节奏问题。 ### [37:55](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2275s) 走向可编辑性：在不牺牲画质的前提下赋予世界模型多模态控制 当 Casado 笑着探询未来能否直接生成“可以随意漫步其中的 4D 视频”，并询问除了单纯的“更大、更快、更强（bigger, faster, better）”之外还有何激动人心的方向时，Ben Mildenhall 将焦点引向了多模态与控制条件（control conditioning）。Ben 认为，学术界长期低估了控制机制对于激发模型深层潜力的决定性意义。李飞飞随即一语中的地将其提炼为大白话——“可编辑性（editability）”。 Ben 进一步剖析，顶尖的图像模型已经能够游刃有余地理解复杂的多轮修改指令，支持用户直观地把特定人物、物体和事件无缝融合在一起，而无需用户手动做繁琐的图层调整；但在视频生成和世界模型领域，控制机制仍停留在“输入一句话然后画面中蹦出一只玩具恐龙”这种初级玩具演示的水平。世界模型面临的核心工程挑战在于： - **不妥协画质的高阶控制**：绝不能为了增加控制把手（knobs）而牺牲生成画质，否则模型就沦为虚有其表的“聚会杂耍（party trick）”，专业创作者绝不会为了孱弱的控制力去牺牲前沿视频模型的顶级质量。 - **重构人机交互界面**：赋予用户精细操纵场景布局、对象身份一致性、交互逻辑乃至时间维度的能力，从而自底向上彻底重构人类在计算机中与具有状态的 3D 世界（stateful 3D worlds）的交互方式。 ### [40:34](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2434s) 空间智能的第一性原理与“AI 完备性”：新视角预测即下一个 Next Token 李飞飞将话题拉回智能的第一性原理：在面对物理世界与空间时，真正的智能绝非孤立地坐在原地被动观察或解析世界，而是在于“观察、体验与互动（seeing, experiencing, and interaction）”之间形成严密的闭环。 紧接着，Justin 抛出了计算理论中引人深思的概念——“AI 完备性（AI-completeness）”。正如经典计算复杂度理论中，NP 完全问题（例如 3-SAT）可以通过归约来代表并解决该类别中的一切难题；在人工智能领域，也有某种底层的基本原语具备如此特质： - **大语言模型的范式**：下一 Token 预测（Next Token Prediction）被普遍视为 AI 完备的。正如 Ilya Sutskever 举过的经典悬疑小说案例——让模型读完整本侦探小说，最后一句话是“凶手就是……”，预测下一个 Token 的任务便直接等价于全流程的逻辑推理与综合智能。 - **空间智能的基石**：团队在研发 Atlas 的过程中日益笃信，**新视角预测（New View Prediction）**，尤其是生成式新视角预测，同样是一项具备 AI 完备性的核心原语。通过空间视角在时空中的任意推演，几乎所有关于物理存在、因果关系与交互规划的智能难题，都能在这一框架下得以表达与求解。 ### [42:24](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2544s) 下一个视角预测：空间智能时代的“下一个 Token 预测” 对话在探讨世界模型与生成能力的极致想象中推向高潮。Martin Casado 和团队描绘了极具戏剧性的生成场景：比如模型生成了一整部电影的全部帧画面，直至凶手现身的悬疑时刻，系统能够精确预测并生成凶手从阴影中走出的那一幕；或者构建这样一个虚拟世界——在背景里 Martin 正在白板上书写黎曼猜想（Riemann hypothesis）的数学证明，而镜头随后平滑摇移到另一块白板，揭晓他彻底解开猜想的过程。这种不仅能生成连续剧情，更能依据三维物理与逻辑一致性来预测未知视角的机制，展现了模型对真实动态世界的深刻理解。 随后，李飞飞（Fei-Fei Li）从宏观的生物进化视角升华了这一技术范式的重要性。她指出，大自然赋予了动物能够观察世界的眼睛，却没有给植物和树木眼睛，其根本原因在于动物需要移动。而一旦生物在空间中产生移动，就必须即时应对和理解新的视角（new viewpoint）。这种在运动中预测和感知新视角的能力，正是自然演化为赋予生物生存与智能而必须攻克的核心问题。李飞飞强调，无论人们称其为“AI 完全（AI complete）”还是“智能完全（intelligence complete）”，World Labs 团队都无比坚信：“下一个视角预测”（next viewpoint prediction）就是空间智能领域中等同于大语言模型里“下一个 Token 预测”（next token prediction）的根本基石。 节目在对 Atlas 模型的赞叹中圆满落幕。Martin Casado 对 World Labs 团队取得的突破性模型发布表示热烈祝贺，并对团队未来在世界模型上的持续迭代与前沿探索寄予了厚望。 ```mermaid flowchart LR accTitle: LLM与世界模型的预测范式对比 accDescr: 对比语言智能的下一个Token预测与空间智能的下一个视角预测 subgraph LLM["语言智能 (Language AI)"] A["文本语境 Context"] --> B["Next Token Prediction<br/>(下一个 Token 预测)"] B --> C["抽象语义理解与推理"] end subgraph WorldModel["空间智能 (Spatial Intelligence)"] D["空间与时间感知 Motion"] --> E["Next Viewpoint Prediction<br/>(下一个视角预测)"] E --> F["物理规律与3D世界重建"] end ``` - **生物演化隐喻**：植物保持静止因而不需要视觉感知器官，而动物因为具备移动能力，演化必须解决“移动中对新视角的理解与预测”这一生存难题。 - **技术范式定位**：正如“下一个 Token 预测”点燃了文本大模型的生成与推理能力，“下一个视角预测（Next Viewpoint Prediction）”被 World Labs 视为通往真正空间智能（Spatial Intelligence）与物理世界建模的核心原语。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=0s) 如果说大语言模型（LLM）的基石是“下一个 Token 预测”（next token prediction），现有视频模型的基石是“下一帧预测”（next frame prediction），那么 Atlas 的根本逻辑则是“新视角预测”（new view prediction）。 — Justin Johnson > [08:15](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=495s) 空间智能绝不仅仅是生成像素，而是要赋能我们在空间中进行生成、空间内推理、编辑与交互。 — Fei-Fei Li > [08:15](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=495s) 市面上已经存在海量（"gazillions"）能生成像素的模型，但生成“真正具备空间上下文、并在物理空间中扎根（grounded）”的像素才是最艰难的飞跃。 — Fei-Fei Li > [21:27](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1287s) 仅仅用了五秒钟，我们就达成了共识：就是它了，我们必须全力以赴把它做出来。在此之前，整个计算机视觉领域从未有人见过这样的结果。 — Fei-Fei Li > [21:27](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1287s) 不，我们才刚刚开始。 — Justin Johnson > [29:02](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=1742s) One day it'll be chips but for now it's data — Fei-Fei Li > [34:31](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2071s) 模拟器本身为什么不能直接成为规划器（planner）呢？ — Justin Johnson > [35:22](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2122s) 模型确实令人惊叹，但它需要包含更多动态（dynamics）。 — Martin Casado > [36:53](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2213s) 即便最终想要的是纯粹的静态几何输出，最有效的训练路径恰恰是向模型暴露尽可能丰富的动态现实，让模型在海量静态与动态混杂的数据中自行领悟规律，学会把动态变化“因式分解（factor out）”剥离出去。 — Justin Johnson > [42:24](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2544s) 大自然赋予了动物能够观察世界的眼睛，却没有给植物和树木眼睛，其根本原因在于动物需要移动。而一旦生物在空间中产生移动，就必须即时应对和理解新的视角（new viewpoint）。 — Fei-Fei Li > [42:24](https://www.youtube.com/watch?v=qn1QDDBnTA0&t=2544s) 无论人们称其为“AI 完全（AI complete）”还是“智能完全（intelligence complete）”，World Labs 团队都无比坚信：“下一个视角预测”（next viewpoint prediction）就是空间智能领域中等同于大语言模型里“下一个 Token 预测”（next token prediction）的根本基石。 — Fei-Fei Li ## Mentioned World Labs, Fei-Fei Li, Justin Johnson, Ben Mildenhall, Martin Casado, a16z, Atlas, Marble, Scenix, NeRF, 3D Gaussian Splatting, The Matrix, Stanford Quad, Ilya Sutskever, CVPR, ICCV, iPhone ## Video description World Labs co-founders Fei-Fei Li, Justin Johnson, and Ben Mildenhall join a16z General Partner Martin Casado to discuss Atlas, their latest world model, and what it reveals about the pursuit of spatial intelligence. At the center of Atlas is what the team calls “new view prediction”: given images or views of a scene, the model predicts what that environment should look like from a different position in space and time. This brings generation and 3D reconstruction into the same model, and raises a broader question about whether predicting views could become a useful primitive for understanding the physical world. They discuss the technical bets behind the model, what it can and can’t yet capture, and the importance of dynamics, editability, and simulation as world models develop. The conversation also explores applications in creative work, architecture, and robotics, where Fei-Fei argues that one of today’s biggest constraints is access to real-world training data. Timestamps: 00:00 - Intro 00:51 - What Atlas Is & Why It Matters 05:15 - Is This a Scaled-Up Video Model or a New Architecture? 08:15 - Spatial Intelligence & Why New View Prediction Matters 21:27 - Did You Know It Was Going to Work? 24:42 - Use Cases: Creatives, Games & Robotics 35:21 - The Elephant in the Room: Video Models vs World Models 37:55 - Will We Get 4D Video You Can Walk Around In? 42:22 - Why New View Prediction Is the Next Token Prediction Resources: Follow Fei-Fei Li on X: https://x.com/drfeifei Follow Justin Johnson on X: https://x.com/jcjohnss Follow Ben Mildenhall on X: https://x.com/BenMildenhall Follow Martin Casado on X: https://x.com/martin_casado Learn more about Atlas: https://www.worldlabs.ai/blog/atlas Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends! Find a16z on X: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see http://a16z.com/disclosures. [Source](https://www.youtube.com/watch?v=qn1QDDBnTA0)

[0:00] Justin Johnson: 如果说大语言模型（LLM）的基石是“下一个 Token 预测”（next token prediction），现有视频模型的基石是“下一帧预测”（next frame prediction），那么 Atlas 的根本逻辑则是“新视角预测”（new view prediction）。 [8:15] Fei-Fei Li: 空间智能绝不仅仅是生成像素，而是要赋能我们在空间中进行生成、空间内推理、编辑与交互。 [8:15] Fei-Fei Li: 市面上已经存在海量（"gazillions"）能生成像素的模型，但生成“真正具备空间上下文、并在物理空间中扎根（grounded）”的像素才是最艰难的飞跃。 [21:27] Fei-Fei Li: 仅仅用了五秒钟，我们就达成了共识：就是它了，我们必须全力以赴把它做出来。在此之前，整个计算机视觉领域从未有人见过这样的结果。 [21:27] Justin Johnson: 不，我们才刚刚开始。 [29:02] Fei-Fei Li: One day it'll be chips but for now it's data [34:31] Justin Johnson: 模拟器本身为什么不能直接成为规划器（planner）呢？ [35:22] Martin Casado: 模型确实令人惊叹，但它需要包含更多动态（dynamics）。 [36:53] Justin Johnson: 即便最终想要的是纯粹的静态几何输出，最有效的训练路径恰恰是向模型暴露尽可能丰富的动态现实，让模型在海量静态与动态混杂的数据中自行领悟规律，学会把动态变化“因式分解（factor out）”剥离出去。 [42:24] Fei-Fei Li: 大自然赋予了动物能够观察世界的眼睛，却没有给植物和树木眼睛，其根本原因在于动物需要移动。而一旦生物在空间中产生移动，就必须即时应对和理解新的视角（new viewpoint）。 [42:24] Fei-Fei Li: 无论人们称其为“AI 完全（AI complete）”还是“智能完全（intelligence complete）”，World Labs 团队都无比坚信：“下一个视角预测”（next viewpoint prediction）就是空间智能领域中等同于大语言模型里“下一个 Token 预测”（next token prediction）的根本基石。

[5:15] Fei-Fei Li: Sitting here having been in this field for decades, I cannot tell you how many PhD thesis have been written on the problem of reconstruction or novel view synthesis... This is an elegant model that unifies the problem of reconstruction and generation by anchoring on viewpoints. [21:22] Justin Johnson: Overnight I got a Slack... the fly-through flew through under the table with the soccer ball... That morning the three of us looked at each other in the eyes and said 'that's it. We're going to build this.' [38:00] Fei-Fei Li: editability is the key

[0:00] on the path to spatial intelligence, [0:02] generating pixels that are truly [0:04] spatially contextualized and grounded. [0:06] That is the very hard step that Atlas [0:09] has taken. [0:09] >> We know LLMs are built on next token [0:11] prediction. We've seen video models as [0:13] being built on next frame prediction. [0:14] Atlas is really new view prediction. [0:16] >> This is the real place where AI can [0:18] actually unlock a ton of value for [0:20] people and their process. We're seeing [0:21] like 50 100x reduction. There was a [0:23] famous shot in the first Matrix movie [0:25] where Neo is like falling down. Exactly. [0:27] They had hundreds of cameras viewing [0:29] that angle on a green screen. On Atlas, [0:31] we can do this with just three cameras. [0:33] No studio capture, no green screen, no [0:35] expensive calibration. [0:36] >> No one has ever seen this result. [0:38] >> When you set out to do this, did you [0:40] know it was going to work? I was pretty [0:41] sure. Each time we made the model bigger [0:43] and each time we trained it for longer, [0:44] it got significantly better. Does that [0:46] mean we're going to get 4D video? We [0:47] like go walk around. [0:48] >> Uh, [0:51] so big day yesterday. You launched uh a [0:53] new Frontier model which got an amazing [0:55] reception which is still coming in. I [0:56] think maybe a good way to structure this [0:58] conversation, let's just talk about [0:59] exactly what that was and then we'll go [1:00] back to history and work our way back [1:02] up. So maybe Justin, do you want to talk [1:03] about what was launched yesterday, why [1:04] it's significant? [1:05] >> Yeah. So Atlas is our new next gen [1:07] generation world model. It has three [1:09] basic things. It can generate, [1:10] reconstruct, and simulate the world. Um, [1:12] so within that, there's a couple [1:13] different major capabilities. It has [1:15] really good camera condition generation. [1:16] So you can input an image together with [1:18] a camera trae with a camera trajectory [1:20] and steer the model and have it [1:21] generate, you know, video frames along [1:23] any any perspective you want. It's [1:25] really good at sparse 3D reconstruction. [1:26] You can input one or m multiple up to a [1:29] 100 frames um that are views of the real [1:31] world and use those to reconstruct the [1:33] real world. And that reconstruction can [1:35] take the case either of of a of a novel [1:37] of video flying through the space or an [1:39] explicit 3D reconstruction of the space. [1:41] Um then finally it can be used for [1:42] simulation. Um and for this we show off [1:44] um you know these awesome bullet time [1:46] videos which got a lot of attention [1:47] online and then also robotic simulation. [1:50] >> What's a bullet time video? [1:51] >> A bullet time video this comes from the [1:53] Matrix. You know, there was a famous [1:55] shot in the first Matrix movie where Neo [1:56] is like falling down. Exactly. [1:59] >> So, and then remember in that famous [2:00] shot he's like falling down. It's in [2:01] slow motion and the camera flies all the [2:03] way around. Um, so that's the way that [2:06] they did that shot is they had a ring of [2:07] like hundreds of cameras. So then like [2:10] he fell over in the studio. They had [2:11] hundreds of cameras viewing that angle [2:13] on a green screen and then they used [2:14] those hundreds and hundreds of cameras [2:16] to make that that famous shot in in the [2:17] Matrix. But now with Atlas, we can do [2:19] this with just through three cameras. So [2:22] like no studio capture, no green screen, [2:24] no expensive calibration. We can [2:26] literally stick like three cameras on [2:27] three iPhones on tripods. Um use these [2:30] to take sort of a video of something [2:32] happening um like someone shooting a [2:33] basket, someone dropping a strawberry [2:35] into a bowl of milk and then from those [2:37] three like iPhone videos, we can then [2:39] reframe the shot and imagine like a like [2:41] freeze time, have the camera fly in like [2:43] as the milk is splashing up and get [2:44] these amazing frozen time views. [2:46] >> Um and we can do this with just just a [2:48] couple cameras. Can can you um just [2:50] maybe what is the simplest description [2:52] of what Atlas does? Like what goes in [2:54] and what comes out? [2:55] >> Yeah. So one of the one of the really [2:56] core principles of Atlas like the most [2:58] fundamental thing is it does new view [3:00] prediction. Um and this is a a really [3:02] fundamental primitive that we think is [3:03] super exciting a super new primitive for [3:06] for base models that no one's ever done [3:08] before. Right? So we know LLMs are built [3:10] on next token prediction. We've seen [3:11] video models as being built on next [3:13] frame prediction. Atlas is really new [3:15] view prediction. Right? that given some [3:17] number of views of a scene or a [3:19] description of a scene, um, those go [3:21] into what we call a spatial context that [3:23] describes implicitly what is the world [3:24] that we want to talk about, then you can [3:26] point a virtual camera at at an [3:28] arbitrary point in space and time and [3:30] Atlas will understand what that world is [3:32] supposed to look like from that position [3:33] in space and time. you know, you know, [3:35] Ben, that you know, with with a with a [3:37] bajillion video models out there all [3:39] claiming to be world models and all [3:41] claiming to have novel views and can you [3:44] maybe tease apart [3:46] kind of more concretely how this is [3:48] different from like the myriad models [3:50] that have come before. [3:51] >> Mhm. Yeah, I think what Justin was [3:52] saying about the spatial context aspect [3:54] is super important here. Um, so there's [3:56] many video models, a lot of video models [3:58] actually got their claim to fame from [3:59] their single image input or their start [4:01] to last frame interpolation. Now, we're [4:03] starting to see models that can do this [4:05] kind of omni reference thing with, you [4:06] know, 20, 30, 50 images. But what's key [4:09] with Atlas is that it actually has a [4:11] kind of like spatially grounded meaning [4:12] to every frame you put into it. So, it's [4:14] not just an image that the model's going [4:16] to interpret whatever way it wants or [4:18] you can kind of try to argue with it in [4:19] the the text prompting and get it to do [4:21] something specific. With Atlas, every [4:23] image actually has an associated [4:25] threedimensional camera pose. And that [4:26] means that you can perform this task of [4:28] reconstruction with an extremely high [4:30] degree of precision. Right? So, if we [4:31] had four views of this room, one at each [4:33] corner, you can put those into the model [4:35] and then get an exact replication of [4:37] everything you see in this room. And [4:38] it's not going to guess what's in the [4:40] other corner, like a relationship [4:42] between things. It's just going to [4:43] reproduce exactly what you give it. Um, [4:45] and you can also do that in a kind of [4:46] creative or imaginative sense, too. If [4:48] you take two photos from different, you [4:51] know, AI generations or real world [4:52] locations, you can actually position and [4:54] stage those to build these kind of [4:56] intentionally directed flythroughs um [4:59] that are really governed by exactly the [5:01] precise place that you put the content [5:03] you want and where the camera's going to [5:04] look and travel. Uh which is very [5:06] different, I think, than the kind of [5:08] like more slot machine effect you get of [5:09] having to retry generations over and [5:11] over with just that kind of higher level [5:13] of text control you get with video [5:14] models. Is this just kind of an obvious, [5:18] you know, scaled up version of a [5:20] traditional video model or is it a new [5:22] architecture? [5:23] >> I think it's it's a pretty new thing for [5:25] a couple different reasons. Um, one that [5:27] like we talk about is it does both [5:29] generation and reconstruction jointly in [5:31] the same model. Um, like Ben was saying, [5:33] this thing can take a couple views of [5:34] this room and then reconstruct [5:35] everything in this room exactly as you [5:37] see it. And historically, reconstruction [5:39] has been its own subfield in computer [5:40] vision with its own specialized task, [5:42] its own specialized models. Um, and [5:44] generation is what all the text, all the [5:46] text to video models are really good at, [5:48] like what all the all the big diffusion [5:49] models we've seen the last couple years. [5:51] And those are great for creative [5:52] applications. I want to imagine [5:53] something that's never been there [5:54] before. Um, but now with Atlas, for the [5:57] first time, we're putting these two [5:58] different parts of visual intelligence [6:00] together in one model. So, it can do [6:02] both 3D reconstruction and generation [6:04] together in one architecture. So, to do [6:06] that, we had to make a couple changes. [6:08] Um, one is we had to make it multimodal [6:10] from the start. So, this thing natively [6:12] works on text. It works on images. It [6:14] works on videos. It also works on camera [6:16] poses um as a native input to the model, [6:19] which I don't think anyone's ever done [6:20] at the pre-training phase before. Um and [6:22] it uses um it uses 3D as a native [6:24] modality that it works on. So, this [6:26] thing from the beginning was designed to [6:27] be natively multimodal in a way that no [6:29] one else I think. [6:30] >> Sorry, I just um I don't know the space [6:32] super well. By 3D, is this like depth or [6:36] models or like what does that what does [6:38] that mean? [6:38] >> Yeah. So the formulation we used so far [6:40] is depth maps. Okay. Right. So um right [6:42] now you can have a when you have a frame [6:44] that has a a a virtual camera telling [6:46] its position in 3D space. That camera [6:48] position and and camera parameters are a [6:50] native input to the model. And then what [6:52] attached to that camera position, you [6:53] can have both um like RGB telling you [6:55] what is that what does that position in [6:57] space look like and you can have a depth [6:58] map that tells you what is the spatial [7:00] structure of that position in 3D space. [7:02] So then you know text, image, video, 3D [7:05] cameras are these modalities that this [7:07] thing all does jointly in a multimodal [7:09] way. [7:10] >> I want to add something because I think [7:12] what Justin just said is actually [7:14] >> so important and also what Ben said that [7:17] it's underappreciated. It's the first [7:20] time [7:20] >> we have a unification of pixel [7:23] generation and pixel reconstruction in [7:25] the world of computer vision. This field [7:28] has been around for more than half a [7:30] century. um sitting here having been in [7:33] this field for decades, I cannot tell [7:36] you how many PhD thesis have been [7:38] written on the problem of reconstruction [7:41] or novel view things uh synthesis and [7:44] also our field traditionally [7:48] uh have multiple tracks. You go to a [7:50] computer vision conference, you have the [7:52] pixel generation track, you have some [7:55] recognition track and you have 3D [7:57] reconstruction track. This is a elegant [8:00] model that combines uh or unifies the [8:03] problem of reconstruction and generation [8:05] by anchoring on um viewpoints and uh the [8:09] the viewpoint uh estimation and that's [8:13] just incredibly powerful. [8:15] >> Can can you can you maybe well can we [8:18] take a step back and then maybe you just [8:19] fill something out. So when you started [8:21] the company I remember you saying uh you [8:23] know you want to tackle um spatial [8:25] intelligence right and uh you know now [8:28] we have this new model and so it feel I [8:31] mean like as a lay person feels very [8:33] general to me you've got next to [8:34] prediction and this is next new view [8:36] prediction [8:36] >> new view prediction right and so like [8:38] you can get one view set of views and [8:39] you get a new view can maybe you pencil [8:41] out like like h how this is a [8:43] significant step to this general problem [8:45] of spatial intelligence m maybe by like [8:47] starting to describe what spatial [8:49] intelligence is [8:50] >> well spatial intelligence eventually [8:53] must enable us to both generate what the [8:57] space is reason within it and being able [9:00] to edit and interact within it. Yeah. [9:03] >> Now we can argue is it 3D or 4D [9:06] ultimately it's 4D with the time [9:07] dimension but even just 3D these are the [9:10] fundamental tasks that one has to do or [9:14] spatial intelligence has to enable and [9:16] then we talk about with that you can [9:18] render you can simulate and you can plan [9:21] actions but to do that a fundamental [9:24] problem to solve is to understand the [9:26] geometry and structure and the physics [9:29] of the space. [9:30] >> Yeah. And I do believe Atlas is a [9:33] significant step forward because now [9:36] with every single frame you have a you [9:40] you can generate and estimate an [9:42] important piece of information which is [9:44] the the view viewpoint, the camera pose. [9:47] And that is the most critical [9:49] information one needs about the geometry [9:52] of the of the space. and that can lead [9:54] to all the emergent uh behaviors we see [9:57] in our um in in the downstream of the [10:00] model which we showed in the blog. So, [10:03] so in the on the path to spatial [10:06] intelligence, [10:08] >> generating pixels is definitely a a [10:10] early step which we have seen with what [10:12] you call gazillions of uh models. But [10:16] generating pixels that are truly [10:18] spatially contextualized and grounded is [10:21] absolutely another major step and that [10:23] is the very hard step that Atlas has [10:26] taken. [10:27] >> I we definitely have you know we can [10:30] just keep going here right like there is [10:32] the fourth dimension of time which will [10:35] bring in dynamics and there is more [10:37] higher fidelity simulation and [10:40] delineation of the space. So this is [10:43] part of the the road map of spatial [10:46] intelligence. [10:47] >> Great. Yeah. I mean and I definitely [10:49] want to like dig into like where this is [10:51] going, but first maybe let's talk about [10:53] getting here. Uh how long has world labs [10:55] been in existence? [10:58] >> Two and a half. [10:59] >> Two and a half. Yeah. [11:00] >> And so you you've actually released [11:01] models before. So why didn't you just [11:04] jump right to Atlas? [11:07] >> Great question. [11:08] >> It's so magic, right? Yeah. Justin's [11:10] team needs a lot of chips. [11:12] >> Yeah, you need a lot of GPUs to actually [11:13] scale this thing up. Um, so what one of [11:15] the Well, last year we released our [11:17] Marble world model and that was the [11:18] first kind of big major world model that [11:20] we put out. Um, that that powers our [11:21] current Marble product and Mar and [11:23] Marble is really cool. Marble can take [11:24] images, it can take videos, it can take [11:26] text prompts and use these to generate [11:27] 3D worlds. Um, but one of the biggest [11:29] differences between Marble and Atlas is [11:31] exactly what is that output modality. So [11:33] Marble was really focused on Gausian [11:35] splats as an output representation. So [11:37] whatever you're inputting um it's going [11:38] to output a 3D world represented as a as [11:40] a gausian splat and gausian splats are [11:42] really useful right they're really nice [11:43] they're easy to render they are they [11:44] they can they can render efficiently on [11:46] mobile devices on VR devices they can [11:48] interoperate with other with with game [11:50] engines with simulation engines so [11:52] there's a lot of nice things about [11:53] gausian splats but you know I think that [11:56] was kind of a bottleneck in the previous [11:57] marble model so what we did with Atlas [11:59] is redesign the thing a little bit um [12:01] and we realized that we need to [12:02] bifurcate these modalities earlier and [12:04] actually have these things all these [12:06] modal ities working in a more unified [12:07] way in the model. So now with Atlas um [12:10] the the fundamental primitive is not [12:11] like a g could generate a gausian splat [12:13] world. The fundamental primitive is is [12:15] as we said new view prediction [12:17] >> um and that can generate RGB frames that [12:19] can generate 3D and we can use those to [12:21] generate beautiful gausian splat worlds [12:23] when you need them. Um but we don't need [12:25] to bottleneck our our outputs through [12:26] the gausian splats when we don't need [12:28] to. And that was a that actually took a [12:30] lot of you know blood, sweat and tears [12:32] to understand like what are all the pros [12:34] and cons of these different [12:35] representations. Yeah, [12:36] >> that's one part of it. Um the other part [12:38] is you got to like climb the scaling [12:40] ladder right you got to like work your [12:41] way up and like do smaller experiments, [12:43] do smaller models like to build your [12:45] conviction on what's going to work and [12:47] what's going to scale. Um and there's [12:48] there's you know if you could instantly [12:50] know the right thing that's going to [12:51] scale you know you should just do that. [12:53] But when we started the company left the [12:56] world was a very different place like [12:57] >> there is there was no scaling law of [12:59] spatial intelligence [13:00] >> right so like when we started the [13:02] company like the world was in a very [13:03] different place the tech was in a very [13:04] different place we had a lot of [13:05] ambitions about where we wanted it to go [13:07] but it took a couple it took a a couple [13:09] iterations for us to hit upon this [13:11] formulation that we thought is actually [13:12] is actually like this is the one this [13:14] this is the one that can scale out [13:16] >> you know Ben um you know being the [13:18] creator of Nerf and doing a lot of 3D [13:20] and reconstruction so it's not so [13:22] obvious to me that like if you have [13:24] multiple views that you actually end up [13:25] with a 3D thing, but like you've kind of [13:27] like made a career of ending up with a [13:28] 3D thing. So maybe talk a little bit [13:30] about like kind of that step. [13:32] >> Yeah. Yeah. Um yeah. I mean, as you [13:34] said, I've spent many many years of my [13:36] career, the vast majority of my career [13:38] actually working on producing 3D things [13:40] from images. Um and this actually [13:42] something we we talked a lot early on in [13:44] the company even of like is this going [13:45] to be the approach that produces 3D, [13:47] right? Are we going to synthesize [13:48] multiple views and then build three out [13:49] of that? Are we going to try to go [13:50] direct to 3D? like there's been a lot of [13:53] um uncertainty in the field around like [13:54] which of those approaches kind of will [13:57] win out or will kind of like reap the [13:59] best advantages earlier on. Um but I did [14:01] have a lot of conviction just from [14:03] seeing the kind of power of what I would [14:06] almost call the brute force scaling [14:08] scaling in a very very very small baby [14:10] scale not like real model scaling but [14:12] the scaling of dense reconstruction that [14:14] we had seen happening over the past [14:16] three years before. Um, so basically we [14:18] put [14:18] >> why is dense reconstruction [14:20] dense? [14:21] >> Yeah, [14:23] dense [14:23] >> because I know we're going to talk about [14:24] sparse and I want to make sure that [14:25] people understand what is dense and what [14:27] is sparse. [14:27] >> Yeah, so I think this has actually even [14:29] on the kind of like business and [14:30] commercial side I think been one of the [14:32] challenges of productizing uh 3D [14:35] reconstruction technology like at a [14:36] fundamental level, right? people kind of [14:38] don't [14:40] uh in in a in a casual sense like you [14:42] think I took three photos of this object [14:44] or I took six photos of this room like I [14:47] I look at the photos I can understand in [14:50] my mind like how those piece together I [14:51] can kind of fill in the gaps and get it [14:53] but there's just never been really any [14:56] kind of reconciliation between those [14:58] like really datadriven priors and then [15:00] the kind of brute force dense [15:02] reconstruction which it actually is much [15:04] more akin to almost like scientific or [15:06] medical imaging what we did intense [15:07] reconstruction, right? You basically [15:08] have to say every single thing I want to [15:10] appear in this reconstruction, I need at [15:12] least three or four views of it. And if [15:14] you think about that, like even just in [15:16] this room, right? There's like under the [15:17] microphone, under the table, between [15:19] every different crack and crevice and [15:21] the plant leaves, right? To actually [15:23] truly get a picture that covers every [15:26] one of those spots, it's this like very [15:28] tedious and exhaustive effort to walk [15:30] around the room. I think you've all seen [15:31] me running around various places like [15:33] capturing them. It takes, you know, for [15:35] for someone who's well trained per se, [15:37] like it can take minutes. But if you [15:39] hand um a casual consumer or even some [15:41] like kind of professional trying to do [15:42] this for the first time, uh an average [15:45] like cell phone camera or capture [15:46] device, like it's going to take them [15:48] probably an hour. I've seen someone for [15:49] their first time trying to scan a [15:50] multiroom environment spend like two [15:52] hours walking through it uh and get [15:54] enough coverage. And that's just this [15:56] like very very exhaustive and tedious [15:58] loop. Um and so yeah, when we say dense, [16:00] we really mean dense. It's like this [16:01] room I want like so many photos, right? [16:04] I want like 100, 200, 300 photos of this [16:06] room to to capture it. And what we're [16:08] trying to do is bring that down to like [16:09] three, right? We're saying like 50, 100x [16:12] reduction. And then that's at that scale [16:14] where it just completely like flips that [16:16] calculus on its head of like what type [16:18] of captures you reconstruct. You can go [16:20] back to existing imagery you have. You [16:22] can go to stuff you find on the internet [16:24] and even build scenes out of that. you [16:25] can go to casual videos and like kind of [16:27] unearth a lot of footage in the past we [16:29] would never have treated as [16:30] reconstructible uh and go back and like [16:32] bring it to life as 3D potentially. This [16:34] is something we've been playing around [16:35] with a lot with Atlas, right? Like [16:37] taking old clips like I've taken a bunch [16:39] of my own old captures that never worked [16:41] before and then put them through the [16:42] system and then kind of seen a [16:44] reconstruction for the first time or [16:46] taken my old captures and thrown away [16:47] 95% of the photos I took. uh and you [16:50] know imagine angles that I never would [16:52] have gotten from a traditional kind of [16:53] like nerf or splat type reconstruction. [16:55] >> One thing that's underappreciated on the [16:57] website of the demos is the Stanford uh [17:01] demo where uh Ben showed uh [17:05] >> anywhere between 3 to 25 images. You can [17:08] uh reconstruct that entire Stanford [17:11] quad. But the thing is we had to show it [17:14] from a aerial view. But every single [17:17] input image is Ben standing on the [17:19] ground taking a picture from the ground. [17:22] So everything you see are generated but [17:26] according to the laws of reconstruction [17:28] and this is really magical [17:30] >> and this is where like generation and [17:31] reconstruction need to interplay in a [17:33] really fundamental way to solve this [17:34] problem because under the classic kind [17:36] of reconstruction stuff that Ben was [17:38] talking about like the reason you need [17:39] so many views is because I need like [17:42] multiple images and I need to [17:43] triangulate this point in 3D space and [17:45] see it from multiple viewpoints. So that [17:47] means like that's required in the [17:49] traditional version. And on the flip [17:51] side, anything that wasn't captured in [17:53] these views, like any pixel that was not [17:55] visible in one of the input views will [17:57] be a hole in the 3D reconstruction [17:59] because fundamentally like if a thing [18:00] wasn't visible in the input views, you [18:03] know, you need to imagine it to fill in [18:04] the gaps. And that's fundamentally a [18:06] generative process. So, even in this [18:07] room, even if we set Ben loose with a [18:09] DSLR and like let him like capture like [18:11] hundreds of views of this room, even the [18:13] world expert on doing these dense [18:15] captures is still going to miss some [18:16] spots. Like he's not going to get like [18:18] underneath all of the microphones or [18:20] underneath all the tables or like in [18:22] between all the chair lights, you're [18:23] always going to miss something no matter [18:24] how many views you get. So that that's [18:26] where you need generation as another [18:27] mechanism in the model because you're [18:29] never going to get everything. So you [18:31] need to have some generative capacity [18:33] for the model to imagine oh based on [18:35] what I'm seeing then like first [18:38] triangulate what I what I can see but [18:40] then fill in the gaps of the stuff that [18:41] inevit inevitably was not captured. [18:44] >> Yeah. And there's something like super [18:45] cool about this that LLMs have really [18:47] understood this for for a long time, [18:48] right? There was kind of almost these [18:49] like context wars. Like the first couple [18:51] years, right? It was like, oh, we got to [18:52] 128 to 26 to like 512, we got a million, [18:55] right? And everyone kind of understands [18:57] now at a pretty tangible level the value [18:59] of, you know, you crank your context [19:00] length to high when you're using your [19:01] coding model. It's a hard problem. Like [19:03] everyone has a feel for that, but like [19:05] no one has pushed that at all on the [19:06] image and video model side in the same [19:08] kind of like principled way. like no [19:10] one's out there trying to like put a [19:11] like an hourong video through and do a [19:13] needle in a hay stack retrieval of like [19:14] a frame at the 30 37 minute mark. Uh [19:17] whereas with reconstruction generation, [19:18] you actually have the same exact thing [19:20] of like reconstruction is just like [19:22] generation with a really long context [19:23] and you put a lot of stuff in it, right? [19:24] Like that's the way to actually build [19:26] this continuum where you kind of bridge [19:27] between those two things. Uh and like [19:29] Atlas like being able to like this is [19:31] something we can never do with Marvel. [19:32] Marvel had this kind of fundamental [19:34] blocker of like you couldn't really jam [19:36] more than honestly like a couple images [19:37] in. Atlas, I can go and I can actually [19:40] take like a 64 image capture uh and do [19:42] like a fly through of an entire house [19:44] and everything is grounded by being, you [19:46] know, seen or like almost seen or like [19:48] slightly extrapolated from what's not [19:50] there. But you're just getting these, [19:51] you know, I'm taking captures I did with [19:53] 2,000 images of a multi-room house and [19:55] taking it down to like 30 40 inputs and [19:57] the fly through looks like basically the [19:58] same. And this is just like totally [20:00] inconceivable before. And it's all [20:01] enabled by building this gracefully [20:03] scaling kind of context window that you [20:05] can dump stuff into. [20:06] >> And and so the way to think about it is [20:08] like the the sparseness are the pictures [20:11] that you physically took [20:13] >> and then Atlas as a model creates the [20:16] rest of the views and then you use [20:17] classic reconstruction techniques. Is [20:19] that roughly the way to think about it [20:20] or [20:20] >> in some sense? Yeah. Yeah. I mean that's [20:22] the beauty of Atlas. It's like you can [20:24] take however many inputs you use you [20:25] have down to like a single view and then [20:27] you can almost use Atlas as this this [20:29] rendering engine to produce anything [20:31] else you want. Right. You can you can [20:32] navigate a virtual camera. [20:34] >> Yeah, exactly. You can just you can say [20:35] like, okay, I have a picture here. I [20:36] want the picture there, there, there. [20:38] You can make a couple of those. Then you [20:39] can stage a dense fly through. You can [20:41] do this in sequence because it's an auto [20:42] reggressive model. It's up to you, [20:44] right? To kind of pick and choose what [20:45] you add interactively into the context [20:47] as you generate. I mean, the thing that [20:50] I just blows my mind is listen, I just [20:53] have a very simple me mental model. I I [20:55] I have four pictures and then I've got [20:57] to like have the model extrapolate [21:00] between them and then it has to fit when [21:02] you're in control. Like it's got to be [21:03] 3D cuz like and I always think of these [21:06] diffusion models as like being visually [21:09] great but not accurate. And so like and [21:12] I don't even know if there's a question [21:14] here but like h how is it like like the [21:17] room fits like how is it that it's three [21:19] 3D consistent? Is it just lots of data [21:22] or [21:22] >> Yeah. I mean it's a part partially it's [21:24] a belief in the scaling hypothesis right [21:26] like you know did you by the way but I [21:28] have to ask when you set out to do this [21:30] did you know it was going to work? [21:32] >> I was pretty sure [21:33] >> that's so so [21:37] were you sure? I think three of us have [21:40] total conviction about the scanning law [21:43] that that I think we do. I do think the [21:46] exact architecture choices and data [21:49] mixtures is where the the devils are in [21:52] the details. I you know have watched [21:56] Justin and his team going from we really [21:59] don't know how long this is going to [22:01] take to oh maybe side of life to wow [22:05] this is going to work. So it it no one [22:09] no one has done it but I think the [22:11] hypothesis two hypothesis one is scaling [22:13] law hypothesis the other one is uh next [22:16] viewpoint prediction we had uh [22:18] conviction of these two things from [22:20] early on [22:20] >> so I think I was very convicted that it [22:22] was going to work I was not sure it was [22:24] going to work this well at this fast [22:26] right like I thought there's a chance [22:27] that we do this maybe it's not clear [22:29] that like the first cycle of [22:31] pre-training a new model with a new [22:32] architecture and a new paradigm like the [22:34] first cycle of that working is insane [22:36] So, I thought there was a chance in [22:38] which we had to we might have had to do [22:39] a couple more turns of that of that [22:40] pre-training cycle before we got to the [22:42] level of quality we were we wanted. Is [22:44] there is it are we kind of like at the [22:46] end of like the scaling for this [22:48] architectural approach? We need another [22:49] breakthrough or is there [22:51] >> No, no, we're at the beginning [22:52] >> really without changing without changing [22:54] the architecture. [22:55] >> I think we're basically at the [22:55] beginning. I think we're basically at [22:56] the beginning and we're basically [22:57] limited by compute at this point, right? [22:59] Like data is very important as FE likes [23:01] to point out but like everything has a [23:02] bottleneck and I think the main [23:04] bottleneck on continuing to scale this [23:05] thing is actually training compute right [23:07] like during development we trained a [23:09] sequence of models we read about this in [23:10] the blog post a little bit but we [23:12] trained a couple models the like the [23:14] first couple rungs of the scaling ladder [23:16] um and each time we made the model [23:18] bigger and each time we trained it for [23:19] longer each time we put it on more chips [23:21] like it got significantly better [23:23] >> and the model size that we like the [23:25] model that we showed in the blog post is [23:26] obviously the biggest and best one that [23:28] we trained but The thing that was [23:29] limiting it was not the scale or the [23:31] data or anything like that. It was [23:33] literally like we had a deadline of when [23:34] we wanted to release this thing and [23:35] therefore we backed up what we could [23:37] afford to train and that deadline. But [23:38] here is a little bit of an insider story [23:41] right like Justin and team are are [23:43] training the from the smaller and [23:45] slightly bigger you know are train [23:48] having these road maps and then there [23:50] was one day in summer early summer that [23:54] it's not even this the current atlas [23:57] model size it's a smaller model and then [24:00] Ben Justin Ben feed it into you know the [24:02] viewpoint generation and remember that [24:06] famous table the garden table for Nerf [24:09] paper and many papers that [24:12] >> overnight I got a slack. I mean we all [24:14] saw the slack from Ben that [24:16] >> the flight flew through under the table [24:19] >> with the soccer ball. [24:20] >> Yes, with the soccer ball. [24:21] >> Is the soccer ball emergent or is that [24:23] in the original [24:23] >> picture? It's in it was real, right? [24:26] >> Okay. [24:27] >> That morning the three of us looked at [24:30] each other in the eyes and say that's [24:31] it. This is we're going to build this. [24:34] Like we we made a decision within five [24:36] seconds. how this is just a no one has [24:39] ever seen this result. [24:41] >> Ben, um can you talk through maybe more [24:44] specifically the use cases? So, so uh [24:48] World Labs has historically had a lot of [24:49] users that were creatives and they use [24:51] it for like consistency in you know like [24:54] whatever 2D images and for movies and [24:56] for 3D and for games etc. And so maybe [24:59] can you talk about how this extends use [25:01] cases or cater to the existing ones and [25:02] then we'll I'd like to talk about [25:03] robotics actually. [25:04] >> Yeah, sure. Um, yeah. I mean, it's kind [25:07] of funny actually. One of the kind of [25:09] main ways we even saw people using [25:11] marble plays exactly into this new view [25:13] prediction case. Like a lot of our [25:15] >> marble being the previous, [25:16] >> sorry, Marvel our previous product. [25:17] >> Um, like people would take that product, [25:19] put an image in, get a full 3D scene as [25:21] a gausian splat, take a couple [25:23] screenshots of it from different points [25:25] of view and leave, right? And we're [25:26] like, [25:27] >> we can just make those images and that's [25:29] in data role, right? So, so I think like [25:31] and you know there's a lot of [25:32] degradation there. They're like, oh, [25:33] this flag could look better. it's like [25:35] okay what if we just generatively model [25:36] those viewpoints with that exact [25:38] mentality of control. So I think like [25:39] even that core capability of just like [25:42] view synthesis um it's sort of been this [25:45] academic problem for a long time but in [25:46] the sense of oh you're going to do this [25:48] really dense capture like like [25:49] generative view synthesis is a [25:50] relatively quite a new problem and we [25:52] just see so many people who uh in this [25:55] creative pipeline right people have a [25:57] multi-stage workflow right I don't think [25:58] there's a single person out there using [26:00] one monolithic model not even cance or [26:02] whatever for for their entire uh task uh [26:04] people will have this like you know kind [26:07] of bunch of storyboards and mood boards [26:08] of images they pull out from like their [26:10] favorite collection of image models and [26:11] then they'll go to different video tools [26:13] and like pull those together as key [26:14] frames and then they'll go and like clip [26:16] and edit those later, right? So, we were [26:17] seeing this like sort of, you know, [26:19] niche but very specific use case for [26:21] marble as just providing that like [26:23] sanity that you can ground your [26:24] generations in some kind of 3D [26:26] consistent world, right? people, you [26:28] know, I know I' I've fought with with [26:30] various image models to ask them to like [26:32] give me different viewpoints of a room [26:33] and every time you can just look and [26:35] see, oh, things kind of moved around [26:36] like it's not stable and like even that [26:38] one seed of a use case I think kind of [26:40] signals that there's this value and [26:41] there's hiding under the surface there [26:43] like there's just decades of people [26:46] being used to persistent 3D state like [26:49] virtually modeling what they would be [26:50] doing in the real world and having you [26:53] know a stage and props and like elements [26:55] there whether it is or uh a movie or a [26:58] show or marketing shot or like building [26:59] out game environments like this this [27:02] statefulness and persistence is so key [27:04] in how people think about spatial [27:06] reasoning and like developing an [27:08] environment over time. Like people don't [27:09] think in this ephemeral like generate a [27:11] thing, generate a thing, like just throw [27:12] it away, keep my text prompts. Like [27:14] people want to build this like [27:15] collection of assets and like model a [27:17] world in that way. So we're trying to [27:18] provide like again with with this [27:20] spatial context mechanism and other [27:22] things like we're trying to provide that [27:23] level of control and precision uh and [27:26] the ability to ingest different [27:28] modalities of input starting with the [27:30] pose images but you know we want to give [27:31] people more control over the elements of [27:33] the things in the scenes you're looking [27:35] at and editing and interaction all that [27:37] as we go forward. Um, and I think that [27:38] that it unlocks like further use cases [27:41] in those areas we're already seeing, but [27:43] um, also expanding out into kind of any [27:45] place people want to create a virtual [27:47] replication or or like you know a [27:49] pre-imagination of a real world space [27:51] they need to build, right, for [27:52] architecture and construction. Like I [27:54] talked to a guy at some point building [27:56] booths for conferences, right? There's [27:57] just so many things in the world you [27:58] don't think about need to be fabricated. [28:00] And every single one of those basically [28:02] goes through this like pretty [28:03] painstaking virtual design phase. Uh, [28:05] and of that process, like the part where [28:08] you go into 3D software is kind of one [28:10] of the most like arduous and like labor [28:12] intensive parts right now. Like like [28:14] taking feedback on a 3D design from kind [28:18] of like verbal commentary or sketches or [28:20] really really quick um stuff you got [28:22] from like a creative director or like a [28:25] design director or an architect or [28:26] whatever. Like mapping that back into [28:28] the 3D representation is like 95% of the [28:31] work, right? you can have a meeting and [28:32] get feedback and then you go back and do [28:34] a weaker revisions. And that's just [28:35] because like our software is kind of [28:37] decades old at this point. And it's it [28:39] just never became as intuitive as you [28:41] know playing with Legos or like pottery [28:43] or doing this stuff with your hands or [28:44] sketching with a pencil. Um and this is [28:46] the real place where AI can actually [28:48] unlock a ton of value for people in [28:50] their process whether it's a creative [28:52] application or something more industrial [28:53] or design or or whatever. Uh, and that [28:56] like really motivates me to kind of [28:58] build different flavors of of our model [29:00] to cater to those kind of people. [29:02] >> Yeah, I I I can understand how it helps [29:05] with the creatives because like Marble [29:06] did that and also how that extends to [29:08] things like design or architecture. Uh, [29:10] but FE you acquired a robotics company. [29:12] So, it's [29:13] >> we just talked about it too. [29:15] >> I know I know but it's less obvious to [29:17] me especially in the context of Atlas [29:18] like how that maps to robotics. So, if [29:21] you wouldn't mind just penciling that [29:22] out. Yeah, actually Atlas is a a key [29:25] part of the puzzle. So, um, we acquired [29:28] this company that was formerly known as [29:31] Scenix and what is their key technology? [29:33] Right now, their key technology is a [29:35] system that goes from real to SIM and [29:38] and then SIM to real. And what does that [29:40] mean in robotics situation? You want to [29:43] train a robotic arm to, you know, uh, [29:46] figure out how to um, do cabling, let's [29:49] say, in a in a industrial setting. Well, [29:52] you need a whole bunch of data to first [29:56] train a robotic policy to do these cable [29:59] cables cabling activity and then you [30:02] want to evaluate if the robotic policy [30:05] is doing a good job and then you deploy [30:07] the robot into the cabling environment. [30:10] Yeah. in order to train what you what [30:13] this company uh uh Cynics and now our [30:16] robotics team uh used to be doing is [30:19] doing exactly what Ben was saying dense [30:21] reconstruction. You take, you know, [30:24] pictures of a situation and then and try [30:28] to reconstruct that environment is [30:30] excruciatingly painful, takes a long [30:33] time, laborious, and it really blocks [30:35] the velocity of robotic simulation, real [30:39] to sim, right? So, Atlas really is the [30:42] next generation technology for for that. [30:46] And this is not just for robotics [30:48] cabling or anything. We should zoom out [30:50] and recognize the biggest problem right [30:54] now in robotics is actually data. One [30:57] day it'll be chips but for now it's data [31:01] >> because it's so hard to collect real [31:04] world data where robots you know are [31:07] operating on and uh and uh in order to [31:12] uh not only you need to uh collect the [31:15] data of let's say the cabling situation [31:17] or dishwashing situation or whatever [31:19] there is also a very important step [31:21] called randomization is that you have to [31:24] take the same environment and then [31:26] randomize the condition. So the cable [31:28] doesn't literally only you know bend [31:31] this way. It can bend a different way or [31:33] the box can have different sizes, [31:35] colors, different lids and all that and [31:37] and or in different parts of the scene. [31:40] So you have to go through a real to sim [31:44] situation in order to get enough of that [31:46] data in addition to other data you can [31:49] get from internet. So this real to some [31:52] uh step will be um you know really [31:55] helped by by Atlas. That's just the [31:58] first part of this is meeting the the [32:00] the robotics needs in the current [32:03] technology because we don't yet have a a [32:05] frontier foundation model that's robust [32:08] enough for for uh robotics. But agless [32:11] is a uh omnimodel is a multi uh um [32:14] multimodal model. It takes a different [32:17] kinds of input and generates different [32:20] kind of output. You can totally imagine [32:23] the next step is at this um taking in d [32:27] uh data that's in the uh in that's [32:30] dynamical and that can really start to [32:34] bridge the gap between you know action [32:37] planning and uh and um [32:41] and and robotics and and the atlas uh [32:44] output. So that's on that road map. Were [32:47] you going to say? [32:48] >> Yeah, I was going to say there's [32:49] something fundamentally different about [32:50] training a robotics policy compared to [32:52] really any other application in AI we've [32:53] seen before. Um, and that's like if [32:55] you're generating a piece of code like [32:56] you're generating an image, you're [32:57] generating a video. I'm I the model is [33:00] fundamentally creating this this [33:01] artifact and that artifact like there's [33:03] a lot of examples of artifacts that you [33:05] can go out on the web or somewhere and [33:06] collect, right? You want to generate [33:08] images, there's a lot of images out [33:09] there. You want to generate videos, [33:10] there's a lot of videos out there. You [33:11] want to generate a codebase, there's a [33:12] lot of code bases out there you can [33:13] learn from. [33:14] >> Yeah. A robotics policy is something [33:15] fundamentally different. It's not [33:17] producing a static thing. It's instead a [33:19] policy that's going to go out into the [33:21] world, make actions, and like try to [33:23] achieve a goal. And the world's not [33:25] always going to respond the way you [33:26] expect, right? Unexpected stuff is going [33:28] to happen. So a robotics policy is like [33:30] fundamentally an agent that is out in [33:32] the real world interacting with the real [33:34] world and stuff happens. So you need [33:37] like a critical part of that is th those [33:39] those policies during training need to [33:41] be exposed to every possible thing that [33:43] could go wrong during their deployment. [33:46] >> Um and that's where simulation is really [33:48] key for robotics, right? So there's the [33:50] and then there's two angles on that like [33:51] one is the kind of classical simulation. [33:53] You can go out, you can go and like go [33:56] to your favorite physics engine and like [33:57] try to imagine creatively as a human [33:59] designer what are all the scenarios that [34:01] might happen in this in this in this uh [34:03] when achieving this task and then try to [34:05] write explicit code that models them [34:07] all. That that's one angle. Um and [34:09] that's an interesting angle with coding [34:10] agents like that actually gets [34:11] supercharged too. But there's another [34:13] angle which is try to more data driven [34:15] simulation, right? Like maybe we can't [34:17] can we have a learned model that can [34:19] understand how the environment how the [34:21] world is going to respond to actions and [34:23] maybe it might respond in unexpected [34:25] ways sometimes. Then could we build [34:27] these neural simulators that are trained [34:29] on as much data as we can then use these [34:31] neural simulators these learned neural [34:33] simulators you know as a simulation bed [34:35] to train robotic policies. Um and that's [34:38] that's a really interesting future [34:39] direction of Atlas. But but then it [34:41] doesn't stop there, right? And so so um [34:45] but once you have you know this learn [34:47] simulator like this learn simulator kind [34:49] of already has in its like mental brain [34:51] like it understands the world. It [34:53] understands how the world is going to [34:54] respond to actions and why doesn't the [34:56] simulator itself become the planner, [34:59] >> right? the same and that's kind of the [35:00] core thesis that we've had around world [35:02] models and their generality that there's [35:04] some core stuff that a model should [35:06] understand around generating worlds, [35:08] simulating them, understanding how they [35:10] appear in different situ situations and [35:12] you know understanding how the world's [35:14] going to respond to an action is highly [35:16] related to imagining what kind of action [35:18] I need to take to make the world respond [35:20] in a particular way. [35:21] >> Yeah. [35:22] >> What what um one piece of feedback that [35:25] I got, by the way, congrats on the [35:26] launch. It was overwhelmingly positive. [35:28] I think it was probably the most [35:29] significant model launch this year and [35:31] you know every said glowing things but [35:33] one person who's an expert in the space [35:35] who I texted was like what do you think [35:38] and great person is fantastic it's [35:40] amazing but there needs to be more [35:41] dynamics [35:43] and so it would seem to be at least in [35:45] the robotics case but generally it was [35:46] kind of ideal to actually have a world [35:48] that moves and so maybe talk a little [35:50] bit about that and then any other future [35:53] directions that a you're comfortable [35:54] sharing but you think are worth talking [35:56] through [35:56] >> yeah I mean like dynamics is clearly [35:57] going to happen. Like actually um [35:59] >> we have baby dynamics. [36:00] >> We actually do have baby dynamics [36:01] already. And this is something I think [36:02] people didn't quite appreciate. We [36:04] didn't really highlight in the blog [36:05] post, but like the previous Marble World [36:07] model, it was like fundamentally static. [36:08] Yeah. Like the the model just like could [36:10] not handle any dynamics at all. And that [36:12] was just like baked into the model [36:13] architecture, baked into the training. [36:15] Like the whole thing was fundamentally [36:17] static. [36:17] >> Um we already knew that that was a big [36:19] problem post Marvel and we already fixed [36:20] it in Atlas, right? Like the Atlas [36:22] architecture is already fundamentally [36:24] supports dynamics. Um the Atlas training [36:26] data fundamentally has dynamics. And if [36:28] you look carefully in some of the videos [36:30] that we even posted, it actually is [36:33] moving a little bit. [36:34] >> Yeah. The waves, the water waves. [36:36] >> Yeah. So like some of the examples, [36:37] there's like waves in the water. Like in [36:38] some of the like air generated aerial [36:40] views, there's like little cars moving [36:41] around. So like dynamics is actually [36:44] already in this model. [36:45] >> But by the way, dynamics seems very [36:47] problematic to me if you're trying to [36:48] reconstruct 3D from multiple views, [36:50] right? And so like are these things like [36:52] at odds or [36:53] >> No. So actually one of our thesis here [36:55] is that like you know if you're going to [36:56] do fundamental 3D reconstruction you [36:57] actually want to have no dynamics like [36:59] you want to be able to model like exact [37:01] views of the scene with exact frozen [37:02] time. Yeah. Um but then like this is [37:04] actually kind of a problem with our [37:06] previous marble approach, right? Like [37:07] there like you can try to find data [37:10] that's fully static, but that's really [37:11] hard to scale and really hard to get [37:13] more of. And the thing we realized is [37:15] that even in the case where I want [37:17] static output in the end, the best way [37:19] to get it is actually expose the model [37:21] to dynamics, right? Like expose the [37:22] model to as much dynamic stuff as you [37:24] got, as much static stuff as you got, [37:26] and let the model figure out how to [37:27] factor out the dynamic stuff. So in [37:29] especially in like the code and this is [37:31] like and this is actually so in in the [37:33] Atlas pre-training already like it it [37:35] saw a ton of dynamics in the [37:36] pre-training already then the post [37:38] training that we did specific to this [37:39] checkpoint in this release was focused a [37:41] lot more on on static stuff focused a [37:43] lot more on spatial movement and less on [37:45] temporal but like we already have we [37:47] already like I'm pretty sure this the [37:49] pre-train checkpoint already has a lot [37:51] of latent dynamics in it um and this is [37:53] something we're going to improve quite a [37:54] lot going forward. So, so Ben, does that [37:56] mean we're going to get 40 video like go [37:58] walk around? [38:00] >> I mean, [38:00] >> you can see the smile on their face. [38:04] >> So, I I actually can see if like you [38:06] just stopped now and you only did kind [38:08] of bigger, you know, faster, better, you [38:12] could build almost an entire industry, [38:13] right? I It feels like a very horizontal [38:15] primitive. And then and if you did [38:16] nothing else, but are there other things [38:17] that are not just kind of bigger, faster [38:19] that you're excited about for the [38:20] applications you're focused on, which [38:22] tend to be kind of more on the kind of [38:23] content creative 3D side? Yeah, I'm [38:24] really excited about pushing that kind [38:26] of multimodal aspect. I think different [38:27] modes of control is so critical here. I [38:29] think like it's super underappreciated, [38:31] especially in the academic community, [38:33] how critical it is to add control [38:35] conditioning to these models to kind of [38:36] get out what's inside. I mean, honestly, [38:38] this dynamic versus I'm not sure I even [38:40] understand what those [38:41] >> to translate in layman's language is [38:44] editability. I think editability is is [38:47] the key here. [38:48] >> Yeah. So I mean this is something we've [38:49] seen in like sort of single image models [38:51] and starting this year in video models [38:53] is starting to be unlocked in terms of [38:55] oh like getting that flavor of like [38:58] multi-turn or really like intuitively [39:00] interpreting like I want like this [39:02] person and this object and this thing to [39:04] happen and kind of combining those all [39:05] together in like one paste without [39:07] having to do a lot of like manual work [39:09] with the system like it just interprets [39:10] it like kind of frontier image models [39:13] are kind of there right for in terms of [39:14] editing but we haven't seen that [39:15] propagate out as strongly into video yet [39:18] and and into world models, right? We've [39:19] seen some really kind of toy examples [39:21] of, oh, I can like put in a sentence and [39:24] like, you know, a dinosaur appears or [39:25] something with these like sort of [39:26] real-time models. Um, but I want to like [39:28] turn that up to really industrial [39:30] strength and make that cuz like the [39:31] trick here is you got to add control but [39:34] not compromise the quality of the model [39:35] or it just becomes a party trick [39:37] basically. Like it's like no one is [39:38] going to seriously think about swapping [39:41] their like cutting edge frontier video [39:43] model usage for your model if you give [39:45] them extra knobs, but the quality [39:47] degrades. So, I think it's really that [39:48] game of like how can we maintain like [39:50] the high bar we've set was the outputs [39:52] we're able to get in the current model [39:53] and then add all kinds of interesting [39:55] stuff that people will ask us for in [39:57] terms of like I want to interact with [39:58] the scene or control the layout or [40:00] control like the identity of the objects [40:02] and the things that we're seeing within [40:03] there or control time right um and I [40:06] think that's like an access where it [40:07] opens up like a ton of really [40:09] interesting product and interface work [40:10] the more complexity you add there and [40:12] richness in terms of kind of like [40:14] enabling you to really think about like [40:17] redesigning almost from scratch the way [40:18] people interact with sort of like [40:21] stateful, you know, 3D worlds in the [40:23] computer. Like that's that's really the [40:24] end goal here is getting like all the [40:26] capabilities you need to build that kind [40:28] of system. [40:28] >> Awesome. Anything you'd add to that as [40:30] far as new functionality that you'd be [40:32] excited about that's not just big or [40:33] better. [40:34] >> I think for me, let's go back to the [40:37] first principle of intelligence. [40:39] Intelligence is not sitting there stack [40:41] and just seeing something or [40:43] interpreting something when it comes to [40:44] space and physical space, right? It's [40:47] really this uh closing the loop between [40:50] seeing and experiencing and interaction. [40:52] So just thinking about going up that [40:54] ladder is exactly what uh Ben said. [40:56] >> Cool. [40:57] >> I think one interesting notion there is [40:58] this notion of AI completeness. Have you [41:00] heard this before? [41:00] >> Yeah. Yeah, I have. Yeah. [41:01] >> So like everyone [41:02] >> I I hear about AI complete by the way in [41:04] terms of LLMs which is like you have to [41:06] be basically you know like the smartest [41:08] LM to answer the question what the [41:10] smartest LLM will need to answer or you [41:12] have to solve general intelligence. No, [41:14] no, it's basically it's a it's a [41:15] connection to turning completeness, [41:16] right? Like the idea being that like if [41:18] a task is turning complete like in [41:20] classical complexity theory if like I [41:22] can take any class any problem in this [41:24] category reduce to that one problem, [41:26] right? Three example, right? So you can [41:28] take any np hard problem then reduce it [41:29] to three therefore you can use threes to [41:32] solve any any problem. [41:33] >> Yeah. It's Yeah. So, so then like the [41:35] the kind of like the soft definition of [41:37] AI completeness is like there's this [41:38] fundamental primitive that's it's an AI [41:40] task but if I could solve this AI task [41:42] in its full broadest generality it would [41:45] solve any intelligence problem and like [41:46] the classic example at LLM is like next [41:48] token prediction is AI complete because [41:50] I could like you know there's the [41:52] classic example I think from Ilia where [41:54] like I there's a mystery novel and like [41:55] the thing has to read the whole mystery [41:56] novel and the final sentence of the [41:58] mystery novel is like and the killer was [42:00] predict the next token. So like you [42:02] could basically like frame any kind of [42:03] intelligence task in terms of that. So [42:05] clearly next token prediction is [42:06] something that people believe is AI [42:08] complete. [42:08] >> But I think that something we're kind of [42:10] realizing and Ben was talking about this [42:12] earlier today is like new view [42:13] prediction this primitive that we have [42:15] in Atlas especially generative new new [42:17] view new view prediction. This is also [42:19] AI complete right and because I could [42:21] take something like [42:22] >> you could you could have the the movie [42:24] and you do all of the frames of the [42:26] movie and then like the killer walks out [42:28] and then you predict exactly who walks [42:29] out. [42:30] >> Exactly. Exactly. Not just that could [42:32] say like I want I want to have a world [42:34] where like Martin is like writing a [42:36] proof of the remon hypothesis on the [42:37] back and the camera over to the next [42:39] white solves the [42:42] okay so to take a evolutionary view [42:45] right that new viewpoint uh prediction [42:48] is exactly evolution had to solve by [42:51] making animals move [42:54] >> you you nature give animals eyes [42:58] >> but nature didn't give trees these eye [43:02] >> eyes why because when you move you see a [43:05] new viewpoint [43:06] >> and that is the the the whether you call [43:08] it AI complete or intelligence complete [43:11] so so we do believe very strongly that [43:14] next viewpoint prediction is is the [43:17] equivalent of next token prediction [43:19] >> amazing well with that congratulations [43:22] all of you on a phenomenal model launch [43:24] we're very excited for future model [43:25] launches and thanks for coming so