Fei-Fei Li’s World Labs Launches Atlas to Unify 3D Capture and Video

李飞飞创立的 World Labs 推出 Atlas，统一 3D 捕捉与视频 创始人指出，下一视角预测是等同于下一词元预测的 AI 完全图元，可通过神经模拟解决机器人数据瓶颈。 World Labs 联合创始人李飞飞、Justin Johnson 与 Ben Mildenhall 详细介绍了他们的新型世界模型 Atlas。该模型用基于相机的“新视角预测”取代了传统的下一帧视频生成。通过将开放式生成扩散与确定性 3D 重建相结合，该模型仅凭少量稀疏的智能手机抓拍照片，便能合成完整的 3D 环境、相机导向的漫游运镜以及时间定格的“子弹时间”效果。创始人们讨论了 3D 领域的扩展定律，摆脱了前代模型 Marble 中受制于高斯泼溅输出瓶颈的设计，并探讨了视频数据中的潜在动力学如何助力分离出静态几何结构。最后，他们解析了世界模型如何开启从现实到模拟（real-to-sim）的管线以应对机器人的数据瓶颈，并提出预测新视角是一种直接等同于下一词元预测的 AI 完全图元。 World Labs 推出了旗舰世界模型 Atlas，该模型建立在“新视角预测”之上，在单一架构中统一了生成式图像建模、3D 重建与模拟。 (0:00) 通过将几何三角测量与生成先验相结合，Atlas 将高保真 3D 场景重建所需的输入密度降低了 50 到 100 倍，仅需 3 到 25 张稀疏照片即可工作。 (13:14) 与其前代模型 Marble 将输出严格限制在 3D 高斯泼溅不同，Atlas 将新视角预测视为一种自回归图元，能够跨 3D 网格、RGB 视频和潜在动力学进行扩展。 (10:00) 创始人认为，下一视角预测是一种等同于大型语言模型中下一词元预测的 AI 完全视觉图元，并且在生物学上契合了移动生物为导航演化出视觉的机制。 (40:53)

World Labs co-founders Fei-Fei Li, Justin Johnson, and Ben Mildenhall argue that spatial intelligence requires shifting AI's core visual primitive from ungrounded 2D pixel generation to "new view prediction." Their model, Atlas, unifies 3D reconstruction and generative imagination to build geometrically consistent, interactive environments from sparse physical viewpoints. New View Prediction as a Physical Primitive (0:00) Unlike standard video diffusion models that generate frames sequentially without spatial constraints—often resulting in an unpredictable "slot machine" effect—Atlas pairs every input view with an explicit 3D camera pose. This spatial context window allows users to specify virtual camera trajectories and predict exactly what an environment looks like from any commanded vantage point. Justin Johnson notes that this paradigm enables effects like the Matrix's "bullet time" using just three consumer iPhones, eliminating the traditional need for hundreds of synchronized studio cameras and green screens. Bridging Dense Triangulation and Generative Hallucination (13:14) Classical computer vision techniques like photogrammetry and Neural Radiance Fields (NeRF) depend on dense reconstruction, requiring 100 to 300 photos per room to triangulate points and leaving holes wherever occlusions occur. Conversely, purely generative models imagine missing visual data but lack structural coherence. Atlas bridges these paradigms by combining geometric triangulation with generative spatial priors, slashing capture requirements by 50× to 100×. In one demonstration, the team reconstructed the entire Stanford Quad from 3 to 25 ground-level phone snapshots, accurately extrapolating a high-altitude aerial fly-through. Architectural Shift Beyond Output Representation Bottlenecks (10:00) World Labs' earlier model, Marble, ingested text and imagery but bottlenecked its output by forcing scenes exclusively into 3D Gaussian splats. Atlas redesigns this framework by establishing autoregressive new-view prediction as its foundational primitive, natively pre-training on images, video, text, depth maps, and camera poses. This architectural shift allows the model to simultaneously synthesize RGB video, reconstruct underlying 3D geometry, and target optional formats like Gaussian splats, with Justin Johnson emphasizing that early prototypes scaled cleanly with added compute. Spatial Grounding for Simulation and Robotics (28:34) Bridging simulation and physical reality remains an acute bottleneck in robotics because training contact-rich manipulation tasks on physical hardware is costly and slow. Fei-Fei Li highlights that Atlas provides the real-to-sim foundation needed to turn sparse physical imagery into editable digital twin environments. Justin Johnson outlines two routes forward: classical human-coded simulation engines or learned neural simulators. By training world models to understand how physical environments react to dynamic interventions, researchers aim to turn learned simulators into direct action planners for embodied agents. Viewpoint Prediction as an AI-Complete Task (40:53) The founders frame generative new view prediction as an AI-complete primitive structurally equivalent to next-token prediction in large language models. Drawing an analogy to computational complexity theory, Johnson argues that complex reasoning can be reduced to visual prediction, such as generating the visual sequence that reveals a mystery culprit or shows an agent writing out a mathematical proof. Fei-Fei Li grounds this concept in evolutionary biology, noting that active vision evolved specifically in mobile animals to resolve shifting viewpoints during locomotion, reinforcing the view that spatial interaction is central to general intelligence. While the Atlas team reports strong geometric consistency and early signs of dynamic movement, achieving full spatial intelligence requires expanding fine-grained control and 4D temporal simulation. Unresolved challenges remain in scaling post-training dynamics and introducing robust scene editability without degrading visual fidelity.

World Labs 团队（Fei-Fei Li、Justin Johnson 与 Ben Mildenhall）主张，空间智能的核心基元并非大语言模型的下一个词元预测或常规视频模型的下一帧生成，而是“新视角预测”；其旗舰世界模型 Atlas 通过将几何重建与生成式先验融为一体，试图打通从 3D 场景理解到具身机器人的物理模拟路径。 新视角预测与多模态架构的统一 (0:00) 传统计算机视觉长期将几何三维重建与生成式扩散模型割裂开来：前者依赖多视角三角测量，虽具几何确定性却无法填补遮挡盲区；后者虽具备开放想象力，却因缺乏物理坐标锚定而产生严重的视角漂移与“老虎机效应”。Atlas 在预训练阶段即原生吸收文本、RGB 图像、视频、深度图与显式三维相机位姿，将生成、重建与模拟整合在单一架构内。 与上一代将输出强制限制在 3D Gaussian Splatting（高斯泼溅）的 Marble 模型不同，Atlas 将“新视角预测”确立为更底层的基元，使模型能依据显式相机轨迹直接推断出空间任意坐标上的对应渲染。由于摆脱了固定输出表示的结构瓶颈，系统既能输出可交互的动态视角视频，亦可向下编译为高精度的 3D 网格或高斯泼溅场景。 稀疏视角重建与长上下文空间窗口 (13:14) 在经典摄影测量和 NeRF（神经辐射场）管线中，构建单间房屋常需 100 至 300 张高密度照片，整栋建筑甚至需要多达 2,000 张，且任何未被多视角覆盖的死角都会产生空洞。Ben Mildenhall 指出，Atlas 通过引入生成式空间先验，将所需输入量削减了 50 至 100 倍。例如在斯坦福四合院测试中，仅凭 3 至 25 张地面手机抓拍，模型便能推断并生成从空中俯瞰的完整飞跃轨迹。 该机制在技术形态上类似于大语言模型的长上下文窗口扩展。Atlas 允许用户以自回归方式将生成的新视角图像不断追加进空间上下文，以此支持如 64 张稀疏照片跨越整栋房屋的大场景推断。对于视野外被遮挡的物体背面或家具底部（如初代 NeRF 经典的花园桌下方足球），系统能够自动依据物理常识完成几何一致的生成补全。 具身智能与物理模拟的数据引擎 (28:34) Fei-Fei Li 认为当前机器人学面临的核心瓶颈在于物理交互数据的极度匮乏，直接在实体硬件上采集布线、洗碗等接触密集型操作的成本过高且难以规模化。Atlas 提供的稀疏重建能力大幅降低了“实转仿”（real-to-sim）的门槛，使得工程团队能够快速将少量现实照片转化为具备域随机化（改变光照、刚度、物体姿态）的数字孪生场景，供机器人在仿真中完成数百万次试错后再“仿转实”（sim-to-real）。 Justin Johnson 进一步区分了两种模拟路线：一种是依赖人工或代码智能体显式编写物理规则的传统物理引擎；另一种则是完全由数据驱动、直接学习世界对动作做出反应的神经模拟器。团队设想，世界模型在完全掌握环境推演规律后，将不再局限于充当离线训练沙盒，而是能直接演变为实体机器人的动作规划器。 潜在动力学与交互编辑的产品演进 (35:00) 针对外界对 3D 几何重建难以兼容时间维度的质疑，团队阐明静态重建与物理动态并非对立。虽然 Atlas 首个公开发布版本主要微调用于静态空间漫游，但其预训练过程直接灌入了海量动态视频数据（如流水波浪和行驶车辆），模型由此学会了在解耦运动的同时捕捉静态几何。这类潜藏在底层权重中的“幼年期动力学”构成了通向 4D 动态交互视频的基础。 在商业与产品界面维度，Ben Mildenhall 强调高保真度下的“可编辑性”是取代传统耗时 CAD 工具的前提。目前 2D 图像生成已实现多轮细粒度编辑，而 3D 与视频生成仍停留在粗放的原型演示。专业创作者需要的是具备持久空间状态的资产调控能力，能在不降低渲染质感的前提下直接调控物体属性、环境布局与时间流向。 AI 完全性与生物演化的视角预测假说 (40:53) 团队从理论计算视角将“新视角预测”提升到了 AI 完全性（AI-completeness）的理论高度，将其视为与大语言模型“下一个词元预测”完全等价的通用基元。正如预测小说结尾能够迫使系统推断出凶手身份，给定相机轨迹推断下一个视角同样能涵盖极复杂的通用认知任务，例如在思想实验中要求模型生成“黑板上完整写出黎曼猜想证明过程”的空间视线。 Fei-Fei Li 从演化生物学角度补充了这一假说的合理性：具备移动能力的动物进化出了复杂的视觉感知系统，而静止的植物则不需要眼睛，其根源在于生物在物理空间中的自主移动必然要求大脑持续预测与矫正全新的视觉透视。因此，闭环感知、空间预测与实体交互构成了通向通用空间智能的基石。 World Labs 团队确信空间智能的发展正处于算力受限而非架构见顶的初期阶段；但如何使模型在保持极高视觉保真度的同时，跨越静态场景走向高精度的 4D 动态物理模拟并提供工业级可编辑界面，仍是该体系面临的核心挑战。

## Background 本期内容由 a16z 普通合伙人 Martin Casado 与 World Labs 联合创始人李飞飞（Fei-Fei Li）、Justin Johnson 及 Ben Mildenhall 共同参与，于 2026 年 9 月 4 日发布于播客平台。节目主要围绕 World Labs 最新发布的世界模型 Atlas 展开，深入探讨了其核心原语“新视角预测”背后的技术构想、空间智能的发展路径，以及该模型在创意设计、建筑与机器人仿真等领域的潜在应用与现实挑战。 ## TL;DR 对话深入拆解了 World Labs 旗舰世界模型 Atlas 的底层逻辑，阐述了其如何将 3D 重建与像素生成原生统一在“新视角预测”这一核心基元之下。团队回顾了从前代产品 Marble 向 Atlas 演进的技术抉择，展示了仅凭 3 台普通 iPhone 即可重现《黑客帝国》“子弹时间”等高保真空间渲染突破，并将传统 3D 捕捉的数据需求降低了 50 到 100 倍。李飞飞与团队进一步探讨了世界模型在机器人 Real-to-Sim 管道中的关键价值，指出当前最大的瓶颈在于现实数据与训练算力，而非模型架构本身。最后，团队从计算复杂性与演化生物学的第一性原理出发，论证了“下个视角预测”正是空间智能领域等价于大语言模型“下一个 Token 预测”的 AI 完备性基元。 ## Atmosphere & style 整场对谈洋溢着严谨、前沿且极具信心的科研极客氛围，既有顶尖学者对学术第一性原理的深度推导，又不乏创业团队在算力极限与工程落地间反复试错的真实脉动。Martin Casado 作为主持人和投资人，带着敏锐的好奇心不断追问技术分水岭与商业落地痛点；李飞飞、Justin Johnson 与 Ben Mildenhall 则展现出高度默契的技术协同，生动分享了半夜渲染出“花园桌下的足球”等令人战栗的顿悟时刻。听众能清晰感受到一种见证历史技术拐点的激动情绪，不仅能领略高深的空间几何理论，更能切实体会到开拓空间智能未知疆域时的智力激荡与宏大抱负。 ## Bulletin - `00:00` World Labs 推出新一代世界模型 Atlas，将核心范式确立为“新视角预测”（new view prediction），把传统的 3D 重建与像素生成首次在同一预训练架构中实现统一。 - `02:40` Atlas 大幅降低了空间捕捉门槛，仅需 3 到 6 张稀疏照片或 3 台普通 iPhone，即可实现以往需要上百台精密相机才能完成的“子弹时间”与高质量 3D 漫游重建，采集工作量缩减 50 到 100 倍。 - `30:00` 李飞飞指出机器人领域当下的最核心瓶颈是现实数据匮乏，Atlas 通过从极少量影像重建并自动生成随机化仿真环境，打通了 Real-to-Sim 与 Sim-to-Real 的闭环。 - `41:00` 团队提出“新视角预测”具备 AI 完备性（AI-completeness），在理解与交互物理世界层面，它是大语言模型中“下一个 Token 预测”（next token prediction）的空间等价物。 ## Key takeaways - `00:00` Atlas 建立了世界模型的三大核心能力基石：生成（可控相机轨迹生成）、重建（稀疏多视角 3D 场景重建）与模拟（子弹时间特效及机器人动作仿真）。 - `05:00` 摆脱扩散模型的“老虎机效应”：Atlas 在预训练阶段原生注入 3D 相机位姿与深度图，使生成的每一帧都具备几何约束与确定性控制，而非凭空臆测空间内容。 - `10:00` 架构范式从 Marble 演进到 Atlas：上一代模型受制于将 3D 高斯泼溅作为单一端到端输出，Atlas 则转向以新视角预测为底层原语，兼顾保真 RGB 图像、点云网格及按需导出的高斯泼溅。 - `13:17` 从稠密重建迈向稀疏重建：传统 NeRF 或摄影测量要求目标点在多张图像中严苛曝光，Atlas 结合几何三角测绘与生成先验，使 3D 资产采集成本降低 50 至 100 倍并能唤醒沉睡的零散历史素材。 - `16:03` 重建本质上是被喂入大量观测结果的超长上下文生成：Atlas 将重建与生成统一在连续光谱上，通过长上下文窗口无缝推理物理空间的遮挡盲区。 - `22:15` 团队坚信空间智能的缩放定律（Scaling Law），当前架构的唯一瓶颈在于训练算力；模型在微小阶段成功穿行 NeRF 经典“花园桌”底并渲染出足球，直接坚定了团队推进全尺寸 Atlas 的决心。 - `25:00` 解决创意与建筑工业管线的持久化 3D 状态痛点：传统 2D 扩散模型的瞬态生成难以维持跨视角几何一致性，Atlas 提供了可复用且具备连续空间控制能力的设计基底。 - `30:00` 赋能机器人 Real-to-Sim 管道：针对布线或家务等复杂任务，Atlas 能低成本构建三维数字孪生，并引入光影、尺寸与形变的环境随机化，解决机器人策略训练的数据饥渴。 - `32:30` 数据驱动的神经仿真器（Learned Neural Simulators）不仅能学习环境对动作的动态物理响应，其认知本质还能直接演化为机器人的动作规划器（Action Planner）。 - `35:00` 预训练中的“动静结合”哲学：纯静态数据在现实中严重匮乏，Atlas 通过在预训练中摄入海量动态数据学会剥离动态干扰，既保留了静态几何重建的高保真度，又为未来的 4D 视频漫游奠定了潜在动态基础。 - `37:30` 世界模型迈向工业级可用的关键在于细粒度可编辑性（editability），且控制维度的增加绝不能以牺牲生成保真度与几何质量为代价。 - `41:00` 从进化生物学与计算复杂度视角，“新视角预测”在空间智能上等价于大语言模型的“下一个 Token 预测”，大自然仅赋予可移动生物双眼，正是因为视角的动态推演构成了物理交互与行动闭环的根基。 ## Detailed walkthrough ### `00:00` 空间智能的基元：从“下一个 Token”到“新视角预测” 在探索空间智能（Spatial Intelligence）的道路上，生成真正具备空间语境与物理接地的像素，是 World Labs 推出的世界模型 Atlas 跨出的极为关键且艰难的一步。大语言模型（LLM）建立在“下一个 Token 预测”（next token prediction）之上，视频生成模型普遍依赖“下一帧预测”（next frame prediction），而 Atlas 则将核心范式确立为“新视角预测”（new view prediction）。团队发现，每次将模型参数做大、训练更久，其表现都会出现显著提升。这种突破为实际工作流程带来了 50 到 100 倍的效率缩减，让人不禁思考：这是否意味着我们正走向可以自由漫游的 4D 视频世界？ World Labs 联合创始人李飞飞（Fei-Fei Li）、Justin Johnson、Ben Mildenhall 与 a16z 普通合伙人 Martin Casado 展开对话，深入拆解了 Atlas 的底层机制。团队认为，Atlas 作为世界模型具备三大核心能力：**生成（Generate）**、**重建（Reconstruct）** 与 **模拟（Simulate）**。在生成层面，它支持高度可控的相机轨迹条件生成，用户输入单张图像并附带相机运动路径，即可驱动模型渲染出任意视角的连贯视频；在重建层面，它专精于稀疏 3D 重建，哪怕只输入 1 到 100 张现实世界的照片，也能重建出完整的真实空间，输出形式既可以是穿越空间的漫游视频，也可以是显式的 3D 场景；而在模拟层面，它不仅能支持机器人动作仿真，还实现了以往只有好莱坞顶级制作才能达到的视觉特效。 ```mermaid flowchart TD accTitle: Atlas 核心能力与输入输出流程 accDescr: 展示 Atlas 将多模态输入转化为生成、重建与模拟三大核心能力的过程。 subgraph Inputs["输入条件"] A["图像 / 视频帧"] B["文本描述"] C["3D 相机位姿 & 轨迹"] D["深度图 (Depth Maps)"] end subgraph Core["Atlas 核心范式"] E["空间上下文 (Spatial Context)"] F["新视角预测 (New View Prediction)"] end subgraph Outputs["三大能力输出"] G["可控相机视角视频生成"] H["稀疏 3D 场景重建"] I["子弹时间 / 物理与机器人模拟"] end A --> E B --> E C --> E D --> E E --> F F --> G F --> H F --> I ``` ### `02:40` 三台 iPhone 重现《黑客帝国》“子弹时间” 在展示模拟能力时，Atlas 引发全网关注的标志性成果是“子弹时间”（Bullet Time）特效。Justin Johnson 解释道，这一概念源自电影《黑客帝国》（The Matrix）中基努·里维斯饰演的 Neo 仰身躲子弹的经典镜头：画面中时间被极端放慢甚至凝固，而摄影机却在空间中围绕角色大幅度高速旋转。在电影工业时代，为了实现这一秒钟的视觉震撼，摄制组必须在摄影棚内搭建绿幕，架设由数百台昂贵摄影机严密排列组成的环形阵列，并经过极其繁琐复杂的设备标定。 相比之下，Atlas 彻底打破了高昂的硬件门槛，仅凭 3 台普通 iPhone 就能完成同样的视觉效果。无需摄影棚、无需绿幕、更不需要昂贵的标定设备，研发人员只需要把三台手机支在三脚架上，从不同角度记录同一动态事件——例如投篮入网，或是将一颗草莓扔进牛奶碗中。随后，利用 Atlas 的新视角预测能力，用户可以任意重设画面取景，在牛奶激起水花的那一刹那让时间定格，驱使虚拟摄像机穿透飞溅的奶滴进行全方位环绕运镜。Justin Johnson 总结道：“针对基础模型，我们认为这是一个超级令人兴奋的全新基元，此前从未有人做过。” - 传统好莱坞子弹时间：绿幕环境 + 数百台高规格相机精密环形排列 + 繁杂标定。 - Atlas 实现方案：3 台普通 iPhone 置于三脚架上拍摄日常动态，即可在任意时空切片下自由运镜。 - 输入与输出机制：输入若干视角或文本场景描述，模型将其编码为隐式的“空间上下文”（spatial context）；用户在时空任意坐标架设虚拟摄像机，Atlas 即可精确预测该位置所见的世界形态。 ### `05:00` 彻底告别“开盲盒”：统一 3D 重建与像素生成 很多现有视频大模型也自称是“世界模型”或能生成“多视角视频”，但 Ben Mildenhall 指出，Atlas 与它们有着本质区别。以往的模型大多从单图生成或首尾帧插值起家，最近开始支持输入 20 至 50 张参考图的“全向参考”（omni-reference），但这些图像在模型内部缺乏真正的空间几何约束，模型完全凭想象去自由拼凑。而 Atlas 赋予了每一帧图像严格的 3D 相机位姿（Camera Pose）。如果在一个房间的四个角落各拍一张照片输入 Atlas，模型会输出与房间完全一致的精确几何复原，绝不会像普通扩散模型那样凭空臆测另一角有什么物体。Ben Mildenhall 将这种确定性控制与传统视频模型的随机抽卡做对比，强调这种精确度彻底摆脱了“依靠高层文本提示词反复碰运气的‘老虎机效应’（slot machine effect）”。 Justin Johnson 进一步强调，计算机视觉领域过去一直割裂为两大独立流派：一派专攻 3D 重建，追求几何精度的绝对严谨；另一派专攻图像生成（如近年的扩散模型），追求凭空想象与创意发挥。Atlas 是首个在同一模型架构内原生统一生成与重建的系统。为了达成这一目标，Atlas 在预训练阶段就将文本、图像、视频、3D 相机位姿（包括空间坐标与内在相机参数）以及空间结构深度图（Depth Maps）作为原生多模态数据一同输入训练，这在 AI 模型预训练史上尚属首次。 李飞飞对这套架构的高度统一给予了极高评价。她谈到，计算机视觉顶会（如 CVPR/ICCV）历史上一直分为图像生成、物体识别和 3D 重建等互不相融的赛道，而 Atlas 通过以视点及其位姿估计为锚点，极其优雅地统一了像素生成与像素重建。谈及空间智能的终极愿景，李飞飞表示空间智能必须赋予机器构建空间、在其中推理、进而交互与编辑的能力，其终极形态必然是涵盖时间维度的 4D 现实，但即使在 3D 层面，几何结构与物理规律也是理解空间的前提。正如李飞飞所言：“我认为 Atlas 迈出了重大的一步，因为现在伴随生成的每一帧，你都能生成并估计出一个关键信息，那就是视点与相机位姿。而这正是人们理解空间几何时所需要的最关键信息。” ### `10:00` 迈向空间智能：从 Marble 到 Atlas 的架构范式转变 飞飞（Fei-Fei Li）强调，仅仅生成像素只是通往空间智能（Spatial Intelligence）路线图上的早期一步，市面上已经有“数不清的模型”（gazillions of models）在做这件事。但真正的难点在于生成**具有真正空间上下文并完成物理锚定**（spatially contextualized and grounded）的像素，这正是 Atlas 模型所攻克的核心壁垒。未来的演进路径还会引入第四维——时间维度带来的动态物理特性（dynamics），以及更高保真度的空间模拟与划分。 ```mermaid flowchart TD accTitle: Marble 与 Atlas 架构演进对比 accDescr: 对比展示 World Labs 从 Marble 到 Atlas 在输入表征、核心原语以及输出灵活性上的转变 subgraph Marble["Marble 模型架构（前代瓶颈）"] M_In["输入：文本 / 图像 / 视频"] --> M_Core["端到端直接生成 3D 世界"] M_Core --> M_Bottleneck["输出强绑定单一表征：<br/>3D 高斯泼溅 (Gaussian Splats)"] end subgraph Atlas["Atlas 模型架构（统一新范式）"] A_In["输入：稀疏图像 / 多视角帧"] --> A_Core["核心原语：新视角预测 (New View Prediction)<br/>（多模态提前分流与联合学习）"] A_Core --> A_Out1["高保真 RGB 图像"] A_Core --> A_Out2["几何 3D 点云与网格"] A_Core --> A_Out3["按需导出 3D 高斯泼溅 (Gaussian Splats)"] end ``` 针对主持人 Martin Casado 提出的“为什么 World Labs 成立两年半后没有直接发布 Atlas，而是先做出了 Marble”这一疑问，团队给出了技术与算力探索的真实历程。飞飞打趣道：“贾斯汀的团队需要大量的芯片”（Justin's team needs a lot of chips）。贾斯汀·约翰逊（Justin Johnson）解释称，上一代发布的 Marble 模型专注于将高斯泼溅（Gaussian Splats）作为唯一的输出模态。尽管高斯泼溅极易渲染且能在移动端和 VR 设备上流畅运行、方便与游戏引擎交互，但将其作为端到端输出的唯一接口，成了模型扩展性的技术瓶颈： - **解除输出瓶颈**：Atlas 对模型做了重大重构，不再将模型局限于生成单一的 3D 高斯泼溅，而是将**新视角预测（New View Prediction）**确立为底层核心原语。通过更早地解耦与融合不同模态，模型既能生成 RGB 图像，也能直接导出 3D 几何；只有在用户需要时，才按需转化为精美的高斯泼溅场景。 - **爬升 Scaling Ladder（缩放天梯）**：团队经历了大量的“鲜血、汗水与泪水”去权衡不同 3D 表征的优缺点。贾斯汀直言，在公司创立初期技术环境完全不同： > “当公司创立时，世界上根本不存在空间智能的缩放定律（There was no scaling law of spatial intelligence）。” - **自底向上的探索**：团队必须从小型实验与轻量级模型起步，通过持续试错建立对架构可扩展性的确信，最终历经数次迭代才锁定这套真正具备大规模 Scaling 潜力的技术路径。 --- ### `13:17` 稠密重建的痛点与稀疏输入的跨越 作为 NeRF 的核心发明人，本·米尔登霍尔（Ben Mildenhall）回顾了自己职业生涯中关于 3D 路线的长期争论：究竟该“先合成新视角再融合重建成 3D”，还是“端到端直接预测 3D”？虽然学界一直存在摇摆，但本对多视角重建的缩放潜能始终抱有极大信心。 针对 Martin 关于“为什么传统 3D 重建被称为‘稠密（Dense）重建’”的提问，本解释了商业化与落地中面临的巨大阻碍： | 维度 | 传统稠密重建 (Dense Reconstruction) | Atlas 驱动的稀疏生成与重建 | |:--- |:--- |:--- | | **底层机理** | 类似科学/医学成像，依赖严苛的视线交会与三角测量，要求每一个目标点在 3~4 张图像中同时曝光 | 结合数据先验与几何规律，既保留可见区域的三角测绘，又依靠生成能力脑补遮挡物 | | **单房间采集成本** | 需要 100 到 300 张高度重叠的照片；即便专业人员也要拍摄数分钟，普通用户往往耗时 1~2 小时 | 仅需 3 到 6 张随意拍摄的稀疏照片（采集量直降 50~100 倍） | | **采集容错度** | 极度繁琐乏味，一旦遗漏麦克风底座、桌底或叶片夹缝就会导致几何破洞 | 模型结合视觉常识自动外推并补全未见区域，无破洞或伪影 | | **资产兼容性** | 无法处理历史偶发拍摄的短片、互联网素材或零散家庭录像 | 能够直接唤醒沉睡在互联网与旧相册中原本“不可重建”的零碎素材 | 本感慨这种 50 到 100 倍的数据降维彻底逆转了 3D 捕捉的计算逻辑。他近期使用 Atlas 进行测试时，翻出了大量过去用传统 NeRF 或高斯泼溅根本无法重建的老素材，甚至把过去拍摄的上千张高密度全屋照片直接剔除 95% 后输入模型，依旧获得了前所未有的重建质量与多角度漫游体验。 --- ### `16:03` 生成与重建的交融：从斯坦福四合院到长上下文视角 飞飞列举了官方演示中极具突破性的一幕： > “最常被低估的一个演示是斯坦福四合院（Stanford quad），本只用了 3 到 25 张站在地面平视拍摄的照片就完成了全场景重建，而我们在网页上呈现的却是壮丽的鸟瞰视角（aerial view）。” 整个鸟瞰视角从物理上从未被相机的输入捕获，但生成的每一个新像素都精准遵循着真实物理世界的重建几何法则。这揭示了 Atlas 的技术本质：**经典重建与前沿生成的深度互补**。 贾斯汀与本剖析了这两种机制的深层协同： 1. **解决不可避免的视觉盲区**：即便让世界上最顶尖的专业拍摄者拿着单反在房间里扫视数百张照片，物理遮挡（如椅腿后方、桌面盲区）也是绝对存在的。传统几何测绘遇到未曝光区域必然留下空洞，而生成模型的想象力能够基于已知世界结构，先通过几何三角测量咬合可见像素，再用生成能力补足未见之物。 2. **长上下文（Long Context）的视界拓展**：大语言模型经历了从 128k、256k 到百万 Token 的“上下文军备竞赛”，所有人都能理解大上下文窗口对于解决复杂代码任务的关键价值；然而在视觉与视频模型领域，过去极少有人从底层原理出发追求这种长上下文能力，没人能将长达数小时的视频喂给模型并做到精准检索。 3. **重建即长上下文生成**：Atlas 把重建与生成统一在同一个连续光谱上——**“重建本质上就是被喂入大量观察结果的超长上下文生成”**。前代 Marble 架构无法塞入超过几张图像，而 Atlas 能够原生吃进 64 张乃至更多的全景视线输入。过去需要 2000 张高密度图像才能勉强重建的多房间住宅，现在 Atlas 仅凭三四十张图像就能无缝串联并生成沉浸式穿梭漫游，完成了对物理世界的空间推理。 ### `20:00` Atlas 的新视角预测与 3D 几何一致性 Martin Casado 探讨了 Atlas 的核心运作逻辑：用户物理拍摄的稀疏照片作为输入放入模型，Atlas 作为世界模型生成其余缺失的视角，随后再结合经典的 3D 重建算法生成连贯的飞越（fly-through）轨迹。Justin Johnson 确认了这一概念框架，并指出 Atlas 本质上可以作为一个通用的神经渲染引擎。由于 Atlas 是一个自回归（autoregressive）模型，用户可以在空间中像摆放虚拟相机一样指定任意机位，分阶段、交互式地将新视角追加到上下文窗口（context window）中，从而完成密集的场景重建。 Martin 对此表达了强烈的震撼。在他的直觉中，扩散模型（diffusion models）通常视觉表现优异但缺乏几何精确性，输入四张照片后让模型脑补中间画面，如何能保证整个房间在三维空间中严丝合缝、完全自洽？当被问及团队在立项之初是否笃信这一定能实现时，Fei-Fei Li 明确表示团队对底层逻辑具有绝对的信心：“I think three of us have total conviction about the scaling law.”（我认为我们三个人对缩放定律有着百分之百的信念）。她指出，虽然具体架构选择与数据配比充满了细节工程的挑战，但团队从一开始就坚定押注两大核心假设：缩放定律假设（Scaling Law Hypothesis）与下个视角预测假设（Next Viewpoint Prediction Hypothesis）。 Justin 补充道，尽管对长远方向深信不疑，但他最初并未料到会在第一阶段取得如此迅猛的进展。在一个全新的架构和范式下，仅仅经历第一轮预训练循环（pre-training cycle）就能直接达到可用水准，在深度学习工程中是极为罕见的现象。 ```mermaid flowchart LR accTitle: Atlas 视角生成与重建工作流 accDescr: 用户输入少量稀疏照片，Atlas 预测新视角，最后通过经典重建输出 3D 场景 subgraph Input["输入阶段"] A["物理拍摄稀疏图像<br/>(Sparse Real Photos)"] end subgraph AtlasEngine["Atlas 模型引擎"] B["自回归上下文窗口<br/>(Context Window)"] C["下个视角预测<br/>(Next Viewpoint Prediction)"] B --> C C -.->|"交互式追加机位"| B end subgraph Reconstruction["3D 重建阶段"] D["稠密多视角图像集<br/>(Dense Multi-views)"] E["高斯泼溅 / 3D 资产<br/>(Gaussian Splat / 3D Asset)"] D --> E end A --> B C --> D ``` --- ### `22:15` 算力瓶颈与花园桌下的“顿悟时刻” 在谈及该架构是否已经触及缩放极限时，Justin 断然否定，明确指出当前仅处于扩展的开端，核心限制完全在于训练算力（training compute）。在 Atlas 的开发过程中，团队沿着缩放阶梯训练了一系列模型，每次扩大参数量、延长训练时间并增加芯片集群规模，模型效果都会产生显著的质的飞跃。博客文章中展示的旗舰模型仅仅受限于产品发布的截止日期，团队只能在有限的时间窗口内投入相应规模的算力。 Fei-Fei 分享了一段团队内部的关键转折点。在初夏的某一天，团队测试的还是一个比当前 Atlas 小得多的早期模型。Ben Mildenhall 将 NeRF 论文中最经典、最具挑战性的“花园桌子”（garden table）测试集输入该模型。在半夜，Ben 向团队 Slack 发送了一段渲染视频：虚拟相机极其丝滑地贴地穿行至花园桌子的正下方，视角正中赫然展现出那颗真实的足球。Fei-Fei 回忆道：“That morning, the three of us looked at each other in the eyes and said, that's it. We're going to build this. We made a decision within five seconds.”（那天早晨，我们三个人面面相觑，互相看着对方的眼睛说：成了，我们就要做这个。我们在五秒钟内做出了决定）。这证明了模型不是在进行无序的 2D 幻觉生成，而是真正理解了复杂遮挡关系下的 3D 几何结构。 - **花园桌场景**：NeRF 领域的经典 Benchmark 场景，桌底存在极强的光影与遮挡关系，相机穿行桌底展现足球的渲染效果直接奠定了团队全速推进 Atlas 的决心。 - **扩展性现状**：当前架构在不发生结构性变更的情况下仍具备极高的扩展上限，核心瓶颈纯粹在于硬件训练算力的供给。 --- ### `25:00` 创意管道与设计流程中的持久化 3D 状态 Ben Mildenhall 解析了 Atlas 在实际工作流中的应用价值。他回顾了 World Labs 的上一代产品 Marble（输入单图生成 3D 高斯泼溅场景）。团队在观察用户行为时发现了一个耐人寻味的现象：大量用户在 Marble 中生成完整 3D 场景后，仅仅是在不同角度截取几张特定视角的屏幕截图，然后便直接离开。这种工作习惯反映出传统 3D 生成管线的图像质量存在严重损耗，同时也催生了团队的思考：既然用户真正需要的是多角度的高清受控视图，为什么不直接利用生成模型来端到端建模这些视角？ Ben 强调，在现有的影视概念设计、情绪板（mood boards）和分镜制作中，创作者面临的最大痛点就是缺乏统一的“持久化 3D 状态”（persistent 3D state）。主流 2D 图像扩散模型完全是瞬态的（ephemeral），用户输入提示词抽卡，每次生成的空间布局都会发生漂移，极难维持跨视角的几何一致性。而在现实世界的创意与建筑设计中，人们依赖的是固定的空间、道具和布景，通过物理资产的组合来推演空间。Atlas 的空间上下文机制正是为了提供这种精密的空间控制能力。 在建筑施工与工业展陈设计领域（例如展会展位搭建），从业者往往承受着巨大的 3D 建模工作量。设计总监或建筑师提出的口头修改意见或手绘草图，往往需要建模师在历史长达数十年的老旧 3D 软件中耗费整整一周进行手动修改。Ben 总结道：“mapping that back into the 3D representation is like 95% of the work, right? You can have a meeting, get feedback, and then you go back and do a week of revisions.”（将修改意见重新映射回 3D 表示中占据了 95% 的工作量。开一次会收到反馈，接着你就得花整整一周去做模型修改）。AI 模型若能直接理解空间并实现直觉式的三维推演，将彻底解放设计流程中的生产力。 --- ### `28:40` 从虚拟世界到物理实体：Atlas 与机器人仿真 Martin 顺势将话题引向更具工业野心的方向：World Labs 此前收购了一家名为 Dexai（原名 Cinex）的具身智能初创团队，但 Atlas 这类多视角生成模型与物理机器人训练之间的联系初看并不直观。 Fei-Fei 明确指出，Atlas 正是补齐全自动机器人管线中最核心的基石拼图。该被收购团队的核心技术壁垒在于构建了一套高度自动化的“虚实闭环”（Real-to-Sim 和 Sim-to-Real）系统。在现实的工业机器人训练场景中（例如让机械臂在装配线上学习线缆插拔与布线），最大的瓶颈始终是缺乏足够多样化、带有物理交互标注的高质量训练数据。如果能够依托 Atlas 从极少量物理世界影像中精确外推并重建出高精度的物理仿真环境，就能在数字孪生空间中无限扩增训练样本，从而打破物理世界数据获取成本极高的天然制约。 ### `30:00` 真实世界到仿真（Real-to-Sim）与机器人训练的数据瓶颈 在讨论机器人策略（robotic policy）的具体应用时，李飞飞（Fei-Fei Li）以机柜布线（cabling）和洗碗为例，拆解了训练机器人落地的严苛流程。过去，像团队此前的具身智能创业项目及现有的机器人团队，通常必须依赖密集重建（dense reconstruction）：工作人员需要在现场拍摄大量照片，然后竭力重建出周围的三维物理环境。李飞飞坦言，这一过程“极其痛苦、耗时费力，严重阻碍了机器人仿真及 Real-to-Sim 的推进速度”。而 Atlas 正是针对这一痛点打造的下一代生成式平台，能够直接加速整个环境构建与数据生成闭环。 李飞飞强调，目前制约机器人学发展的根本瓶颈并非算力或模型结构，而是现实数据极度匮乏：“The biggest problem right now in robotics is actually data.” 在复杂的作业场景下，单纯收集一次现场布线或洗碗的几何数据还远远不够，更关键的步骤在于环境随机化（randomization）——线缆不可能永远向同一个方向弯曲，箱子会有不同的尺寸、颜色、盖子形态，并散落在场景的各个角落。为了让策略具备泛化性，机器人必须通过 Real-to-Sim 管道引入海量的变体数据。作为一款多模态、全能型（omni-model）模型，Atlas 能够摄入包含动态变化在内的多种数据，从而在底层的动作规划（action planning）与机器人的真实执行之间搭建起桥梁。 - **密集重建的困境**：传统方法依赖人工采集多视角静态照片拼凑 3D 网格，成本高、迭代周期漫长。 - **环境随机化的必要性**：机器人必须在仿真中经历线缆弯曲角度、容器外观、物体位置等多维度的扰动，避免过拟合于特定实验室环境。 - **Atlas 的角色定位**：作为连接互联网多模态数据与现实动态环境的桥梁，大幅缩短 Real-to-Sim 的数据生成链路。 ```mermaid flowchart LR accTitle: Real-to-Sim pipeline with Atlas accDescr: 真实场景通过 Atlas 生成随机化虚拟仿真环境，进而训练并评估机器人策略 subgraph RealWorld ["现实物理世界"] A["多视角环境数据<br>(布线/家务场景)"] end subgraph AtlasEngine ["Atlas 世界模型"] B["密集环境重建与生成"] --> C["环境条件随机化<br>(光影/尺寸/形变)"] end subgraph SimToDeploy ["仿真到部署"] D["神经仿真测试床"] --> E["机器人策略训练<br>& 鲁棒性评测"] E --> F["部署至实体机器人"] end A --> B C --> D ``` --- ### `32:30` 交互式智能体的本质：从物理引擎到神经仿真器与规划器 贾斯汀·约翰逊（Justin Johnson）深入剖析了机器人策略训练与其他常规 AI 应用在底层逻辑上的根本分水岭。如果让模型生成一段代码、一张图像或一段视频，它本质上是在产出一种静态的人工制品（artifact），而互联网上本就充斥着数以亿计的代码库、图片和视频供其学习。然而，机器人策略完全不同：“It's instead a policy that's going to go out into the world, make actions, and like, try to achieve a goal.” 真实世界并不会总是按照预设逻辑予以反馈，意外情况随时发生；因此，智能体在训练期间必须尽可能暴露在所有可能出错的极端情境中，这使得仿真对于机器人而言不可或缺。 针对仿真的实现路径，Johnson 指出了两种思维方式： 1. **经典物理仿真（Classical Simulation）**：依赖人类设计师或辅助编程智能体，在物理引擎中通过手写显式代码穷举可能发生的情景。 2. **数据驱动的神经仿真器（Learned Neural Simulators）**：训练一个能自主理解环境如何对物理动作做出响应的神经网络，允许它以非确定性的方式展现意外情况，并将其作为训练策略的基础测试床。 Johnson 进一步推导指出，世界模型的终极形态绝不仅仅是一个被动的仿真测试环境。一旦神经仿真器在其内部表征中彻底理解了物理世界并知晓动作产生的反馈，它本身就能自然演进为动作规划器（planner）。世界模型的核心假设正是：理解动作会导致世界如何变化，与反向推导“我需要采取何种动作才能达成某种世界状态”，在认知本质上是高度同构的。 --- ### `35:00` Atlas 模型的动态性争议：动静结合的训练哲学 主持人马丁·卡萨多（Martin Casado）首先向团队致贺，指出 Atlas 的发布获得了行业内压倒性的正面评价，甚至堪称年内最具影响力的模型发布之一。但他也转述了一位资深专家的关键反馈：模型固然令人惊叹，但在机器人及普遍应用中，“需要拥有更多的动态性（dynamics），构建一个真正会动的世界”。 对此，团队坦言 Atlas 其实已经内置了“雏形动态（baby dynamics）”。Johnson 解释道，World Labs 之前的第一代 Marble 世界模型在架构与训练逻辑上完全是静态的，根本无法处理时间维度上的动态变化。而在开发 Atlas 时，团队从一开始就在架构底层支持了动态机制，并喂入了海量动态数据。如果在视频演示中仔细观察，就能发现水面激起的层层波纹，以及航拍鸟瞰视角下缓缓行驶的微缩汽车。 针对 Casado 提出的核心技术困惑——“若想从多视角精确重建 3D 几何，动态性难道不是一种致命的干扰吗？”，团队给出了极具洞察力的解释： - **静态重建与海量动态数据的张力**：虽然传统 3D 重建追求冻结时间点（frozen time）的绝对静态几何，但纯静态的训练数据在现实中极其稀缺，严重限制了扩展规模（scale）。 - **预训练解耦物理世界**：团队发现，为了获得高质量的静态场景重建能力，最佳途径反而是让模型在预训练阶段接触大量的动态数据，使其自主学会“剥离”（factor out）动态干扰。 - **分阶段训练策略**：Atlas 在预训练阶段吸收了大量的时空动态特征，具备丰富的潜在动态能力（latent dynamics）；而本次发布的 Checkpoint 则在后训练（post-training）阶段有意识地聚焦于空间移动性与静态一致性。本·米尔登霍尔（Ben Mildenhall）幽默地顺应补充，未来的目标将是迈向可任意漫游、捕捉微表情细节的完整“4D 视频”。 --- ### `37:30` 迈向工业级控制：可编辑性与保真度的平衡 面对 Casado 关于“除扩大规模（bigger, faster）之外，三维内容创作等垂直场景还需要什么”的追问，Mildenhall 认为多模态条件控制（control conditioning）是下一阶段的核心。李飞飞用通俗的大白话将其总结为：“To translate in layman's language is editability. I think editability is the key here.” Mildenhall 详细阐释道，当前的前沿图像生成模型已经支持多轮对话与复杂的局部精准编辑——用户提出想要某个人物、某个物体以及特定事件发生，系统能自然融合；而在视频模型与世界模型中，这种交互大多仍停留在简陋的玩具式演示阶段（如打入一行字弹出一只恐龙）。要把世界模型提升至工业级（industrial strength）的实用标准，核心在于赋予创作者细粒度的控制旋钮，同时绝不能损伤模型底层的生成质量： > “The trick here is you've got to add control, but not compromise the quality of the model or it just becomes a party trick, basically.” Mildenhall 指出，如果引入额外控制维度的代价是画面崩坏或保真度骤降，任何严谨的工业管线都不会考虑用世界模型替代既有的前沿视频工具。因此，在维持超高质量的前提下，不断融入用户所诉求的交互式操作界面，将是团队接下来产品化推进的攻坚重点。 ### `40:00` 与有状态3D世界交互：产品界面的全新范式 讨论聚焦在超越单纯模型规模之外的未来功能上。在谈到用户期望如何与生成的场景互动时，团队认为未来的核心轴线在于控制力：用户不仅希望能查看场景，更希望能直接操控场景布局、控制场景内物体的身份与属性，甚至是掌控时间维度的变化。这种对交互性和控制力的追求，为产品设计与人机界面（UI/UX）开辟了巨大的探索空间。随着系统能力和丰富度的提升，它将促使人们几乎从零开始重新思考用户与计算机中有状态（stateful）3D世界交互的方式，这也是团队打造此类系统所追求的最终愿景。 当主持人 Martin Casado 询问除了“更大、更好”之外还有哪些令人兴奋的新功能时，李飞飞（Fei-Fei Li）提出应当回归智能的第一性原理。她强调，当涉及物理与空间时，智能绝非仅仅呆坐在那里看东西或被动解释事物，其核心在于完成从感知、体验到行动交互的闭环。因此，沿着这条阶梯向上攀登，真正实现这种交互闭环，才是空间智能发展的终极目标。 - 用户对 3D 场景控制的需求涵盖三大核心维度：空间布局控制、物体与身份属性控制、时间维度演进控制。 - 终极目标是赋予计算机完整的感知与生成能力，支撑人类与有状态、动态演化的 3D 数字世界进行全新交互。 ### `41:00` AI完备性与“新视角预测”即空间智能基元 随后，对话切入了一个深刻的理论概念——“AI 完备性”（AI-completeness）。Justin Johnson 将其类比于经典计算复杂度理论中的图灵完备性（Turing completeness）或 NP 完全性：在计算理论中，任意 NP 问题都可以归约到 3SAT 问题；而在人工智能领域，如果存在某种基础基元任务，只要能在最广泛的通用层面上解决该任务，就能解决一切智能问题，那么这个任务就是 AI 完备的。在大语言模型（LLM）语境下，人们普遍认为“下一个 Token 预测”（next token prediction）具备 AI 完备性，因为任何智能问题——比如推理悬疑小说的凶手——都可以形式化为预测接下来的词。 Justin Johnson 指出，团队意识到 World Labs 在 Atlas 模型中采用的核心基元——“生成式新视角预测”（generative new view prediction）——同样具有 AI 完备性。Martin Casado 随即举例说明：你可以输入一部电影的所有帧，当凶手即将走出来的瞬间，去预测那一刻的全新画面视角，这就等同于推断出了凶手是谁；Justin Johnson 进一步补充，你甚至可以在场景中构建一个环境，预测 Martin 正在写下黎曼猜想（Riemann hypothesis）的证明并最终解出它的画面。李飞飞从进化论视角给予了有力论证：大自然赋予了能够移动的动物双眼，却没有赋予静止的树木眼睛，正是因为移动必然伴随着新视角的产生与预测。她斩获核心论点：“So we do believe very strongly that next viewpoint prediction is the equivalent of next token prediction.”（所以我们极其坚信，下一个视角预测就是下一个 Token 预测的空间等价物。）对谈在对 Atlas 成功发布的祝贺中圆满收尾。 ```mermaid flowchart TD accTitle: AI完备性与预测基元映射 accDescr: 对比自然语言领域的Next Token Prediction与空间智能领域的Next Viewpoint Prediction的等价结构 subgraph LLM["自然语言模型 (LLMs)"] A["核心基元：Next Token Prediction"] --> B["形式化表示：推断悬疑小说结尾预测凶手姓名"] B --> C["AI 完备性：覆盖人类符号与逻辑思考"] end subgraph WorldModel["世界模型 (Atlas / World Labs)"] D["核心基元：Next Viewpoint Prediction"] --> E["形式化表示：移动并生成新视角预测凶手走出或解开猜想"] E --> F["AI 完备性：闭环'看-移动-交互'的空间物理智能"] end C <==>|"演化与信息等价"| F ``` - **理论归约类比**：正如 NP 难题可归约为 3SAT，任意通用智能任务亦可被形式化为空间几何与时间演进下的视角预测。 - **演化生物学证据**：移动产生新视角，因此大自然仅为具备位移能力的生物进化出了视觉器官，证明视觉预测与空间行为密不可分。 - **团队核心信念**：“新视角预测”（Next Viewpoint Prediction）不仅是 3D 生成的技术路径，更是理解与模拟真实物理世界的最本质基元。 ## Notable quotes > `41:00` So we do believe very strongly that next viewpoint prediction is the equivalent of next token prediction. — 李飞飞 > `30:00` The biggest problem right now in robotics is actually data. — 李飞飞 > `37:30` The trick here is you've got to add control, but not compromise the quality of the model or it just becomes a party trick, basically. — Ben Mildenhall > `05:00` 伴随生成的每一帧，你都能生成并估计出一个关键信息，那就是视点与相机位姿。而这正是人们理解空间几何时所需要的最关键信息。 — 李飞飞 ## Mentioned World Labs, Atlas, Marble, Fei-Fei Li, Justin Johnson, Ben Mildenhall, Martin Casado, a16z, Dexai, NeRF, 3D Gaussian Splatting, The Matrix, Keanu Reeves, iPhone, Stanford University ## Video description World Labs co-founders Fei-Fei Li, Justin Johnson, and Ben Mildenhall join a16z General Partner Martin Casado to discuss Atlas, their latest world model, and what it reveals about the pursuit of spatial intelligence. At the center of Atlas is what the team calls “new view prediction”: given images or views of a scene, the model predicts what that environment should look like from a different position in space and time. This brings generation and 3D reconstruction into the same model, and raises a broader question about whether predicting views could become a useful primitive for understanding the physical world. They discuss the technical bets behind the model, what it can and can’t yet capture, and the importance of dynamics, editability, and simulation as world models develop. The conversation also explores applications in creative work, architecture, and robotics, where Fei-Fei argues that one of today’s biggest constraints is access to real-world training data. [Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/7a8beefc-2d73-42f7-8b7a-d2f15ad0358d/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=7a8beefc-2d73-42f7-8b7a-d2f15ad0358d&feed=JGE3yC0V)

[41:00] 李飞飞: So we do believe very strongly that next viewpoint prediction is the equivalent of next token prediction. [30:00] 李飞飞: The biggest problem right now in robotics is actually data. [37:30] Ben Mildenhall: The trick here is you've got to add control, but not compromise the quality of the model or it just becomes a party trick, basically. [5:00] 李飞飞: 伴随生成的每一帧，你都能生成并估计出一个关键信息，那就是视点与相机位姿。而这正是人们理解空间几何时所需要的最关键信息。

[40:53] Fei-Fei Li: We do believe very strongly that next viewpoint prediction is the equivalent of next token prediction. [7:00] Fei-Fei Li: Spatial intelligence eventually must enable us to both generate what the space is, reason within it, and being able to edit and interact within it. [30:00] Fei-Fei Li: The biggest problem right now in robotics is actually data.

[0:00] On the path to spatial intelligence, generating pixels that are truly spatially contextualized and grounded, that is the very hard step that Atlas has taken. We know LLMs are built on next token prediction. We've seen video models as being built on next frame prediction. Atlas is really new view prediction. This is the real place where AI can actually unlock a ton of value for people and their process. We're saying like 50, 100x reduction. that angle on a green screen. On Atlas, we can do this with just three cameras. No studio capture, no green screen, no expensive calibration. No one has ever seen this result. When you set out to do this, did you know it was going to work? I was pretty sure. Each time we made the model bigger and each time we trained it for longer, it got significantly better. Does that mean we're going to get 4D video? Can I go walk around? Language models are built around predicting the next token. What happens when a model instead learns to predict the next view of the world? with World Labs co-founders Fei-Fei Li, Justin Johnson, and Ben Mildenhall to discuss Atlas and the broader challenge of building AI that can reason about physical space. They explain how Atlas combines generation and 3D reconstruction, why the team chose NewView prediction as its underlying primitive, and what they learned trying to scale an approach that hadn't been tested before. They also get into what's still missing, including richer dynamics with robotics and planning. Underlying it all is a bigger hypothesis. Could new view prediction play a similar role for spatial intelligence that next token prediction has played for language? the world model. It has three basic things. It can generate, reconstruct, and simulate the world. So within that, there's a couple different major capabilities. It has really good camera condition generation. So you can input an image together with a camera trajectory and steer the model and have it generate video frames for any perspective you want. It's really good at sparse 3D reconstruction. You can input one or multiple up to 100 frames. that are views of the real world and use those to reconstruct the real world. And that reconstruction can take the case either of novel video flying through the space or an explicit 3D reconstruction of the space. Then finally, it can be used for simulation. And for this, we show off these awesome bullet time videos, which got a lot of attention online. And then also robotic simulation. What's a bullet time video? A bullet time video. This comes from The Matrix. There was a famous shot in the first Matrix movie where Neo is like, oh, yeah, exactly. So and then remember in that famous shot, he's falling down. It's in slow motion and the camera flies all the way around. The way that they did that shot is they had a ring of hundreds of cameras. So then he fell over in the studio. They had hundreds of cameras viewing that angle on a green screen. hundreds and hundreds of cameras to make that famous shot in Matrix. But now with Atlas, we can do this with as few as three cameras. So no studio capture, no green screen, no expensive calibration. We can literally stick three iPhones on tripods, use these to take sort of a video of something happening, like someone shooting a basket, someone dropping a strawberry into a bowl of milk. And then from those three iPhone videos, we can then reframe the shot and imagine like freeze time, have the camera fly in as the milk is splashing up and get these amazing frozen time views. And we can do this with just a couple of cameras. What is the simplest description of what Atlas does? What goes in and what comes out? Yeah, so one of the really core principles of Atlas, the most fundamental thing, is it does new view prediction. And this is a really fundamental primitive that we think is super exciting, a super new primitive for base models that no one's ever done before. So we know LLMs are built on next token prediction. We've seen video models as being built on next frame prediction. Atlas is really new view prediction. that given some number of views of a scene or a description of a scene, those go into what we call a spatial context that describes implicitly what is the world that we want to talk about. Then you can point a virtual camera at an arbitrary point in space and time, and Atlas will understand what that world is supposed to look like from that position in space and time. Ben, the video models out there all claiming to be world models and all claiming to have novel views, have come before. Yeah, I think what Justin was saying about the spatial context aspect is super important here. So there's many video models, a lot of video models actually got their claim to fame from their single image input or their start to last frame interpolation. Now we're starting to see models that can do this kind of omni-reference thing with 20, 30, 50 images. But what's key with Atlas is that it actually has a kind of like spatially grounded meaning to every frame you put into it. So it's not just an image that the model is going to interpret whatever way it wants, or specific. With Atlas, every image actually has an associated three-dimensional camera pose and that [5:00] is that you can perform this task of reconstruction with an extremely high degree of precision, right? So if we had four views of this room, one at each corner, you can put those into the model and then get an exact replication of everything you see in this room. And it's not going to guess what's in the other corner, like the relationship between things. It's just going to reproduce exactly what you give it. And you can also do that in a kind of creative or imaginative sense too. If you take two photos from different AI generations or real world locations, you can actually position and stage those directed fly-throughs that are really governed by exactly the precise place that you put the content you want and where the camera is going to look and travel, which is very different, I think, than the kind of like more slot machine effect you get of having to retry generations over and over with just that kind of higher level of text control you get with video models. Is this just kind of an obvious scaled-up version of a traditional video model, or is it a new architecture? I think it's a pretty new thing for a couple different reasons. One that we talk about is it does both generation and reconstruction jointly in the same model. Like Ben was saying, this thing can take a couple views of this room and then reconstruct everything in this room exactly as you see it. And historically, reconstruction has been its own subfield in computer vision with its own specialized task, its own specialized models. good at, like with all the big diffusion models we've seen the last couple of years. And those are great for creative applications. I want to imagine something that's never been there before. But now with Atlas, for the first time, we're putting these two different parts of visual intelligence together in one model. So we can do both 3D reconstruction and generation together in one architecture. So to do that, we had to make a couple of changes. One is we had to make it multimodal from the start. So this thing natively works on text, it works on images, it works on videos. It also works on camera poses as a native input to the model, which I don't think anyone's ever done at the pre-training phase before. And it uses 3D as a native modality that it works on. So this thing from the beginning was designed to be natively multimodal in a way that no one else, I think. Sorry, I don't know the space super well. But 3D is this like... depth or models or like, what does that mean? Yeah. So the formulation we used so far is depth maps. Okay. Right. So right now when you have a frame that has a virtual camera telling its position in 3d space, that camera position and camera parameters are a native input to the model. And then what attached to that camera position, you can have both like RGB telling you what does that position in space look like? And you can have a depth map that tells you what is the spatial structure of that position in 3d space. So then text, image, video, 3D cameras are these modalities that this thing all does jointly in a multimodal way. I want to add something because I think what Justin just said is actually so important and also what Ben said that it's underappreciated. It's the first time we have a unification of pixel generation and pixel reconstruction. or novel view synthesis. And also... Our field traditionally have multiple tracks. You go to a computer vision conference, you have the pixel generation track, you have some recognition track, and you have 3D reconstruction track. This is an elegant model that combines or unifies the problem of reconstruction and generation by anchoring on viewpoints and the viewpoint estimation. And that's just incredibly powerful. Can we take a step back and then maybe you just fill something out? the company, I remember you saying you want to tackle spatial intelligence, right? And now we have this new model. And so as a layperson, it feels very general to me. You've got next token prediction and this is next new view prediction, right? And so like you can get one view or a set of views and you get a new view. Can maybe you pencil out like how this is a significant step to this general problem of spatial intelligence? Maybe by starting to describe the what spatial intelligence is? Well, spatial intelligence eventually must enable us to both generate what the space is, reason within it, and being able to edit and interact within it. Now, we can argue, is it 3D or 4D? Ultimately, it's 4D with the time dimension. But even just 3D, these are the fundamental tasks that one has to do or spatial intelligence has to enable. And then we talk about with that, you can render, you can simulate, and you can plan actions. But to understand the geometry and structure and the physics of the space. And I do believe Atlas is a significant step forward because now with every single frame, you can generate and estimate an important piece of information, which is the viewpoint, the camera pose. And that is the most critical information one needs about the geometry of the space. [10:00] And that can lead to all the emergent behaviors we see in the downstream of the model, which we showed in the blog. on the path to spatial intelligence. Generating pixels is definitely an early step, which we have seen with what you call it, gazillions of models. But generating pixels that are truly spatially contextualized and grounded is absolutely another major step. And that is the very hard step that Atlas has taken. We definitely have, you know, we can just, keep going here, right? Like there is the fourth dimension of time, which will bring in dynamics. And there is more higher fidelity simulation and delineation of the space. So this is part of the the roadmap of spatial intelligence. Great, yeah. I mean, I definitely want to dig into where this is going. But first, maybe let's talk about getting here. How long has WorldLav's been... Two and a half. Two and a half, yeah. And so you've actually released models before. So why didn't you just jump right to Atlas? It's so magic, right? Justin's team needs a lot of chips. Yeah, you need a lot of GPUs to actually scale this thing up. So last year we released our Marvel world model, and that was the first kind of big major world model that we put out that powers our current Marvel product. And Marvel is really cool. Marvel can take images, it can take videos, it can take text prompts and use these to generate 3D worlds. But one of the biggest differences between Marble and Atlas is exactly what is that output modality. So Marble was really focused on Gaussian splats as an output representation. So whatever you're inputting, it's going to output a 3D world represented as a Gaussian splat. And Gaussian Splats are really useful, right? They're really nice. They're easy to render. They can render efficiently on mobile devices, on VR devices. They can interoperate with game engines, with simulation engines. There's a lot of nice things about Gaussian Splats. But, you know, I think that was kind of a bottleneck in the previous marble model. So what we did with Atlas is redesign the thing a little bit. And we realized that we need to bifurcate these modalities earlier and actually have these things, all these modalities working in a more unified way in the model. So now with Atlas, the fundamental primitive is not like a good generate a Gaussian splat world. The fundamental primitive is, as we said, new view prediction. And that can generate RGB frames that can generate 3D. And we can use those to generate a beautiful Gaussian splat world when you need them. But we don't need to bottleneck our outputs through the Gaussian splats when we don't need to. And that actually took a lot of, you know, blood, sweat and tears to understand, like, what are all the pros and cons of these different representations? That's one part of it. The other part is you've got to like climb the scaling ladder, right? You've got to like work your way up and like do smaller experiments, do smaller models like to build your conviction on what's going to work and what's going to scale. And there's, you know, if you could instantly know the right thing that's going to scale, you know, you should just do that. But. When we started the company, the world was a very different place. There was no scaling law of spatial intelligence. Right. So when we started the company, the world was in a very different place. The tech was in a very different place. We had a lot of ambitions about where we wanted it to go. But it took a couple iterations for us to hit upon this formulation that we thought is actually, this is the one that can scale out. You know, Ben, you know, being the creator of Nerf and doing a lot of 3D and reconstruction. So it's not so obvious to me that like, if you have multiple views that you actually end up with a 3D thing, but like you've kind of like made a career of ending up with a 3D thing. So maybe talk a little bit about like kind of that step. Yeah, yeah. Yeah, I mean, as you said, I've spent Many, many years of my career, the vast majority of my career, actually working on producing 3D things from images. And it's actually something we talked about a lot early on in the company, even if like, is this going to be the approach that produces 3D, right? Are we going to synthesize multiple views and then build 3D out of that? Or are we going to try to go direct to 3D? Like, there's been a lot of uncertainty in the field around like, which of those approaches kind of will work? we now will kind of like reap the best advantages earlier on. But I did have a lot of conviction just from seeing the kind of power of what I would almost call the brute force scaling. Scaling in a very, very, very small baby scale, not like a real model scaling, but the scaling of dense reconstruction that we had seen happening over the past three years before. So basically, we put up... Why is dense reconstruction dense? Yeah, dense. Because I know we're going to talk about sparse, and I want to make sure that people understand Yeah, so I think this is actually, even on the kind of like business and commercial side, I think been one of the challenges of productizing 3D reconstruction technology, like at a fundamental level, right? People kind of don't, in a casual sense, like you think I took three photos of this object or I took six photos of this room, like, I look at the photos, I can understand in my mind, like how those piece together, I can kind of fill in the gaps and get it. But there's just never been really any kind of [15:00] reconciliation between those like really data-driven priors and then the kind of brute force dense reconstruction which it actually is much more akin to almost like scientific or medical imaging what we did in dense reconstruction right you basically have to say every single thing i want to appear in this reconstruction i need at least three or four views of it and if you think about that like even just in this room right there's like under the microphone under the table between every different crack and crevice and the plant leaves right to actually truly get a picture that covers every one of those spots. It's this very tedious and exhaustive effort to walk around the room. I think you've all seen me running around various places capturing them. It takes, for someone who's well-trained per se, it can take minutes, but if you hand a casual consumer or even some professional trying to do this for the first time an average cell phone camera or capture device, it's going to take them probably an hour. I've seen someone for their first time trying to scan a multi-room environment, spend two hours walking through it and get enough coverage. And that's just this very, very exhaustive and tedious loop. And so yeah, when we say dense, we really mean dense. It's like this room, I want like 100 photos. So many photos, right? I want like 100, 200, 300 photos of this room to capture it And what we trying to do is bring that down to like three right Wow We saying like 50 100x reduction And then that at that scale where it just completely like flips that calculus on its head of like what type of captures you reconstruct You can go back to existing imagery you have. You can go to stuff you find on the internet and even build scenes out of that. You can go to casual videos and like kind of unearth a lot of footage in the past. We would never have treated as reconstructible and go back and like bring it to life as 3D potentially. This is something we've been playing around with a lot with Atlas, right? Like taking old clips, like I've taken a bunch of my own old captures that never worked before and then put them through the system and then kind of seen a reconstruction for the first time or taking my old captures and thrown away 95% of the photos I took. And, you know, imagine angles that I never would have gotten from a traditional kind of like Nerf or Splat type reconstruction. One thing that's underappreciated on the website of the demos is the Stanford demo where Ben showed anywhere between 3 to 25 images. You can reconstruct that entire Stanford quad. But the thing is, we had to show it from an aerial view. But every single input image is Ben standing on the ground, taking a picture from the ground. So everything you see are generated but according to the laws of reconstruction And this is really magical And this is where like generation and reconstruction need to interplay in a really fundamental way to solve this problem Because under the classic kind of reconstruction stuff that Ben was talking about, like the reason you need so many views is because I need like multiple images and I need to triangulate this point in 3D space and see it from multiple viewpoints. So that means like that's required in the traditional version. and on the flip side, anything that wasn't captured in these views, like any pixel that was not visible in one of the input views will be a hole in a 3D reconstruction. Because fundamentally, like, if a thing wasn't visible in the input views, you know, you need to imagine it to fill in the gaps. And that's fundamentally a generative process. So even in this room, even if we set Ben loose with a DSLR and like let him like capture like hundreds of views of this room, even the world expert on doing these dense captures is still going to miss some spots. Like he's not going to get like underneath all of the microphones or underneath all the tables or in between all the chair legs, you're always going to miss something, no matter how many views you get. So that's where you need generation as another mechanism in the model because you're never going to get everything. So you need to have some generative capacity for the model to imagine, oh based on what I seeing then first triangulate what I can see but then fill in the gaps of the stuff that inevitably was not captured Yeah and there something like super cool about this that LLMs have really understood this for a long time right There was kind of almost these like context wars, like the first couple of years, it was like, oh, we got to 128, to 256, to like 512, we got a million, right? And everyone kind of understands now at a pretty tangible level the value of, you know, you crank your context length to high when you're using your coding model. It's a hard problem. Like everyone has a feel for that, but like no one has pushed that at all on the image and video model side in the same kind of like principled way. Like no one's out there trying to like put a, like an hour long video through and do a needle in a haystack retrieval of like a frame at the 37 minute mark. Whereas with reconstruction and generation, you actually have the same exact thing of like reconstruction is just like generation with a really long context and you put a lot of stuff in it, right? Like that's the way to actually build this continuum where you kind of bridge between those two things. And like Atlas, like being able to, like this is something we can never do with Marvel. Marvel had this kind of fundamental blocker of like you couldn't really jam more than honestly like a couple images in. But at least I can go and I can actually take like a 64 image capture and do like a fly through of an entire house and everything is grounded by being, you know, seen or like almost seen or like slightly extrapolated from what's not there. But you're just getting these, you know, I'm taking captures I did with 2000 images of a multi-room house and taking it down to like 30, 40 inputs. [20:00] and the fly-through looks like basically the same. And this is just like totally inconceivable before. And it's all enabled by building this gracefully scaling kind of context window that you can dump stuff into. And so the way to think about it is like the sparseness are the pictures that you physically took. And then Atlas as a model creates the rest of the views. And then you use classic reconstruction techniques. Is that roughly the way to think about it? In some sense, yeah, yeah. I mean, that's the beauty of Atlas. It's like you can take however many inputs you have and then you can almost use Atlas as this rendering engine to produce anything else you want. You can navigate it. Like the virtual camera. Yeah, exactly. You can say like, okay, I have a picture here. I want the picture there, there, there. You can make a couple of those. Then you can stage a dense fly-through. You can do this in sequence because it's an autoregressive model. It's up to you, right, to kind of pick and choose what you add interactively into the context as you generate. I mean, the thing that I just blows my mind is, listen, I just have a very simple mental model. I have four pictures, and then I've got to like have the model extrapolate between them and And then it has to fit when you're in control. Like, it's got to be 3D. And I always think of these diffusion models as, like, being visually great but not accurate. And so, like... And I don't even know if there's a question here, but like, how is it like the room fits? Like, how is it that it's 3D consistent? Is it just lots of data or? Yeah, I mean, it's a part partially it's a belief in the scaling hypothesis, right? Like, you know, did you by the way, but I have to ask when you set out to do this, did you know it was going to work? I was pretty sure. Were you sure? I think three of us have total conviction about the scaling law. That I think we do. I do think the exact architecture choices and data mixtures is where the devils are in the details. I, you know, have... I watched Justin and his team going from, we really don't know how long this is going to take, to, oh, maybe sign of life, to, wow, this is going to work. So no one has done it. But I think the hypothesis, two hypotheses, one is scaling law hypothesis. Another one is next viewpoint prediction. We had conviction of these two things from early on. So I think I was very convicted that it was going to work. this fast, right? Like I thought there's a chance that we do this. Maybe it's not clear that like the first cycle of pre-training a new model with a new architecture and a new paradigm, like the first cycle of that working is insane. So I thought there was a chance in which we might have had to do a couple more turns of that pre-training cycle before we got to the level of quality we wanted. Are we kind of like at the end of like the scaling for this architectural approach? We need another breakthrough or is there? No, no, we're at the beginning. Really? Without changing the architecture. I think we're basically at the beginning and we're basically limited by compute at this point. Like data is very important, as Fei-Fei likes to point out, but like everything has a bottleneck. And I think the main bottleneck on continuing to scale this thing is actually training compute. During development, we trained a sequence of models. We read about this in the blog post a little bit, but we trained a couple of models, the first couple rungs of the scaling ladder. And each time we made the model bigger and each time we trained it for longer, each time we put it on more chips, it got significantly better. And the model size, the model that we showed in the blog post is obviously the biggest and best one that we trained. But the thing that was limiting it was not the scale or the data or anything like that. It was literally like we had a deadline of when we wanted to release this thing and therefore we backed up what we could afford to train in order to that deadline. But here's a little bit of an insider story, right? or having these roadmaps. And then there was one day in... summer, early summer, that it's not even the current Atlas model size. It's a smaller model. And then Ben, Justin, Ben feed it into, you know, the viewpoint generation. And remember that famous table, the garden table for Nerf paper and many papers that... Overnight, I got a Slack. I mean, we all saw the Slack from Ben at The camera flew through under the table. With the soccer ball. Yes, with the soccer ball. Is the soccer ball emergent or is that in the original picture? It was real, right? That morning, the three of us looked at each other in the eyes and said, that's it. We're going to build this. We made a decision within five seconds. This is just a no one has ever seen this result. Right. Ben, can you talk through maybe more specifically the use cases? So World Labs has historically had a lot of users that were creatives, and they use it for consistency in whatever 2D images and for movies and for... [25:00] 3D and for games, etc. And so maybe can you talk about how this extends use cases that are catered to the existing ones? And then we'll talk about robotics, actually. Yeah, sure. Yeah, I mean, it's kind of funny, actually. One of the kind of main ways we even saw people using Marble plays exactly into this new view prediction case. Like a lot of our... Marble being the previous... Sorry, Marble, our previous product. Like people would take that product, put an image in, get a full 3D scene as a Gaussian splat, take a couple screenshots of it from different points of view and leave. And we're like, We can just make those images and that's daily data. So I think there's a lot of degradation there. They're like, oh, this flag could look better. And it's like, okay, what if we just generatively model those viewpoints without exact modality of control? So I think even that core capability of just view synthesis, it's sort of been this academic problem for a long time. But in the sense of, oh, you're going to do this really dense capture, generative view synthesis is relatively quite a new problem. And we just see so many people who, in this creative pipeline, stage workflow rate, I don't think there's a single person out there using one monolithic model, not even C-Dance or whatever, for their entire task. People will have this, like, you know, kind of bunch of storyboards and mood boards of images they pull out from like their favorite collection of image models. And then they'll go to different video tools and like build those together as keyframes. Then they'll go and like clip and edit those later, right? So we were seeing this like sort of, you know, niche, but very specific use case for Marvel as just providing that like sanity that you can ground your generations in some kind of 3D consistent world, right? People, you know, I don't know, I've, I've fought with, with various image models to ask them to like give me different viewpoints of a room. And every time you can just look and see, stable and even that one seed of a use case I think kind of signals that there's this value and there's hiding under the surface there, there's just... decades of people being used to persistent 3D state, like virtually modeling what they would be doing in the real world and having, you know, a stage and props and like elements there, whether it is for a movie or a show or marketing shot or like building out game environments like this, this statefulness and persistence is so key in how people think about spatial reasoning and like developing an environment over time. Like people don't think in this ephemeral, like generate a thing, generate a thing, like just throw it away, keep my text prompts. Like people this collection of assets and model a world in that way. So we're trying to provide, again, with this facial context mechanism and other things, we're trying to provide that level of control and precision and the ability to ingest different modalities of input starting with the post images. But we want to give people more control over the elements of the things in the scenes they're looking at and editing and interaction and all that as we go forward. And I think that it unlocks further use cases in those areas we're already seeing, but also kind of any place people want to create a virtual replication or like, you know, a pre-imagination of a real world space they need to build, right, for architecture and construction. Like I talked to a guy at some point building booths for conferences, right? There's just so many things in the world you don't think about need to be fabricated. And every single one of those basically goes through this like pretty painstaking virtual design phase. And of that process, like the part where you go into 3D software is kind of one of the most like arduous and like labor intensive parts right now. King. feedback on a 3D design from kind of like verbal commentary or sketches or really, really quick stuff you got from like a creative director or like a design director or an architect or whatever, like mapping that back into the 3D representation is like 95% of the work, right? You can have a meeting, get feedback, and then you go back and do a week of revisions. And that's just because like our software is kind of decades old at this point. And it just never became as intuitive as, you know, playing with Legos or like pottery or doing this stuff with your hands or sketching with a pencil. And this is the real place where AI can actually unlock a ton of value for people in their process, whether it's a creative application or something more industrial or design or whatever. And that like really motivates me to kind of build different flavors of our model to cater to those kind of people. Yeah, I can understand how it helps with the creatives because like Marvel did that and also how that extends to things like design or architecture. But they say you acquired a robotics company. We just talked about it too. I know, I know. But it's less obvious to me, especially in the context of Atlas, like how that maps to robotics. So if you wouldn't mind just penciling that out. Yeah, actually, Atlas is a key part of the puzzle. So yeah. We acquired this company that was formerly known as Cinex. And what is their key technology? Right now, their key technology is a system that goes from real to sim and then sim to real. And what does that mean in robotics situation? You want to train a robotic arm to, you know, figure out how to... do cabling, let's say, in an industrial setting. Well, you need a whole bunch of data to first train a robotic [30:00] policy to do these cable-to-play. Cables. Cabling. And then you want to evaluate if the robotic policy is doing a good job, and then you deploy the robot into the cabling process. In order to train, what this company, Cynix, and now our robotics team used to be doing is do exactly what Ben was saying, dense reconstruction. You take pictures of a situation and then try to reconstruct that environment. It's excruciatingly painful, takes a long time, laborious, and it really blocks the velocity of robotic simulation, real to simple. So Atlas really is the next platform. generation technology for that. And this is not just for robotics cabling or anything. We should The biggest problem right now in robotics is actually data. One day it'll be... But for now, it's data. Because it's so hard to collect real-world data where robots are operating. And in order to – not only you need to collect the data of, let's say, the cabling situation or dishwashing situation or whatever, there is also a very important step called randomization, is that you have to take the same environment and then randomize the conditions. literally only, you know, bend this way. It can bend a different way, or the box can have different sizes, colors, different lids, and all that, or in different parts of the scene. So you have to go through a reel-to-sim situation in order to get enough of that data, in addition to other data you can get from Internet. So this reel-to-sim step will be... really helped by Atlas. That's just the first part of this, is meeting the robotics needs in the current technology, because we don't yet have a frontier foundation model that's robust enough for robotics. But Atlas is an omni-model, is a multi-modal model. It takes different kinds of at this taking in data that's dynamical. and that can really start to bridge the gap between action planning, And, uh, and, um, and robotics and the Atlas output. So that's, On that roadmap, what are you going to say? Yeah, I was going to say there's something fundamentally different about training a robotics policy compared to really any other application in AI we've seen before. And that's like, if you're generating a piece of code, like you're generating an image, you're generating a video, the model is fundamentally creating this artifact. And that artifact, like, there's a lot of examples of artifacts that you can go out on the web or somewhere and collect, right? You want to generate images, there's a lot of images out there. You want to generate videos, there's a lot of videos out there. You want to generate a code base, there's a lot of code bases out there you can learn from. A robotics policy is something fundamentally different. It's not producing a static thing. It's instead a policy that's going to go out into the world, make actions, and like, try to achieve a goal. And the world's not always going to respond the way you expect, right? Unexpected stuff is going to happen. So a robotics policy is like fundamentally an agent that is out in the real world interacting with the real world and stuff happens. So you need like a critical part of that is those policies during training need to be exposed to every possible thing that could go wrong during their deployment. And that's where simulation is really key for robotics. And then there's two angles on that. One is the kind of classical simulation. You can go to your favorite physics engine and try to imagine creatively as a human designer, what are all the scenarios that might happen when achieving this task? And then try to write explicit code that models them all. That's one angle. And that's an interesting angle with coding agents like that actually gets supercharged, too. But there's another angle, which is tried a more data driven simulation, right? Like maybe we can't, can we have a learned model that can understand how the environment, how the world is going to respond to actions? And maybe it might respond in unexpected ways sometimes. then could we build these neural simulators that are trained on as much data as we can, then use these neural simulators, these learned neural simulators, you know, as a simulation bed to train robotic policies. and that's a really interesting future direction about this. But then it doesn't stop there, right? But once you have this learn simulator like this learn simulator kind of already has in its mental brain, it understands the world, it understands how the world is going to respond to actions and why doesn't this [35:00] later itself become the planner. That's kind of the core thesis that we've had around world models and their generality, that there's some core stuff that a model should understand around generating worlds, simulating them, understanding how they appear in different situations, and understanding how the world's going to respond to an action is highly related to imagining what kind of action I need to take to make the world respond in a particular way. One piece of feedback that I got, by the way, congrats on the It was overwhelmingly positive. I think it was probably the most significant model launch this year. And, you know, I said glowing things. But one person who's an expert in the space who I texted was like, what'd you think? And it's a great person. It's fantastic. It's amazing. But there needs to be more dynamics. And so it seemed to be at least the robotics case. But generally, it was kind of ideal to actually have a world that moves. And so maybe talk a little bit about that and then any other future directions that, A, you're comfortable sharing, but you think are worth talking through. Yeah, I mean, like, dynamics is clearly going to happen. Like, actually... We have baby dynamics. We actually do have baby dynamics already. And this is something I think people didn't quite appreciate. We didn't really highlight in the blog post. But, like, the previous Marble World model, it was, like, fundamentally static. Yeah. Like, the model just, like, could not handle any dynamics at all. And that was just, like, baked into the model architecture, baked into the training. Like, the whole thing was fundamentally static. We already knew that that was a big problem post-Marble. And we already fixed it in Atlas, right? Like, the Atlas architecture is already fundamentally supports dynamics. data fundamentally has dynamics. And if you look carefully in some of the videos that we even posted, there actually is. The waves. The waves. The water waves. Yeah. So like some of the examples, there's like waves in the water, like in some of the like air generated aerial views, there's like little cars moving around. So like dynamics is actually already in this model. By the way, dynamics seems very problematic to me if you're trying to reconstruct 3D from multiple views. It is. It is. Are these things like at odds? No. So actually one of our theses here is that like, fundamental 3D reconstruction, you actually want to have no dynamics. Like you want to be able to model like exact views of the scene with exact frozen time. But then like this is actually kind of a problem with our previous marble approach, right? Like there, like you can try to find data that's fully static, but that's really hard to scale and really hard to get more of. And the thing we realized is that even in the case where I want static output in the end, the best way to get it is actually expose the model to dynamics, right? Like expose the model to as much dynamic model figure out how to factor out the dynamic stuff. So especially in like the, this is like, and this is actually, so again, in the Atlas pre-training already, like it saw a ton of dynamics in the pre-training already. then the post-training that we did specific to this checkpoint and this release was focused a lot more on static stuff, focused a lot more on spatial movement and less on temporal. But like we already have, we already like, I'm pretty sure this, the pre-trained checkpoint already has a lot of latent dynamics in it. And this is something we're going to improve quite a lot going forward. So Ben, does that mean we're going to get 4D video? Yeah. You can walk around. You can see the smile on their face. So I actually can see if you just stopped now and you only did kind of bigger... you know, faster, better, you can build almost an entire industry. It feels like a very horizontal primitive. And if you did nothing else, but are there other things that are not just kind of bigger, faster that you're excited about for the applications you're focused on, which tend to be kind of more on the content created 3D side? Yeah, I'm really excited about pushing that kind of multimodal aspect. I think different modes of control is so critical here. I think like it's super underappreciated, especially in the academic community, how critical it is to add control conditioning to these models to kind of get out what's inside. I mean, honestly, this I even understand what those mean. To translate in layman's language is editability. I think editability is the key here. Yeah, so I mean this is something we've seen in single image models and starting this year in video models is starting to be unlocked in terms of, oh, like, getting that flavor of like multi-turn or really like intuitively interpreting like i want like this person this object and this thing to happen and kind of combining those all together in like one pastiche without having to do a lot of like manual work with the system like it just interprets it like kind of frontier image models are kind of there right for in terms of editing but we haven't seen that propagate out as strongly into video yet and then into world models right we've seen some really kind of toy examples of oh i can like put in a sentence and like, you know, a dinosaur appears or something with these like sort of real time models. But I want to like turn that up to really industrial strength and make that because like the trick here is you've got to add control, but not compromise the quality of the model or just becomes a party trick, basically. Like it's like no one is going to seriously think about swapping their like cutting edge frontier video model usage for your model if you give them extra knobs, but the quality degrades. So I think it's really that game of like, how can we maintain current model and then add all kinds of interesting stuff that people will ask us for in terms [40:00] like I want to interact with the scene or control the layout or control like the identity of the objects and the things that we're seeing within there or control time, right? And I think that's like an axis where it opens up like a ton of really interesting product and interface work. The more complexity you add there and richness in terms of kind of like enabling you to really think about like redesigning almost from scratch the way people interact with sort of like stateful, you know, 3D worlds in the computer. Like that's really the end goal here is getting like all the capabilities you need to build that kind of system. Awesome. Anything you'd add to that as far as new functionality that you'd be excited about that's not just bigger, better? I think for me, let's go back to the first principle of intelligence. Intelligence is not sitting there stuck and just seeing something or interpreting something when it comes to space and physical space, right? It's really this closing the loop between seeing and experiencing an interaction. So just thinking about going up that ladder is exactly what Ben said. I think one interesting notion there is this notion of AI completeness. Have you read this before? Yeah, yeah, I have. I hear about AI complete, by the way, in terms of LLMs, which is like you have to be basically the smartest LLM to answer the question what the smartest LLM will need to answer, or you have to solve general intelligence. No, no, it's a connection to Turing completeness, right? Like the idea being that a task is Turing complete, like in classical complexity theory, if I can take any class, any problem in this category, reduce to that one problem. So you can take any NP-hard problem and reduce it to 3SAT, therefore you can use 3SAT to solve any problem. Yeah. So then the kind of soft definition of AI completeness is there's this fundamental primitive, it's an AI task, but if I could solve this AI task in its full broadest generality, it would solve any intelligence problem. final sentence of the mystery novel is like, And the killer was predict the next token. So like you could basically like frame any kind of intelligence task in terms of that. So clearly next token prediction is something that people believe is AI complete. But I think that's something we're kind of realizing. And Ben was talking about this earlier today is like new view prediction. This primitive that we have in Atlas, especially generative new view prediction. This is also AI complete. And because I could take something like – You could have the movie and you do all of the frames of the movie and then the killer walks out and then you predict exactly who walks out. Exactly. Not just that. You could say, I want to have a world where Martine is writing a proof of the Riemann hypothesis. I don't know. And he solves the – So to take an evolutionary view, right? evolution had to solve by making animals move. Nature gave animals eyes, but nature didn't give trees eyes. Why? Because when you move, you see a new viewpoint. And that is whether you call it AI complete or intelligence complete. So we do believe very strongly that next viewpoint prediction is the equivalent of next token prediction. Amazing. Well, with that, congratulations to all of you on a phenomenal model launch. We're very excited for future model launches and thanks for coming. Thank you. Thanks so much. dot substack.com. Thanks again for listening. And I'll see you in the next episode. forward slash disclosures.