The Role of World Models: Teaching AI to See, Move, and Plan

世界模型的作用：教会AI感知、移动与规划 专家小组探讨了物理世界模型的技术机制与巨大经济潜力，并在数据获取、计算需求和安全风险方面将其与基于文本的LLM进行了对比。行业领袖讨论了相互竞争的数据策略，对比了基于网络视频训练的通用基础模型、具有地理参考的4D空间数字孪生以及专业越野机器人车队。讨论还深入剖析了关键的落地瓶颈，包括传感器碎片化、采集效率低下、边缘计算限制以及细棚架钢丝等从仿真到现实（sim-to-real）的异常情况。演讲者强调，尽管物理AI在农业、机器人和物流领域带来了规模达数万亿美元的巨大市场机遇，但现实部署仍需要强大的视觉定位能力、算法效率以及端侧本地的隐私保护。 世界模型代表了人工智能的新兴前沿，超越了基于文本的LLM，教会机器如何在物理现实中进行感知、模拟与规划。 (0:04) 业界主要通过两种理念推进世界建模：用于“真实仿真”定位的高精度、地理参考4D数字孪生，以及用于预测动态未来物理状态的通用基础模型。 (5:16) 现实世界真值数据的采集受到低效率的制约，典型的人工采集班次仅能产生2到3小时的可用数据，而机器人硬件之间严重缺乏传感器标准化使这一问题更加严峻。 (10:01) 与旨在避开所有障碍物的城市自动驾驶汽车不同，越野农业机器人必须主动判断并分类哪些有机障碍物（如茂密的灌木丛）可以直接穿透和破坏，同时还要忽略传感器伪影。 (13:48) 在现场仅部署由100至200台自主工业设备组成的车队，就能形成自我维持的数据飞轮，捕获关键的现实极端边界案例。 (13:48)

Industry practitioners frame world models as artificial intelligence's next physical frontier, dividing between predictive foundation models and precision spatial twins to bridge the gap between simulation and real-world autonomy. While digital models face virtual hallucination risks, embodied AI must resolve sensory, compute, and physical safety hurdles across multi-trillion-dollar industrial sectors. Predictive Foundation Models Versus Precision Spatial Twins (5:16) Panelists articulate distinct architectures for teaching machines physical reality. Oliver Cameron of Odyssey advocates for general-purpose foundation world models that forecast dynamic physical states across robotics, defense, and gaming, arguing that scaling models on broad data allows systems to predict multi-second horizons much like autonomous driving stacks. Conversely, Niantic Spatial CTO Brian McClendon prioritizes metric-accurate, georeferenced 4D digital twins that fuse aerial and ground-level views into photorealistic "real sim" environments tailored for precise machine localization and navigation. Cameron rejects a strict division between large language models and world models, projecting a symbiotic future where world models serve as interactive physics environments for reasoning agents. However, McClendon emphasizes that while text data for language models is largely exhausted, structured 3D spatial ground truth remains exceptionally scarce, driving demand for specialized real-to-sim pipelines such as 3D Gaussian splatting. Data Yield Limits and Internet-Scale Generalization (10:01) Training physical AI exposes severe data supply bottlenecks. Sce Pike of TELUS Digital notes that human collection shifts in homes or factories yield only two to three hours of usable signal per eight-hour recording period. Furthermore, hardware fragmentation across bespoke tactile gloves, teleoperated grippers, and differing camera depths forces teams to rebuild pipelines from scratch, prompting TELUS Digital to lead an industry standardization consortium. Pike also highlights the threat of data poisoning, noting automated checks are required to catch hostile inputs and degraded synthetic files. To scale training, Cameron asserts internet video is the only expanding medium diverse enough to teach models core physical concepts and novel entities, such as encountering an elephant outside urban settings. While Pike questions whether web video captures dexterous failure modes like fine kitchen manipulation, Cameron maintains web scale must instill baseline physical causality and intuitive physics, which developers can subsequently augment with curated expert ground truth. Unstructured Edge Realities and Inverted Safety Rules (13:48) Off-road robotics face operational demands that directly invert standard autonomous vehicle rules. Agtonomy CEO Tim Bucher explains that while road-bound vehicles must avoid all physical contact, agricultural equipment operating in high-value specialty crops must actively classify and push through or destroy organic matter like brush and weeds while disregarding visual artifacts like spiderwebs. Deploying 100 to 200 field tractors creates a self-sustaining data flywheel, yet unpredictable edge cases persist. Bucher cites a recent incident where an autonomous tractor struck a thin trellis wire across a missing crop row because neither onboard sensors nor the simulation registered the wire. To manage occlusions, systems combine contextual environmental priors with transformer-based vision models that infer geometric continuity. Pike emphasizes that physical AI carries far higher stakes than software models, as perceptual miscalculations cause tangible infrastructure damage, destroyed harvests, or bodily injury across agriculture and construction sectors that Bucher notes represent $15 trillion annual addressable markets each. Edge Compute Limits and On-Device Privacy (28:50) Operating under dense canopies and in remote fields prevents continuous cloud offloading, forcing physical AI into stringent edge compute trade-offs. Bucher explains that Agtonomy handles processing saturation on hardware like NVIDIA Orin and Thor by utilizing "self-recovering interrupts"—controlled machine pauses lasting between 10 milliseconds and 500 milliseconds that allow edge queues to clear safely. Addressing infrastructure moats where foundational labs commit 50% to 60% of mega-round funding to GPU reservations, McClendon argues that compute rationing enforces beneficial algorithmic discipline, driving Niantic toward single-pass, feed-forward architectures capable of mapping global geometry efficiently. In response to consumer adoption questions, McClendon stresses that long-term domestic robotics must perform inference entirely on-device, as transmitting private home layouts and persistent spatial models to the cloud would inevitably trigger consumer pushback and regulatory friction. Deploying physical AI into multi-trillion-dollar industries ultimately depends on balancing massive predictive models with the unforgiving realities of the physical world. While web-scale video and 4D digital twins advance simulation capabilities, commercial viability remains bound to edge compute constraints, standardized ground-truth data pipelines, and strict physical safety thresholds.

业内专家指出，世界模型正成为具身智能和物理AI的核心演进方向，但行业在发展路线上分化为高精度4D空间数字孪生与基于海量视频的通用物理预测两大阵营。与处理文本的大语言模型不同，物理AI面临严重的数据标准化缺失、边缘算力硬约束及现实破坏风险，其实际落地高度依赖端侧推理与可靠的环境真值反馈。 空间数字孪生与通用物理预测的技术路径 (5:16) Niantic Spatial与Odyssey代表了构建物理世界模型的两种不同哲学。Niantic CTO Brian McClendon主张以厘米级、带地理参考的4D空间孪生为核心，通过融合地面视角与无人机倾斜摄影数据，并结合3D高斯泼溅技术生成“真实仿真”（real sim）环境。该架构专为解决跨天气与光照条件下的传感器无关特征匹配而设计，使机器人能在高精度模拟空间中完成训练并无缝映射到现实。McClendon同时强调，团队采用单向的前馈架构以实现更低成本且高效的实时设备端推理。 与此相对，Odyssey联合创始人Oliver Cameron提出基于海量互联网视频训练跨领域的通用基座世界模型。Cameron结合其在自动驾驶领域（曾创立初创公司并加入Cruise）的经验指出，自动驾驶已经验证了系统在窄域道路内预测数秒未来物理状态的可行性。通过吸收海量互联网视频，模型不仅能学习特定场景，还能内化因果律与基本物理动力学规律，并借由诸如识别“大象”等通用概念的学习，实现面对未曾预见的物理实体时的泛化推理能力。 数据瓶颈：硬件异构与低良率采集挑战 (10:01) TELUS Digital副总裁Sce Pike指出，物理AI在现实环境中受挫的根源往往不在于神经网络架构，而在于训练数据覆盖的严重不足。Pike揭示了现场采集的效率困境：在长达8小时的人工作业采集中（如工厂操作或家庭清洁），通常仅有2至3小时能产生高信噪比的有效训练信号，造成巨大的人工与算力浪费。与此同时，机器人硬件严重缺乏标准化，触觉手套、遥操作夹爪、腕部传感器及不同视场角的深度相机各行其道，迫使采集管道必须针对异构布局从零构建。 针对数据安全，Pike指出数据投毒已成为物理AI领域的现实威胁。TELUS Digital通过自动化AI检测系统对全球众包文件进行逐一审查，以防范恶意样本及低多样性的合成图像干扰传感器训练。尽管Cameron认为互联网视频提供了指数级增长的多样性，但他同样承认，开放网络数据缺乏灵巧操作和安全关键领域的微观互动细节，必须结合高精度的专业示范与物理真值进行补充。为此，TELUS正牵头建立端到端行业联盟，并与Lightwheel等机构合作推进人形机器人统一数据标准的制定。 非结构化环境落地：逆向障碍与边缘算力 (13:48) 在农业与越野工业等非结构化场景中，物理AI面临着与城市道路截然不同的运行逻辑。Agtonomy首席执行官Tim Bucher指出，城市自动驾驶的首要原则是规避一切碰撞，但农业机器人（部署于Kubota和Bobcat等商业农机）必须执行“逆向障碍处理”——视觉模型必须精准区分破坏性硬障碍与必须强行粉碎穿透的有机物（如割草作业中的浓密灌木丛或遮挡镜头的蛛网）。此外，现实世界的物理混沌极难在仿真中完全重现，例如果园中未悬挂树木但横跨作业通道的极细修剪钢丝，曾导致依赖行识别的无人拖拉机直接撞入。 严苛的边缘计算限制进一步加剧了系统部署难度。在密集树冠遮挡GPS信号、蜂窝或星链网络随时中断的田间，无人农机无法依赖云端数据中心，且必须动态监测和控制数千种机械农具。Bucher介绍，Agtonomy利用工业场景允许短暂等待的特性，设计了10至500毫秒的“自恢复中断”（self-recovering interrupt）机制，在车载算力（如NVIDIA Orin或Thor）饱和时主动暂停以重算状态。Bucher表示，仅需部署100至200台无人设备，车队就能形成可持续捕获罕见长尾案例的物理数据飞轮。 物理AI对比LLM：实体风险与端侧隐私 (33:23) 与会专家一致认为，将物理世界模型等同于大语言模型（LLM）是对具身智能的误解。Pike与Cameron指出，LLM处理的是数字化文本，其失败主要表现为文字幻觉与认知误导；而物理AI运行于实体世界，感知或运动学测算的失误会直接导致农作物损毁、设备破坏乃至人身伤亡。此外，LLM正面临公开网络文本耗尽的瓶颈，而空间世界模型面临的则是结构化、带3D几何标注的高保真真值极度匮乏。尽管开发物理AI需要呈指数级增长的数据与算力，但Bucher援引英伟达CEO黄仁勋的预测指出，仅农业与建筑业各自就拥有15万亿美元的年总体有效市场（TAM），整个物理AI潜在市场规模高达50万亿美元。 在用户信任与隐私层面，前iRobot团队成员Chuck Hudson提出的家庭地图安全问题引发了深入探讨。McClendon强调，尽管开发初期依赖云端集中式训练，但面向家庭消费场景的最终解决方案必须是将世界模型与空间计算完全置于本地设备端运行。若家用机器人将私密空间的几何布局或原始视频回传至云端，将引发巨大的用户抵触；唯有零数据外泄的端侧独立推理，才能使物理具身系统跨越公众接受门槛。 本次研讨表明，世界模型正从概念构想走向高风险的实体工业部署，但行业在基座预测模型与高精空间孪生的技术路线上尚未达成共识。由于无限算力的假设在真实边缘端无法成立，未来物理AI的规模化应用将取决于三大未决瓶颈的突破：跨异构硬件的数据采集标准确立、受限算力环境下的端侧实时推理优化，以及确保完全本地化隐私防护的工程落地。

## Background 该内容由频道“Ai4”于 2026 年 8 月 22 日发布在 YouTube 平台上。页面简介指出，本场专题圆桌汇聚了来自 Niantic Spatial、Odyssey、Agtonomy 和 TELUS Digital 的业界专家，重点探讨空间智能与世界模型如何赋能机器人、增强现实（AR）及自动驾驶等下一代智能系统，并深入剖析数据集构建、仿真与真实世界的权衡以及具身 AI 的算力边界等前沿技术课题。 ## TL;DR 本场圆桌深入探讨了世界模型的技术路线分歧，详细对比了 Niantic 的 4D 高精地理语义建图与 Odyssey 的跨场景时空预测基座模型。嘉宾们直面具身智能数据采集有效率低、硬件标准碎片化等残酷现实，并辩证分析了互联网海量视频与现场真值数据在物理概念泛化中的协同价值。面对“模拟到现实”（Sim-to-Real）的鸿沟与端侧受限算力，讨论分享了 3D 高斯泼溅渲染与自我恢复中断等前沿实战策略。专家们一致认为物理 AI 面临着远比大语言模型更严苛的容错标准，但其在农业与工业等领域的深远变革孕育着数十万亿美元级的巨大商业蓝海。 ## Atmosphere & style 整场对话硬核专业且妙趣横生，嘉宾们摒弃了宏观的概念炒作，坦诚分享了一线工程落地的踩坑血泪史。从推着婴儿车去果园测数据、自动拖拉机撞上极细悬空铁丝的尴尬，到调侃农田作业“遇响尾蛇直接轧过去”的幽默，老兵们在风趣的互捧与辩论中展现出极高的技术密度。现场既充斥着对算力卡脖子与数据浪费的冷峻清醒，又洋溢着开辟物理 AI 新边疆的笃定与激情，是一场信息量极其扎实的思想碰撞。 ## Key takeaways - [05:11](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=311s) 世界模型的技术路线存在分歧：Niantic 押注于融合多模态视角的 4D 高精地理参考与空间语义孪生，而 Odyssey 则致力于构建跨场景预测未来演化状态的通用物理基座模型。 - [10:42](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=642s) 物理世界数据采集面临严峻的废品率与硬件碎片化挑战，单次 8 小时班次通常仅能沉淀 2 至 3 小时有效真值数据，行业亟需建立端到端的数据标准化与评测体系。 - [13:48](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=828s) 非结构化农业场景颠覆了传统自动驾驶的避障逻辑，机器需要主动识别并粉碎有机植被；且仅需 100 至 200 台一线作业设备即可建立起有效的数据飞轮。 - [16:36](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=996s) Odyssey 提出互联网海量视频是学习物理世界极端多样性的最佳载体，通过“大象理论”预训练赋予系统面对罕见长尾实体的通用物理表征与泛化能力。 - [19:21](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1161s) 无人机倾斜航拍视角与地面第一人称数据的协同能够高效建立空间三维基底，但针孔摄像头、全景相机等异构传感器和天气剧变仍给世界模型的泛化带来巨大阻碍。 - [21:39](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1299s) 互联网海量视频擅长构建物理法则与因果动态等通用概念库，但对于触觉反馈与厨房切菜等人形机器人精细灵巧操作，仍必须补充细分领域的专家真值数据。 - [23:43](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1423s) 果园极细悬空铁丝等长尾失效案例表明虚拟仿真无法穷尽现实世界，必须加快一线实体部署以构建不断将真实边缘案例反哺给仿真的数据闭环。 - [26:12](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1572s) 3D Gaussian Splatting 等神经渲染技术大幅缩小了 Real-to-Sim 视觉代差，Sim-to-Real 鸿沟的核心瓶颈正逐步从纯算法演变为端侧硬件算力负荷与模型参数规模的博弈。 - [28:50](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1730s) 在农田断网与端侧算力超载时，非道路机械可利用毫秒级“自我恢复中断”暂停作业以对齐感知状态；同时数万张 GPU 的储备已演变为决定 AI 实验室生死的战略壁垒。 - [32:43](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1963s) 世界模型与大语言模型并非二元对立，世界模型可作为 LLM 进行物理推理的终极学习沙盒；且物理 AI 误判会导致不可逆的灾难性破坏，对真值数据的严谨性要求远高于纯文本。 - [32:43](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1963s) 物理 AI 直面农业、建筑等年产值超 15 万亿美元的庞大实体产业，在应对全球熟练劳动力短缺方面展现出远超传统乘用车自动驾驶与 LLM 的万亿美元级商业潜力。 - [40:42](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=2442s) 消费者对家庭隐私泄露的顾虑促使建图感知与世界模型推理必须严格限制在端侧本地运行，避免敏感家庭视觉数据回传云端。 - [40:42](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=2442s) 世界模型面临 AI 生成虚假数据污染与场景多样性削弱的“隐性数据投毒”，对多模态数据进行防伪与清洗已成为预处理阶段最主要的算力开销之一。 - [45:50](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=2750s) 现代 Transformer 架构通过海量预训练习得了物理世界的空间几何连续性先验，能够在车载传感器像素残缺或遮挡时，自主推断并补全悬空电线等未知障碍物信息。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=4s) 开场介绍：全明星阵容与世界模型的火热探讨 圆桌讨论在全天峰会的尾声正式开启。NBC News 驻纽约的 AI 领域记者 Jared Perlo 担任主持人，向坚持到第一天日程结束的现场观众致谢。Jared 直言，虽然每位主持人都喜欢宣称自己的嘉宾阵容豪华，但本场关于“世界模型”（World Models）的技术专场确实实至名归。台上的嘉宾覆盖了从基础数据、底层架构到前沿落地的整个世界模型技术栈。本场属于技术 Track，讨论不会停留在泛泛的宏观层面，而是会深入技术细节与实战经验。 Jared 鼓励台上的嘉宾畅所欲言，展开激烈的观点碰撞（"hot takes very much encouraged"），分享真实的行业洞察、踩坑故事以及未来愿景。会议最后预留了 5 至 10 分钟的现场观众 Q&A 环节，由现场工作人员 Maria 协助递送麦克风。 --- ### [01:26](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=86s) 嘉宾破冰：横跨四十年地图技术、自动驾驶与真实版“开心农场” 主持人开启了第一轮快速破冰，要求四位嘉宾依次自我介绍，并分享各自案头目前最关注的一项核心任务： Brian McClendon（Niantic Spatial 首席技术官）：拥有长达 40 年的三维与地图构建经验。早年间在 Silicon Graphics 开发飞行模拟器；随后在 Google 领导了 10 年 Google 地图（Google Maps）与 Google 地球（Google Earth）的研发工作；之后加入打造了《Pokémon GO》的 Niantic Labs。如今团队从原公司独立出来成立了 Niantic Spatial，专注于构建面向物理世界的 4D 模型，目标是实现三维重建，帮助人类和具身机器在周围环境中实现高精度的定位与语义理解。 Oliver Cameron（Odyssey 联合创始人兼 CEO）：Odyssey 是一家专注于世界模型的实验室，他们坚信世界模型必将超越现有的语言模型。Oliver 本人深耕无人驾驶技术长达 8 年，2015 年创立了自己的自动驾驶初创企业，后来该团队被 Cruise 收购，他随即在 Cruise 度过了两年半时间，负责自动驾驶系统的规模化落地。 Sce Pike（TELUS Digital AI 增长与解决方案副总裁）：TELUS Digital 的核心业务是为机器人和自动驾驶公司构建用于训练世界模型的地面真值（Ground Truth）数据。Sce 当下最紧迫的任务是“数据标准化”（Data Standardization）。各家客户带着截然不同的传感器配置上门，在真值数据的采集、处理方式上诉求千差万别。为此，他们正在牵头组建一个贯穿全流程的产业联盟，涵盖从原始数据采集一直到工厂制造车间，旨在为机器人和世界模型领域制定数据采集、处理与评估（Eval）的端到端标准。 Tim Bucher（Agtonomy 联合创始人兼 CEO）：Agtonomy 的定位是让拖拉机实现自主作业。他们自己不造拖拉机，而是专注于开发“物理 AI 层”（Physical AI layer），与 Kubota（久保田）、Bobcat（山猫）等一级农机设备制造商深度合作。目前其商业化自主拖拉机已在美国和澳大利亚落地运营，并计划进军更多国家。Tim 幽默地自嘲是“像所有农民一样顺理成章转行搞高科技”的典型代表：他自幼经营农场，同时也创办过建筑公司，但在硅谷创业历程中曾先后为史蒂夫·乔布斯（NeXT 与 Apple 两次）、比尔·盖茨、迈克尔·戴尔以及 EA 创始人 Trip Hawkins 工作。他调侃自己的看家本领是做游戏系统，而现在在农田里指挥自动驾驶拖拉机，本质上就是做“现实版的《开心农场》（Farmville for real）”。 --- ### [05:11](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=311s) 概念分歧：Niantic Spatial 的 4D 地图与 Odyssey 的预测性基座模型 针对外界将世界模型类比为当年“大数据”或“生成式 AI”的泛化狂潮，主持人指出世界模型正成为最新、最火热的流行词，但各家对世界模型的定义和业务实现存在巨大差异。 Brian McClendon 阐述了 Niantic Spatial 的核心押注。与纯粹基于视觉生成的路径不同，他们的核心立足于“带有地理参考标记的高精地图数据”（Georeferenced map data and accuracy）。当年 Google Maps 和 Google Earth 耗费多年为人类绘制了高精度的数字孪生，而今天面对机器人、自动驾驶车辆和 AI 智能体，世界需要一个更加精准、实时并且具备语义理解能力的模型，使大型语言模型也能与之交互。Niantic Spatial 训练通用系统同时理解地表视角（Ground-level）与俯瞰视角（Sky-level）所看到的世界，并将所有数据沉淀进长期的 4D 动态世界模型中。该模型具有多模态特性，不仅能输出相机姿态，更重要的是输出专供机器人和 AI 消费的空间物理环境。团队正大力投入“真实模拟”（Real Sim），生成精度足以在特定真实场景中训练机器人的虚拟仿真环境。 Oliver Cameron 则从自动驾驶的视角给出了截然不同的定义。当人类在路上开车时，面对不可预测的行人和车辆，核心能力在于“预测未来”（Predict what's going to happen next）——例如预判行人在未来 5 秒内的走位或车辆轨迹。2015 年自动驾驶刚起步时，机器仅能预测未来几毫秒，可靠性极低；但随着过去五到七年间 AI 模型与数据集的大幅扩展，机器在演化预测上已达到超越人类的精确度，让人安心把孩子托付给无人车。然而，这种能力的致命短板在于它受限于“极度狭窄的道路场景”，无法直接迁移到办公室、国防或医疗领域。因此，Odyssey 的目标是打造跨任务的通用世界基座模型（Foundation World Model）：无论是机器人推理下一步、无人车预演轨迹，还是游戏渲染、医疗与科研模拟，该世界模型都旨在直接理解物理规律并仿真出多种可能的未来。 ```mermaid flowchart TD accTitle: 两种不同的世界模型构建范式对比 subgraph Niantic["Niantic Spatial：空间几何与语义孪生"] N1["多模态视角数据<br/>地面视角 + 卫星俯瞰"] --> N2["4D 物理世界模型<br/>精准几何坐标 + 语义标注"] N2 --> N3["应用落地<br/>Real Sim 机器人仿真 / 具身定位"] end subgraph Odyssey["Odyssey：未来状态预测基座模型"] O1["海量时空演变序列<br/>自动驾驶预判经验泛化"] --> O2["跨领域世界基座模型<br/>时空演进与物理规律推演"] O2 --> O3["应用落地<br/>全场景行动预测 / 仿真生成"] end ``` - **高精地理参考（Georeferencing）**：Niantic 的立足之本在于让世界模型拥有确定性的毫米级空间坐标，与语义标签结合，供具身 AI 检索和定位。 - **预测视界（Predictive Horizon）**：从早期毫秒级脆弱预测，演进至具备超越人类精度的数秒级多未来概率推演。 - **泛化瓶颈**：自动驾驶在道路单点上验证了物理预测模型的可行性，但要成为真正的 World Model，必须打破道路域限制，跨入通用物理世界。 --- ### [09:58](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=598s) 数据争议的前奏：海量视频与具身采集的博弈 探讨完架构差异后，主持人 Jared 迅速切入核心痛点——数据。世界模型在本质上就是一场“数据战”（Data play）。行业内目前存在明显的技术流派分歧：一种倾向于通过“投喂”海量 YouTube 公开视频来学习物理规律，另一种则坚持必须让具身机器人或工人在真实物理世界中佩戴传感器进行现场采集（例如在印度或中国的工厂中由工人佩戴相机收集真实工业操作轨迹）。Jared 借此将话头转向 TELUS Digital 的 Sce Pike，邀请这位地面真值数据专家详细拆解两种路线的优劣与真实数据采集面临的严峻挑战。 ### [10:42](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=642s) 数据收集的残酷现实：废品率、传感器混乱与标准化困境 主持人 Jared Perlo 向 TELUS Digital 的 Sce Pike 提问，希望了解其在为世界模型构建数据集时的实际日常工作。Sce Pike 首先肯定了无论是预测性世界模型（predictive world models）还是生成式世界模型（generative world models），核心瓶颈都在于数据本身——当模型在实际任务中失败时，问题往往不在于模型架构设计，而在于“数据覆盖率不足”（data coverage failure）。 随后，Sce 揭露了当前具身智能数据采集领域一个鲜为人知的“残酷秘密”（nasty secret）：数据浪费极其严重。无论是让工人在工厂里头戴采集设备走动，还是像纽约某家初创公司那样“为居民免费打扫房屋以拍摄第一人称视频”，在长达 8 小时的连续采集班次中，真正能够用于模型训练的有效高质量数据往往只有 2 到 3 个小时，其余大半都是无用废料。与大语言模型可以直接抓取吸收互联网上的海量文本不同，物理世界的智能需要数百万小时具有真值（ground truth）的多模态数据。目前行业由于缺乏统一的质量评估（Eval）与基准测试（Benchmarking），大家实际上处于一种“胡乱尝试、看哪个能碰巧奏效”（throwing spaghetti at the wall）的摸索状态。 Sce 进一步指出，当前硬件配置极度碎片化：不同客户采用的传感器方案千差万别，从遥操作机械爪、触觉手套、手腕摄像头到各类头戴设备，视场角（FOV）与深度参数均未统一。从近距离的精细灵巧操作，到类似 Niantic 关注的远距离自动驾驶场景，每一次对接新客户几乎都要从零开始。为此，TELUS Digital 正致力于推动数据标准化流程并组建端到端联盟，以规范数据的采集、清洗、质检与入模评估，防止低质数据导致模型性能退化。 ```mermaid flowchart TD accTitle: 物理世界数据采集与清洗管线困境 subgraph RawData ["原始数据采集 (8小时班次)"] A["工厂头戴设备采集"] B["家政入户实景拍摄"] C["遥操作/触觉手套数据"] end subgraph Filtration ["数据筛选与损耗 (现实瓶颈)"] D["无效/模糊/冗余数据<br/>(约 5-6 小时废弃)"] E["高价值真值数据<br/>(仅 2-3 小时可用)"] end subgraph Standardization ["数据标准化需求"] F["统一 FOV / 深度 / 传感器标定"] G["建立质量评估与基准评测 (Eval)"] end subgraph Training ["注入世界模型"] H["预测性 / 生成式世界模型训练"] end A & B & C --> D & E E --> F --> G --> H ``` - **数据利用率极低**：8 小时的真实场景采集通常仅能沉淀出 2 至 3 小时有效训练数据。 - **硬件配置碎片化**：手腕相机、头戴传感器、触觉手套与机械夹爪的视场角（FOV）与深度标准各异，缺乏跨行业通用标准。 - **行业痛点**：目前急需建立从采集、质检到评估（Eval）的端到端数据标准化体系，以支持数百万小时真实世界数据的有效扩增。 --- ### [13:48](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=828s) 农业与非结构化越野环境：主动碰撞与数据飞轮的构建 Agtonomy 首席执行官 Tim Bucher 分享了其在农业与非结构化越野工业场景（unstructured off-road industrial markets）中的数据探索历程。他幽默地回顾了数据采集手段的演进：五年前，团队甚至会将各种摄像头和传感器绑在类似三轮婴儿推车（baby buggy）的装置上，推着穿过各类高价值作物农田（如水果、坚果和葡萄园），以完成最原始的视觉数据冷启动。 谈到数据飞轮（data flywheel）的建立，Tim 指出，一旦越过初始阶段并让实际机器人下场作业，所需的“临界规模”（critical mass）其实并没有想象中那么遥不可及——只需 100 到 200 台农田作业机器人，就能形成源源不断的数据回流，捕捉到极其丰富的环境动态。 然而，非结构化农业场景带来了颠覆常规自动驾驶认知的独特技术挑战： 1. **从“避障”到“破坏与粉碎”**：自动驾驶汽车（如 Oliver Cameron 所探讨的城市道路）核心目标是避让行人与障碍物；但在农业作业中，机器的任务反而是“主动撞击并摧毁目标”，例如使用甩刀割草机（flail mower）直接粉碎大片灌木。 2. **穿透性与有机材质判断**：模型必须精准区分哪些障碍物必须规避，哪些属于可以直接推过去或粉碎的有机植被（organic matter）。 3. **特异性边缘案例**：Tim 调侃称自己能大谈特谈农田里“蜘蛛网”对传感器造成的巨大困扰，引得全场欢笑。他强调，这类越野环境的世界模型虽然极其复杂，但直接关乎粮食安全、基础设施建设与全球物流运输等人类社会运转的基础。 --- ### [16:36](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=996s) 互联网视频作为世界模型源泉：从“大象理论”看极端泛化 主持人将话题转向 Odyssey 首席执行官 Oliver Cameron，询问 Odyssey 如何看待数据飞轮以及如何在海量数据中“去粗取精”（sorting the wheat from the chaff）。Oliver 提出，构建一个真正通用且强大的世界模型，核心在于找到一个能够承载物理世界极端多样性的数据源，而 Odyssey 坚信**互联网视频（Internet video）**就是最佳答案。 Oliver 解释道，互联网视频不仅极其庞大、涵盖了人类所能想象到的一切物理现象，而且其增长速度远超传统文本数据。尽管直接建模如此庞杂、混乱的互联网视频来生成稳定且高性能的世界模型在算法上面临巨大挑战，但这正是正确的科研探索方向。 为了论证这一观点，Oliver 提出了一个生动的“大象类比”： - 如果一辆自动驾驶汽车仅在旧金山市区采集道路数据，它几乎一辈子也遇不到一头大象； - 但如果基于包含海量互联网视频的底层世界模型进行训练，模型在预训练阶段就已经见过成千上万头大象； - 当自动驾驶系统在现实中遇到极其罕见的未见物体时，它调用大象相关概念并不意味着它认为眼前真的是一头象，而是模型已经具备了理解“未知大型动态实体”的通用物理表征能力。 因此，真正的世界模型必须从最广阔、最多样化的数据源中汲取养分，而互联网正是目前唯一具备这种多样性规模的载体。 --- ### [19:21](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1161s) 航拍无人机与异构传感器：多角度扫描与环境多变性 Niantic Spatial 首席技术官 Brian McClendon 阐述了 Niantic 在空间智能与数据构建上的独特路径。除了多年来通过各类终端积累的海量地面级视频（ground level video）外，Niantic 近期通过与 Spexigon（Specsy）等无人机数据提供商合作，在全球多个国家迅速获取了城市级的广域航拍地理覆盖。 Brian 特别强调了无人机航拍斜拍视角（oblique angle）的技术价值： - **空地协同**：倾斜视角并非简单的垂直俯视，而是带有俯角的三维立体视野，这使得无人机采集的几何结构能够与地面第一人称视角的图像产生强烈的关联与融合； - **极高覆盖效率**：无人机能够以极快的速度横扫整座城市，迅速建立起高分辨率、高多样性的空间基底。 针对前面嘉宾提到的挑战，Brian 呼应了**传感器异构性（sensor variation）**这一巨大痛点。他指出，传统自动驾驶公司享有“针对固定专属传感器套件进行专属优化与训练”的奢侈特权；但在场的其他世界模型公司都没有这种待遇——他们必须直面来自普通手机、便携传感器、针孔摄像头（pinhole cameras）、360 度全景相机以及无人机长焦/广角镜头的各种随机数据输入。 此外，Brian 指出环境变量的剧烈扰动同样关键，包括同一地点在不同天气、光照、季节和时间段下的巨大视觉差异。尽管 Niantic 在多时空同一位置的数据积累上具备深厚优势，但他直言行业目前仍只是触及了冰山一角，世界模型对于高覆盖数据的需求永无止境。 ### [21:39](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1299s) 互联网视频能否捕捉细微操作与物理概念 讨论聚焦于互联网视频数据在训练世界模型时的边界，特别是对于极端关键失效模式（critical failure modes）的捕捉能力。现实中的数据需求跨度极大：从无人机或自动驾驶汽车覆盖的广域鸟瞰视角（wide-area view），一路延伸至第一人称视角的个人细微动作（egocentric view）。世界模型既要服务于宏观层面的自动驾驶，也要支持需要触觉反馈与极其灵巧精细动作（fine dextrous movements）的人形机器人（比如切洋葱等精细厨房动作）。 针对互联网视频是否真正能捕捉到精细操作中的关键失效模式，Oliver Cameron 坦言行业目前尚无确切答案。他认为世界模型的核心在于学习一套通用的“概念库”——理解物理法则、因果关系、动态机制以及人类行为。由于互联网规模的视频极其浩瀚，模型如果真正吸收了这些底层概念，便具备了迁移并泛化到全新未知场景的能力。不过，单纯依赖公开网络数据并不足够，团队依然会使用特定领域的“专家数据”与真实基准数据（ground truth data）进行增强，从而补全细分领域的认知短板。 ```mermaid flowchart TD accTitle: 数据源与世界模型构建流程 subgraph DataSources["多模态与多层级数据源"] A["互联网海量视频<br/>（宏观动态/物理概念/通用常识）"] B["专家与第一人称微观数据<br/>（精细操作/触觉反馈/基准真值）"] end subgraph CoreModel["世界模型核心认知"] C["通用概念学习<br/>（因果推断、物理机制、环境动态）"] end subgraph Applications["下游多场景落地"] D["广域自主系统<br/>（自动驾驶、农机作业）"] E["灵巧操作实体<br/>（人形机器人、细微装配）"] end A --> C B --> C C --> D C --> E ``` --- ### [23:43](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1423s) 从模拟到现实：果园里的意外与 Sim-to-Real 鸿沟 关于学术界讨论已久的“模拟到真实”（Sim-to-Real）鸿沟，Tim Bucher 分享了 Agtonomy 在非结构化越野环境中遇到的最新真实案例。尽管仿真系统日趋完善，但现实世界总会抛出工程师从未设想过的边缘情况。Agtonomy 的工作流程通常先利用航拍数据建立果园的基本行距结构，随后车辆在完成自身定位后，会借助作物本身的几何结构进行精细导航和作业。 就在讨论前一天，系统在一处果园遭遇了意想不到的障碍：果园端头的一排树木已被砍伐，但原本用于支撑果树棚架系统（trellis system）的极细金属钢丝依然悬空拉伸着。不仅模拟环境中未曾构建这一场景，车载传感器也未能识别出这些极其细小的悬空导线，导致自动作业车直接撞上了铁丝（所幸无人受伤，设备也未受损）。这类状况让开发者既沮丧又兴奋——它证明了仅靠模拟永远无法覆盖所有场景，必须通过将更多作业车辆推向一线，建立不断将真实边缘案例（比如“葡萄园里突然闯入大象”）反哺给仿真系统的数据飞轮。 - 航拍与作物结构感知：先利用空中图像进行目标识别以规划路径框架，再结合近场传感器捕捉作物实际物理结构进行导航。 - 极细障碍物识别失效：极细的棚架铁丝处于传感器检测与传统仿真场景的盲区，构成了典型的长尾失效模式。 - 数据飞轮机制：越野非结构化环境充满未知，唯有加快实地部署规模，才能用真实世界捕获的极端数据持续扩充模拟库。 --- ### [26:12](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1572s) 弥合技术鸿沟：3D Gaussian Splatting 与模型规模的权衡 谈及 Sim-to-Real 技术演进的历史脉络，Brian McClendon 回忆道，早期的自动驾驶车辆虽然搭载多个传感器，但感知算法往往各自为战，因为当时传感器融合与标定极为困难；而如今多模态模型已经让真正的传感器融合成为提升精度的核心手段。更具突破性的是由真实到模拟（Real-to-Sim）的视觉渲染转变：过去几十年依赖多边形贴图的合成场景虽然廉价快速，但与真实视觉存在代差。如今采用 3D 高斯泼溅（3D Gaussian Splatting）等神经渲染技术，能够生成更贴近真实光学特征的环境呈现，使机器人传感器直接利用逼真的视觉输入进行训练，大幅提升了对现实世界的解析精度。 针对这一鸿沟，Oliver Cameron 给出了一个大胆的工程视角：如果硬件资源不受限制，能够直接在每台机器人身上搭载一个 500 亿（50B）参数的大模型外加一座小型数据中心，Sim-to-Real 鸿沟在算法层面或许早已不复存在。现实的核心瓶颈在于机器人本体的算力负荷与实时响应要求，开发者必须在模型参数规模与车载硬件性能之间反复博弈。不过在过去一年左右的时间里，随着算法优化和渲染技术的迭代，这一鸿沟正在显著缩小。 --- ### [28:50](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1730s) 端侧算力瓶颈与应对策略：“暂停作弊”与 GPU 战略壁垒 在受限硬件上运行世界模型构成了当前落地的核心瓶颈。Tim Bucher 强调，农用拖拉机与固定形态的人形机器人不同，它必须外接成千上万种形态各异的农具（implements），算法必须实时适应这些负载变化。更严苛的是，农田场景不仅没有常驻数据中心支持，还常常面临树冠遮挡导致 GPS 信号丢失、星链（Starlink）与蜂窝网络双双断连的极端工况。 面对端侧算力不足的困境，农用非结构化场景拥有一项独特优势——作业车辆允许在瞬间“暂停”。Agtonomy 利用一种名为“自我恢复中断”（self-recovering interrupts）的机制来巧妙地“作弊”：当车载算力超载或感知出现短时不确定性时，车辆会刹停几十毫秒至半秒来完成密集计算与状态对齐。尽管频繁暂停会降低作业效率，但在端侧芯片从 Nvidia Jetson Orin 迈向 Nvidia Thor 的演进期，这种机制兼顾了功能安全与计算开销。 在算力供给的宏观格局上，Oliver 指出算力储备已演变为决定公司生死的战略优势。头部 AI 实验室动辄数百亿美元的融资中，超过半数资金直接流向了对近一两年 GPU 算力供应的锁定，使得获取数万张 GPU 的门槛极高。Sce Pike 则风趣地插话称 TELUS 在加拿大拥有主权 AI 工厂（Sovereign AI Factory），随时欢迎算力合作。Brian 总结道，对于缺乏数百亿美元硬件预算的团队而言，对算力的严格配给反而倒逼团队在算法效率、稀疏化与架构优化上取得关键突破。 - 农具多样性挑战：拖拉机需适配数千种不同作业挂件，对世界模型的动态适应与物理交互理解提出了极高要求。 - 自我恢复中断（Self-recovering Interrupts）：通过几十毫秒至半秒的短暂停顿为端侧计算腾出缓冲窗口，在无网络和弱 GPS 环境下保障作业安全。 - 芯片代际跃迁：边缘计算平台正逐步从当前主流硬件向更高性能平台迭代，但在任务复杂度同步上升的背景下，算力 rationing 仍将长期存在。 - 算力战略壁垒：顶级大额融资高度集中于算力采购，促使未掌握海量 GPU 资源的团队必须通过精细化工程创新实现突围。 ### [32:43](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1963s) 世界模型与语言模型：从二元对立到物理 AI 的算力与数据挑战 在讨论模型架构时，Brian McClendon 指出，尽管大语言模型（LLM）正在探索避免调用整个系统的各种路径，但在世界模型领域，前向传播（Feed-forward）和单向计算依然是在端侧与推理、以及模型训练构建中最快且最具成本效益的运作方式。开发者必须学会利用现有的硬件条件工作；事实上，要构建一个完整的全球地理模型，除非大幅提升效率，否则在物理上是完全不可能实现的。对此他笑称，自己愿意“在目前这个痛苦的世界里再多忍受一小段时间，但希望越短越好”。 主持人随后抛出核心疑问：大众经常把 LLM 的概念直接套用到物理世界的 AI 上，这其中最大的认知误区是什么？嘉宾们展开了多维度的拆解与辨析： ```mermaid flowchart TD accTitle: LLM与世界模型的差异与互补架构 subgraph LLM["大语言模型 (LLM)"] L1["训练数据: 抓取全网公开文本"] L2["失败代价: 文本幻觉与心理影响 (软性失效)"] L3["当前瓶颈: 文本枯竭与付费造数"] end subgraph WM["世界模型 / 物理 AI (World Models)"] W1["训练数据: 互联网视频 + 高精度结构化真值标注"] W2["失败代价: 损坏高价值农作物 / 医疗工厂事故 (关键致命失效)"] W3["未来愿景: 作为 LLM 终极学习与推理环境"] end WM -.->|"提供物理模拟与推理沙盒"| LLM ``` Oliver Cameron 率先打破了“非此即彼”的传统观念。他指出，许多人受 Yann LeCun 等学者的讨论影响，认为要么专注语言、要么专注世界模型，但事实是世界模型可以成为大语言模型未来最理想的“终极学习环境（Ultimate learning environment）”——设想让多个 LLM 沉浸在世界模型中进行推理和学习，其性能可能会获得极大的跃升，二者应是互相增益的关系。 Brian McClendon 则聚焦于数据获取的本质差异：LLM 几乎已经“像吸尘器一样吸光了全世界的文本（Hoovered up all the world's text）”，甚至开始付费造数，但这种扩展方式难以持续；世界模型虽然拥有海量的互联网视频资源，但极度缺乏结构化、有组织且标注精确的地面真值（Ground truth）数据，迫使模型构建者必须创新地寻找数据合成与利用方法。 Sce Pike 和 Tim Bucher 进一步强调了物理 AI（Physical AI）面对的严苛现实与巨大商业价值： - **物理失效的不可逆性**：Sce Pike 指出，LLM 的失败主要停留在文本幻觉或潜在的心理健康引导层面，而物理世界模型如果在农田作业中毁坏高价值农作物，或在 TELUS Health 的医疗设施和工业制造车间中发生误判，就会引发灾难性的物理致命事故（Critical failure），这使得物理空间对数据覆盖率和真值飞轮的要求远高于简单的网络视频抓取。 - **万亿美元级的市场蓝海**：Tim Bucher 直言，世界模型是比 LLM 难度高得多、数据需求与算力消耗大得多的领域。但正如 Jensen Huang（黄仁勋）所言，物理 AI 代表着 50 万亿美元的巨大机遇。仅仅是农业每年的总产值就高达 15 万亿美元，建筑业也是 15 万亿美元，任何一个行业都令整个汽车行业相形见绌。面对全球熟练劳动力的短缺，攻克这一难题具有巨大的产业价值。 --- ### [38:24](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=2304s) 闪电提问：未来 3 至 6 个月最令人兴奋的前沿研发 在进入观众提问前，主持人邀请四位嘉宾进行了一轮快问快答，分享各自团队在未来 3 到 6 个月内最激动人心的研发成果： - **Brian McClendon（Niantic Spatial）**：致力于攻克纯视觉定位难题。当用户在无 GPS 信号覆盖的弱网或断网环境（GPS-denied environment）中拍摄一张照片时，系统仅凭视觉模型就能精确推断出拍摄者的实际物理位置，不仅能自主定位，还能将照片反哺回世界模型中。 - **Oliver Cameron（Odyssey）**：团队正在雄心勃勃地训练一个通用的单一大世界模型，“既能自动驾驶、操控机器人、飞行无人机，又能实时生成游戏，甚至教小孩识字背字母”，以全能姿态覆盖多种物理与虚拟场景。 - **Sce Pike（TELUS Digital）**：正与 Lightwheel 合作推进人形机器人数据标准化，规范传感器配置及数据采集流程，以减少无效数据。她提到，当前“单次模型训练运行可能耗资数亿美元”，数据标准化有助于优化算力开销。 - **Tim Bucher（Agtonomy）**：刚发布的多点掉头（Multi-point turns）功能已为农场释放了数百万英亩的可作业面积；目前正在全力攻克车辆带挂车倒车（Operating in reverse with trailers）的自动控制，这项极具挑战的功能将在非道路工业与农业场景中解锁更多落地应用。 --- ### [40:42](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=2442s) 观众问答：家庭机器人的隐私墙与世界模型的数据投毒 进入现场提问环节，与会者就消费级硬件落地与数据安全性提出了尖锐问题： **问题一：家用机器人的隐私信任与算力权衡** 前 iRobot 员工 Chuck Hudson 提问，过去扫地机器人的普及瓶颈往往不是技术本身，而是消费者对隐私和监管审查的顾虑；随着世界模型越来越丰富和持久化，该如何避免重蹈覆辙？ Brian McClendon 回应称，从云端到端侧的算力转移是不可避免的历史演进。在早期的原型测试阶段，由于数据不足，开发者倾向于将数据回传到云端；但长远来看，世界模型与视觉模型必须完全运行在端侧（On-device）。用户绝不希望在家里走动、甚至“对自己产生看法”的机器人把家庭内部画面与数据同步到云端。为了彻底消除公众顾虑，唯有将视觉感知与环境建图严格限制在设备本地，收窄上传到云端的数据漏斗。 **问题二：物理世界模型是否会面临“数据投毒（Data Poisoning）”？** 现场观众询问，LLM 容易因维基百科等源头被篡改而遭受数据投毒并产生幻觉，而基于大量原始传感数据的世界模型是否存在对抗性注入的漏洞？ Sce Pike 给出了肯定的答复，并揭示了数据清洗流程中的巨大痛点： - **AI 生成内容的污染**：TELUS Digital 在处理众包与全球劳动力收集的数据时，必须动用大量算力逐个筛查文件，剔除混入其中的 AI 生成数据。 - **非敌意的隐性破坏**：数据投毒往往不仅来自恶意的对抗攻击，还包括伪造数据导致的场景多样性降低、异常遮挡与几何混淆等问题。 - **计算资源开销**：对全量多模态数据进行防伪与防投毒清洗，已成为当前数据预处理阶段最大的算力负担之一。 在最后环节，另一位观众就几何传感器无法采集到的非结构化物理特性向 Tim Bucher 举手提问，讨论进一步深入到物理建模的边界限制。 ### [44:31](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=2671s) 缺失信息的推理补全与农田环境中的不确定性 在实际部署中，传感器经常会遇到分辨率不足或被遮蔽的情况（例如空中的电线或细小障碍物），系统必须在缺乏直接传感数据的前提下，依靠先验知识与上下文进行推理，填补信息空白。针对如何处理这种“信息缺失与猜测博弈”的问题，Tim Bucher 指出农业作业场景的高重复性为推理提供了天然优势。在农田管理中，如果系统此前已经观测到前 10,000 行作物都在某一特定基准线上终止，模型就应当合理推断后续的新行也会在同样的位置汇聚。尽管团队早期曾采取过某些简化的工程捷径，但通过持续学习这种结构化规律，系统在特征推断与边界补全上的表现正变得越来越可靠。 除了固定的作物与电线，动态物体（尤其是各类野生动物）带来了更复杂的不确定性推理挑战。Tim Bucher 幽默地提到，虽然他们在农田里尚未遇到此前讨论中调侃过的“大象”，但蛇类等爬行动物却是极其常见的干扰因素，系统必须实时预判其运动轨迹与潜在行为。他还打趣道，自己已经向公司的首席技术官下达了指令——“遇到响尾蛇时不用避让，直接轧过去就行”，引得全场会心一笑。 - **规律性先验**：农田中上万行作物的结构一致性，使模型能够利用统计规律填补未知视角的物理边界。 - **动态生物推理**：系统需对难以完整连续追踪的野生动物（如蛇类）进行意图与路径预测。 ```mermaid flowchart LR accTitle: 缺失信息推理与补全流程 subgraph S1["感知输入"] A["稀疏/残缺传感器数据"] --> B["环境结构先验<br/>(例如上万行作物规律)"] end subgraph S2["模型推理"] B --> C["连续性与轨迹推断<br/>(填补电线/预测运动)"] end subgraph S3["决策控制"] C --> D["平滑路径规划与避障"] end ``` --- ### [45:50](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=2750s) Transformer 模型的连续性推断与圆桌总结 针对“电线检测”这一典型难题，圆桌嘉宾进一步阐述了现代深度学习模型的内部推理机制。在很多情况下，车载相机的传感器像素可能只在左侧捕捉到一段线段、在右侧捕捉到另一截像素，或是仅识别出了两侧的电线杆，却由于反光或分辨率限制漏掉了中间部分。现代 Transformer 架构的显著优势之一，在于通过海量数据的预训练，它能够自行总结出物理世界的先验几何规律——即电线在空间中通常是连续的，绝不会凭空断裂并悬浮在半空中。因此，模型会自动在表征层填补中间缺失的连接。虽然这种基于概率的补全机制有时也会引入误报（例如让机器误以为前方有障碍而紧急刹车），但绝大多数情况下它都能做出正确的空间推断。 讨论结束后，主持人 Jared Perlo 宣布圆桌时间已到，对所有观众的专注倾听以及台上各位专家的深度洞见表示感谢。他盛赞这是一场“世界级的世界模型圆桌讨论”，并鼓励大家在后续的峰会日程中继续交流探讨具身智能与空间计算的前沿技术。在一片热烈的掌声中，本场专题讨论圆满落幕。 - **Transformer 的空间先验**：模型基于海量视觉数据推断连续几何线条，自动补全像素断裂处的潜在实体。 - **误报与安全的权衡**：自动填补虽可能引发偶发的过度避障（False Positive），但在保障自动驾驶与农机作业安全方面利大于弊。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=4s) hot takes very much encouraged — Jared Perlo > [01:26](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=86s) Farmville for real — Tim Bucher > [05:11](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=311s) Georeferenced map data and accuracy — Brian McClendon > [05:11](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=311s) Predict what's going to happen next — Oliver Cameron > [10:42](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=642s) data coverage failure — Sce Pike > [10:42](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=642s) nasty secret — Sce Pike > [10:42](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=642s) throwing spaghetti at the wall — Sce Pike > [16:36](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=996s) sorting the wheat from the chaff — Jared Perlo > [28:50](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1730s) self-recovering interrupts — Tim Bucher > [32:43](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1963s) 在目前这个痛苦的世界里再多忍受一小段时间，但希望越短越好 — Brian McClendon > [32:43](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1963s) Ultimate learning environment — Oliver Cameron > [32:43](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=1963s) Hoovered up all the world's text — Brian McClendon > [44:31](https://www.youtube.com/watch?v=FHkrDkn6HS0&t=2671s) 遇到响尾蛇时不用避让，直接轧过去就行 — Tim Bucher ## Mentioned Brian McClendon, Jared Perlo, Oliver Cameron, Tim Bucher, Sce Pike, Chuck Hudson, Maria, Steve Jobs, Bill Gates, Michael Dell, Trip Hawkins, Yann LeCun, Jensen Huang, Niantic Spatial, Niantic Labs, Odyssey, Agtonomy, TELUS Digital, TELUS Health, NBC News, Google, Google Maps, Google Earth, Silicon Graphics, Cruise, Kubota, Bobcat, NeXT, Apple, EA, Spexigon, Lightwheel, iRobot, Pokémon GO, Farmville, 3D Gaussian Splatting, Nvidia Jetson Orin, Nvidia Thor, Starlink, Transformer ## Video description Brian McClendon - CTO, Niantic Spatial Jared Perlo - AI Reporter, NBC News Oliver Cameron - CEO, Odyssey Tim Bucher - CEO, Agtonomy Sce Pike - Vice President, AI Growth & Solutions, TELUS Digital This session highlights how spatial intelligence and world modeling are unlocking the next generation of intelligent systems. Panelists will share applications across robotics, AR, and autonomous vehicles, and explore the models that allow machines to build and update internal maps of their environments. Expect insights on dataset creation, simulation vs. real-world tradeoffs, and the computational limits of embodied AI. [Source](https://www.youtube.com/watch?v=FHkrDkn6HS0)

[0:04] Jared Perlo: hot takes very much encouraged [1:26] Tim Bucher: Farmville for real [5:11] Brian McClendon: Georeferenced map data and accuracy [5:11] Oliver Cameron: Predict what's going to happen next [10:42] Sce Pike: data coverage failure [10:42] Sce Pike: nasty secret [10:42] Sce Pike: throwing spaghetti at the wall [16:36] Jared Perlo: sorting the wheat from the chaff [28:50] Tim Bucher: self-recovering interrupts [32:43] Brian McClendon: 在目前这个痛苦的世界里再多忍受一小段时间，但希望越短越好 [32:43] Oliver Cameron: Ultimate learning environment [32:43] Brian McClendon: Hoovered up all the world's text [44:31] Tim Bucher: 遇到响尾蛇时不用避让，直接轧过去就行

[1:42] Tim Bucher: Farmville for real [5:16] Brian McClendon: real sim [5:16] Oliver Cameron: foundation world model [10:01] Sce Pike: nasty secret [18:07] Oliver Cameron: I don't see anything that represents that quite like the internet. [28:50] Tim Bucher: self-recovering interrupts [42:21] Sce Pike: every single file [44:20] Tim Bucher: Animals is a really big deal. We haven't encountered the elephants yet like I mentioned earlier... but snakes are a really interesting one too and you have to make some inferences about where they're going to go or what they're going to do. But I have instructed my CTO not to avoid rattlesnakes.

[0:04] Now we go. Yeah, there we go. We're live. All  right, we made it to the end of day one. Thank [0:08] you so much to everyone for being here. Uh,  as you heard, my name is Jared Pllo. I am the [0:14] AI reporter. I cover AI for NBC News. Uh, based  in New York, but delighted to be here and very, [0:20] very excited to dive into the world of world  models. Uh, I think we have an amazing panel [0:25] today. I feel like every moderator probably  says that, but I do think it's true for our [0:30] panel here. We have a range of uh different uh  industry applications in a range of different [0:35] positions through the world model stack. I think  it's fair to say. So, I'm really excited to dive [0:40] in. This is part of the technical track, so we're  probably going to get somewhat into the weeds. [0:45] Uh I think there's some other world model panels  and keynotes throughout uh the rest of the week [0:50] that might be a little bit more high level but  I think we're going to try to keep it really [0:54] interesting uh and hopefully provide you with  some interesting insights, stories and you know [0:59] visions from the field about how world models are  actually taking shape. So I think we're going to [1:04] have a hopefully very lively discussion hot takes  very much encouraged. Uh and hopefully at the [1:10] end we'll have around 5 to 10 minutes for Q&A. I  think uh our lovely host Maria will come through [1:16] the audience. So if something uh strikes you  as interesting or you want to follow up, please [1:21] do make a note of that and hopefully we can get  back to that at the end. So without further ado, [1:26] I think we're just going to start with a rapidfire  uh question just down the line. would love all of [1:31] our panelists to just introduce yourselves, say  a little bit about who you are, where you're [1:36] coming from, and then maybe one priority issue,  one task that you're working on right now about [1:42] world models that has recently come across your  desks. Uh, so I'm Brian McClendon. I'm a CTO of [1:49] Nitech Spatial. I've been in 3D and mapping for  40 years. uh did silicon graphics building flight [1:55] simulators back in the very old days uh but spent  10 years leading Google maps and Google Earth [2:01] uh joined Niantic Spac Netic Labs which was  the Pokémon Go company if you remember that and [2:07] uh was uh part of that for several years they spun  out uh they're now owned by a different company [2:13] and we are the uh spatial uh startup that focuses  on building a 4D model of the world and our goal [2:21] is to you know both reconstruct the world and help  uh users and machines localize and understand the [2:28] world around them. Awesome. Uh hello everyone.  My name is Oliver Cameron, co-founder and CEO of [2:34] Odyssey. We are a world model lab and we very  much believe that world models will go beyond [2:40] language models. Uh my background is in driverless  cars. Spent eight years building that technology. [2:45] Uh firstly jumping into the field in 2015 building  my own self-driving startup. Sold that company to [2:51] Cruise. ended up spending two and a half years  there scaling driverless systems and uh we're [2:56] living in a incredibly exciting time where we're  talking about world models. It's pretty fun. Hi [3:03] everybody. I'm S Pike. I'm the VP of AI growth and  solutions at Telus Digital. Telles Digital focuses [3:10] on building the ground truth data for robotics and  autonomous vehicle companies and to train their [3:16] world models. Uh currently my priorities is to  focus on the data standardization because a lot of [3:23] our customers come to us with very very different  sensor configurations with different requests on [3:29] um how to collect the ground truth data and  also how to process that. So we're working [3:36] on a consortium that will hopefully be end to end.  So all the way from the collecting of the data all [3:42] the way to the uh factories, manufacturing plants  and provide a standard for how these things should [3:51] be collected, processed and essentially eval. So  that's my quick background. [snorts] Hi everyone, [4:01] my name is Tim Booker and I am the CEO  and co-founder of Agonomy. We basically [4:07] make tractors autonomous and we don't build the  tractors but we focus on the physical AI layer [4:14] and work with tier one manufacturers um uh like  Kabota, Bobcat, many names that a lot of people [4:22] might know and we now are commercially operating  in the United States and Australia uh with more [4:27] plans to go. My background uh is that um I am a  farmer um and I started a farming operation when I [4:35] was really young but then got into high-tech like  all farmers do um and uh at the same time I also [4:43] started a construction company but while I've had  the agricultural and construction businesses I've [4:49] been in Silicon Valley doing many startups having  worked for some incredible entrepreneurs like [4:54] uh Steve Jobs twice next computer and Apple Um,  Bill Gates, uh, Michael Dell, uh, Trip Hawkins, [5:00] the founder of Electronic Arts. I would say my  forte is gaming systems. So, I think what I'm [5:05] doing is kind of appropriate because it's it's  like Farmville for real. Thank you so much. [5:11] And classic Tim just trotting that that classic  farmer to high-tech pathway. Uh, so [laughter] I'm [5:17] excited to to really dive in now. So, I think  I mean at a high level, world models, I mean, [5:22] it's now the buzziest thing I think out there. I  of course cover AI every day and I think uh world [5:28] models are slowly taking the shape of something  like big data or even generative AI. I mean you [5:33] go back a couple years ago and you know that was  really kind of the buzziest thing and now it's [5:36] everywhere. I think uh world models are certainly  on a trajectory to do that uh as well. But as a [5:42] result there's maybe a million and one different  types of world models that uh form the wider world [5:48] model uh umbrella term. And I think we have very  different approaches to what that actually means [5:52] and how that plays out in your business. uh up  here on stage. So I'm curious maybe if we can [5:57] start with Brian um and at Niantic Spatial just if  you could walk us through a little bit more about [6:02] kind of the bets that your company is taking  to uh spatial and world modeling these days. [6:08] So I think our our sort of biggest difference from  maybe some of the other approaches is uh you know [6:12] we're very focused not surprisingly knowing me on  georreerenced map data and accuracy and you know [6:19] in general that uh you know we've always tried to  build a more accurate uh digital twin of the world [6:25] and that's what Google maps and cool earth kind of  ran down the path of for humans uh but we now have [6:30] uh robotic vehicles we have uh AIs and as well  as humans all need a better model of the world [6:37] And so you know h how can you make it better? Well  a you can make it more accurate, you can make it [6:41] more up-to-date and you can provide semantic  understanding of it so that uh the language [6:47] models of the world can interact with it. And so  our focus has been to uh facilitate more accurate [6:54] models through training a system on understanding  how the world looks like from the ground, [7:00] how the world looks like from the sky and building  a generic model that allows for all data to get [7:05] added to this. you know, long-term 4D model of the  world. And so our general model is multimodal. You [7:12] you can get both position of the camera, you can  have it give more uh other types of output for a [7:17] particular image, picture or map, but the output  is really built to be consumed by robots and AIs. [7:25] And so we've had a lot of focus on uh doing real  sim and generating robotic uh environments that [7:31] are accurate enough to train robots at specific  locations. Gotcha. And hopefully we'll get back [7:37] to the real sim and the sim tore real gap uh  later on there. And I'm curious, you know, from [7:42] your point of view, Oliver, what's the approach  that Odyssey is taking? How is it different than, [7:46] you know, Niantic spatials, for example?  Yes. So, our approach is really influenced [7:51] by driverless cars. And if you think about the  task of driving, you are driving down the road, [7:58] you've got all these crazy pedestrians, all  these cars around you. One of the tasks that [8:02] we do as humans while driving a car is predict  what's going to happen next. And we're very good [8:07] at it. We can predict where that pedestrian is  going to be in the next 5 seconds, where that [8:11] car is going to be in the next 5 seconds.  And that means we can drive very reliably, [8:15] very safely through this crazy world. And in 2015,  when I jumped into the the self-driving field, [8:21] those cars were not capable of doing that. They  could perhaps predict a few milliseconds into the [8:25] future, but the reliability was was not good. But  over the course of about a five, six, seven year [8:31] period, we applied artificial intelligence to the  problem. We scaled these models. We scaled these [8:36] data sets. And would you believe it, suddenly  artificial intelligence could predict the world [8:41] and how it was going to evolve over time with  superhuman accuracy. And this just felt like it [8:46] went somehow under the radar. We we had machines  that could predict the future. And they were very [8:53] reliable to the point where you would put your  child in a car that you would trust is going [8:56] to get that child to the to the destination. But  importantly, that was something that was trained [9:02] on a very narrow domain of data, right? It's just  from the roadway. You couldn't put that predictive [9:08] model into an office into I don't know uh other  applications like defense and healthcare and [9:14] so on and expect it to work. And so our approach  at Odyssey is very much a uh an approach that we [9:21] consider to be a foundation world model in that  we're trying to predict the possible futures of [9:26] the world in every possible task and application  that you could imagine. If you're a robot and [9:32] you need to predict what will happen next, our  model should be able to power that. If you're a [9:38] driverless car and you need to predict the future,  it should be able to handle that. If you're a game [9:42] and you need to predict and generate uh a possible  future, it should be able to handle that. Defense, [9:47] healthcare, science, many different applications.  And so, long story short, we believe world models [9:52] are a foundation model that can understand and  simulate the world. Okay. Thank you. So, I think [9:58] we have two very different flavors of world model  already, but thank you so much Oliver because you [10:02] said the magical dword data. Uh so, I think at the  heart of this, I think really world models in many [10:08] ways is just it's a data play. you know, how you  capture data, the different types of data. Some [10:13] people might be familiar with approaches to world  models where you just, you know, absorb as many [10:18] YouTube videos as possible. In other settings,  you want robots going out and collecting data [10:22] about the world. Maybe people have seen videos or  news articles about people in India or in China in [10:29] factories wearing different cameras, uh, getting  data that way to then feed into world models. But, [10:34] uh, we have a data expert up on stage with us. So,  I'm curious to to come to say, and I promise I'm [10:38] not just going to keep going down the line for  everything, but I'm wondering if you could talk [10:42] to us a little bit about your approach to creating  these data sets for world models. I know you work [10:46] with a variety of different customers. So, if you  could just give us some, you know, insight into [10:50] how that actually looks like dayto-day for you.  Yeah. So, I think what's interesting is that, [10:55] and I love your take on the predictive world  model versus kind of like generative world model, [10:59] but the that you're solving for both, no matter  what, all of that requires data, right? And the [11:04] fundamental of it is like if something fails,  it's not necessarily the failure of the model [11:08] and the way the model has been architected,  but it might be a data coverage failure. [11:12] So, the nasty secret though about this, and  we'll get to the compute part of it, is that [11:18] when you see those videos of people collecting  in factories with something attached to their [11:22] heads or even that New York startup where they're  essentially, you know, cleaning people's houses [11:27] for free so that they can videotape all of that,  they're even if they're working eight hour shifts [11:34] of collection of whatever factories or cleaning  houses or tasks out in the wild only two hours, [11:43] maybe three hours of those eight hours is actually  usable for training purposes. It is a huge amount [11:51] of waste. And so, one of the things that we're  working on is how to make that more productive [11:57] because we're going to need millions of hours.  This is not like a large language model that can [12:01] ingest all the text on an internet. We need that  ground truth data because there's and we're I know [12:07] we're going to get into this about simulations  versus video, but we're gonna need both out [12:13] there. And for us to actually get to a useful  place where we're collecting enough data that [12:20] isn't just wasted, there's got to be a certain  level of quality control, eval, and ability for [12:28] us to actually benchmark properly of what data is  actually useful for these models. And so right now [12:35] a lot of us are just throwing spaghetti at the  wall to see whether hey is it grippers that are [12:41] being teleoperated or is it tactile gloves that  people should be wearing or wrist cameras sens [12:48] or wrist cameras sensors combined with all these  different type of headgear. What's the field of [12:54] view? What's the depth? All of this has not been  standardized. So every time a customer comes to [13:00] us, we start from scratch because everybody has  a different sensor configuration for near field [13:06] what is like I would say like dextrous all the  way to kind of what Niantic is working on which [13:10] is more autonomous vehicles which is a little bit  further out and the sensor configuration there. [13:17] So there has been no standardization of how these  things are collected, how it's processed, how it's [13:23] quality checked and then how it actually goes into  the models to improve the models on ben benchmarks [13:30] versus actually degrading the model. So we are  working on this process right now to create a [13:36] data standardization and also again the endto-end  consortium to kind of create what is actually [13:42] going to be valuable for the models coming down  the road. Great. Thank you. Well, just to finish [13:48] the lineup here, Tim, I'm curious if you can  speak to the role of data. I think uh you know, [13:52] you're working with a very uh downto-earth kind  of very grounded as it is uh application. So, [13:58] what is that data struggle look like for you? How  are you thinking about gathering data and then if [14:02] there is waste or you know uh very fuzzy data that  you can't really use, where does that come up? [14:07] Yeah. No, it's we [clears throat] um we have sort  of a a view of a subset of the world model which [14:14] is unstructured off-road industrial markets um  that you know there's a lot of tools now as as my [14:23] colleagues have been talking about and we're going  to talk more about that I'm sure that really help [14:28] but nothing beats being out there with you know  the in the real world and what's really funny is [14:33] when you think about the evolution of sort of the  data and the flywheel of that data. We've come a [14:40] long way. Like 5 years ago, uh I don't know if you  guys know those little tricycle baby buggy things [14:46] that you put a baby in. Um so not only do people  put cameras all over their heads and stuff, but [14:51] they also put it on baby buggies and drive those  through highv value crops uh to get your initial [14:56] start on okay, what are we looking at and what  kind of models do we need to train on? But here's [15:02] the cool thing. After you get through that initial  phase which now can be accelerated a lot more and [15:07] we'll again talk more about that you get to uh  having actual robots in the field and when you [15:16] reach a critical mass and by the way the critical  mass is not that many you can do it I mean like [15:21] 100 200 robots you're getting now this massive  flywheel of data that's seeing all kinds of things [15:28] one of the biggest challenges that we have in  these off-road worlds let's take agriculture let's [15:33] leave the others off for a minute. In agriculture,  in high value crops, which are your your fruits, [15:38] your your nuts, your grapes, etc. You're actually  training on things that you want to destroy. So, [15:47] you know, it's funny like Oliver, you're talking  about like not hitting pedestrians. In a way, [15:51] we kind of want to hit things. um you know, for  example, big brush and that we want to flail mow [15:57] down, but it it's kind of uh an interesting  challenge because you're not just looking at [16:04] uh you know, training your own models on what  things to avoid, but also what's an organic [16:09] matter that you can push through. Like I can tell  you all about spiderw webs and the challenges we [16:14] had there. Um and we have time. We have time.  [laughter] No, I don't think so. But um it's [16:21] it's a it's just a it's really a fascinating uh  aspect of world models and it's one that's pretty [16:28] fundamental to our global society's existence  in terms of being able to feed the world and [16:32] build the world and move the world etc. Awesome.  Thanks. So maybe let's stay on this data topic [16:38] and you know dive a little bit deeper into the  data flywheels. And I'm curious we're going to [16:42] go out of order now to Oliver. Um thinking about I  I know Odyssey has a different range of products. [16:48] to offer several different uh iterations of  world models. And I think uh when we were, [16:54] you know, chatting in the leadup to this, it  was very interesting to hear how Odyssey is [16:58] thinking about different aspects of world models  and positively reinforcing other aspects of of [17:02] your work as well. Um so I'm curious, you know,  from Odyssey's point of view, how are you viewing [17:07] data flywheels right now and then, you know,  sorting the wheat from the chaff as it were, [17:10] if I can borrow a farming metaphor, uh you know,  to increase the quality of your data? Yes, I think [17:17] um if you want to build a model of the world, then  you need to find a data source that can represent [17:24] the diversity of what you want your model to  represent, right? And if you want a true world [17:30] model, then you need a great diversity of data.  And we very much believe internet video serves as [17:37] that data source. It is vast, describes everything  you could possibly imagine about the world, [17:45] and it's growing incredibly fast, far faster than  text, uh, for example. And so that that's the data [17:52] source. That's the the gift because it's describes  everything growing super fast. Um the challenge [17:59] though is that what is represented inside internet  video is so vast and so [snorts] you want to now [18:07] uh train a world model that's stable that's high  performant high performance um and able to solve [18:15] all of these different tasks and serve all these  applications to model that data set correctly is [18:20] incredibly difficult. But we believe it's the  right research track to go on. For example, [18:26] not to overuse the self-driving analogy, but most  self-driving car models today are very narrowly [18:31] trained still. Now, would your self-driving car  uh benefit from having seen lots of elephants, [18:39] for example? Well, if you were to just collect  in San Francisco, you would almost never see [18:42] an elephant, right? In fact, you that would  be weird. Um but if you build a world model [18:49] uh on the back of everything that exists on  the internet everywhere in your data set is [18:54] copious amounts of elephants. And it may be that  the concept that the model learns by observing [18:59] elephants um is in fact applicable to the task  of driving. Not necessarily because an elephant [19:03] has suddenly appeared but because something it  doesn't understand is currently there. And so [19:07] long story short, we we very much believe that a  true world model will learn from the vastest most [19:13] uh diverse source of data and that just I I don't  see anything that represents that quite like the [19:18] internet. Interesting. So maybe coming to you  Brian uh I don't know how central the internet [19:24] is to your training data, your gathering data. Uh  but I'm curious for Niantic Spatial, it seems like [19:30] uh there's been some exciting deals, some exciting  announcements about uh Niantic partnering with [19:34] drones for example or drone companies to harness  the power of you know gathering aerial data. So [19:39] what is data gathering look like for you these  days? So I I think you know we've had large [19:44] collections of you know ground level video over  the years that we have used sort of our base base [19:51] uh data store but with the drone data that we get  uh from Specsy and also from uh other providers um [19:58] we're able to uh get large geographic coverage um  you know citywide coverage of you know a diverse [20:05] set of cities um you know species in many already  collected in many countries around the world [20:10] um and at drone data is interesting because it's  you know relatively high resolution and you know [20:15] we collect it oblique so we get the angle not just  straight down and so there's a lot of correlation [20:21] between what the drone sees and what you would  see from the ground and but the difference is [20:24] you can basically you know sweep a whole city  and get that information very quickly and I [20:29] think that diversity is very useful I you know one  of the things that came up in listening to these [20:33] answers though is that you know sensor variation  is really a problem um you know if you you know [20:39] as self-driving cars tend to optimize for their  sensor set and you know they they train on their [20:45] sensors on their data because that's what they're  going to get. Um I think nobody on the stage has [20:51] that luxury. we are going to be getting data  from an arbitrary set of you know phone portable [20:57] sensors whatever it could be and we need to be  able to handle it and so pinhole cameras and [21:02] 360 cameras and shots from drone you know highly  zoomed in highly zoomed out um all this is going [21:09] to happen the other challenge we have uh which I  think you heard in the zuk's talk before this one [21:14] is around variation in in weather time place and  you know being able to uh see the same location [21:22] from many uh in many variations and luckily we  have actually have a large collection of data in [21:27] that area but I think this we're just just at the  surface of it we need more always always more data [21:35] say did you want to weigh in there yeah I just  wanted to add to that and I did have a question [21:39] for Oliver as well in your the internet video  because I'm also curious about like the critical [21:44] failure videos like are you able to actually  capture that from the internet but then also to [21:50] um to kind of the drone data there's I think a  dis disparity and we do collect everything what [21:56] I would call like wide area all the way to like  very egocentric view which is like your personal [22:02] view right and world models kind of need to cover  everything from wide area autonomous vehicles to [22:08] humanoid robots that have very tactile um [snorts]  and very fine uh fine dextrous movements and so [22:17] there's a ver variety of data that needs there are  data needs I guess across different kind of world [22:23] models and whatever the world models focusing on  whether that's agriculture or humanoid robots that [22:29] are I don't know chopping onions whatever that is  right so but I'm going back to Oliver I do want to [22:34] ask like do you find that the internet videos  actually capture those critical failure modes [22:41] especially in what I would call fine dextrous uh  settings could you just elaborate on that sure [22:47] yeah I mean I don't think we know yet I think  um really what I hope a world model learns is [22:54] a series of concepts. These are concepts that  describe the world, they describe physics, [23:00] they describe cause and effect, dynamics,  human behaviors, all those sorts of things. [23:06] And so I think if the model has an understanding  of those concepts and then it can apply it to [23:10] novel scenarios um because what it's seen in  internet scale video is so vast, it has truly [23:17] learned concepts that allow it to adapt. I I  I I think that could be true. Um that said, [23:23] we we do augment our data sets with sort of  expert data. Um and that I think will be an [23:29] important source as well. So similar notion having  some level of ground truth data to augment it to [23:37] learn those concepts. Okay. Thank you. Yeah.  Awesome. So maybe um you know if we can jump a [23:43] little bit I'm curious you know to go to Tim now.  We've been talking a lot about the worlds of data [23:47] but uh you know data feeds simulations simulations  you know feed the real world. So I'm curious the [23:52] sim toreal gap seems like it's been you know a  a problem through robotics for decades. You know [23:57] world models go back decades. I just also want  to make that clear. Academics have been talking [24:01] about world models for many many years, not just  since the rise of gender of AI. But I'm curious, [24:06] Tim, if you could talk about any specific examples  or real world moments uh where you have struggled [24:13] or you've seen uh the world models that you're  developing not fully be able to make that jump [24:18] from simulations to realities and how autonomy is  working on those. Yeah, we see these gaps every [24:26] day and simulations get better and better and  better, but then we find more and more scenarios [24:33] that we never thought about. Um, I'll tell you one  this week. It actually happened on yesterday. Um, [24:41] where first of all, we we ingest a lot of data,  aerial data, and then we do object recognition to [24:46] know like where the rows are and everything. And  that kind of sets up um not the not the real map, [24:52] but it just sets up kind of where the vehicle  after it's localized uh needs to kind of get [24:59] towards and then it sees the structure of the crop  itself and starts to then exploit that structure [25:04] to navigate through it and do work around it. Um,  and we have a situation where um, wires are used [25:13] for trellis systems and they're very thin. And  we're in an orchard that had a bunch of trees [25:19] missing on the end, but the wires were still  there. Um, so not only was it a simulation gap, [25:25] but it was also something that we talked about  sensors, the sensors couldn't pick up either. So, [25:31] uh, yes, we ran into the wires. Fortunately, there  was, you know, no one was hurt or no damage. Um, [25:38] but we find these situations so often that it it  it both frustrates me, but it also like gets me [25:46] really excited about keeping that flywheel  going and getting more units out there and [25:50] seeing more of these kind of scenarios because I  fundamentally believe we'll never have, you know, [25:55] every scenario covered in simulation. Um, but we  can seed simulation with a lot of new things that [26:02] we do encounter, you know, whether it's elephants  in the middle of vineyards or whatever whatever [26:06] it might be. But, um, yeah, we see we see some  pretty big gaps in the unstructured off-road [26:12] world. Okay. Uh, open question for the panel here,  whoever wants to jump in. But, you've all been in [26:16] the industry for several years. I'm curious how  you've seen that simtoreal uh, discourse change [26:22] over time. I think you know now we're all there's  a ballroom full of people here talking about world [26:26] models but sim to real has always been an issue.  So what's changed and maybe what directions are [26:32] you most bullish about for further closing that  gap? Uh I'll I'll do quickly two things. Um, [26:39] one I think that uh in the early days of you know  self-driving cars going back to the example it was [26:45] you know they had multiple sensors on them but  typically the perception systems focused on a [26:49] single sensor because sensor fusion was extremely  hard both calibration and just training models and [26:56] I think we've seen improvements to the point where  we we can talk about sensor fusion as a value ad [27:01] and you know improving the accuracy and especially  as we have these multimodal models but I think the [27:06] other the other thing is you know, we have uh had  synthetic scene generation for decades and that [27:14] you know that is fast, it's cheap and it generates  a model that's really good at understanding [27:19] synthetic visuals. And so one of the things that  I think is happening now with real to sim is the [27:24] the ability to create an accurate rep you know a  visually accurate representation that is not you [27:30] know textured polygons but is in fact you know  these 3D gaussian splats and they are a new way [27:36] that produces better visuals that is closer to  matching what the robots actually see in the real [27:42] world and we've seen uh with partners like flexion  that they have improved the accuracy of of the [27:48] robot's ability to understand the world because  they can now train on the visual sensors input [27:54] from these Gaussian splats. Um my perspective is  that this gap is solved if you were willing to [28:02] throw as large a model as you possibly could at  the problem. Right? So for example, if you were [28:06] to throw a very very large model uh 50-ish billion  parameters let's say and you were willing to run [28:13] that on a robot with a little data center next  to it, you'd probably find that there is no gap. [28:19] Um but that is not the reality and if you want  uh real time and you want good performance on [28:25] the hardware that runs on the robot that's where  you find your battling the parameter count of the [28:29] model and thus see this gap uh in different ways  but I think the gap has become much much smaller [28:35] over the last year or so. Great to hear and I  should also say 3D Gaussian splats has become one [28:41] of my favorite terms. [laughter] I spend most of  my time wrestling with large language models. So, [28:46] that's been a fun one to learn is I've spent  more time in this realm. Well, I think it's [28:50] a perfect segue to talking about one of the other  bottlenecks when we're talking about world models, [28:53] which is compute. Uh, and again, we're working  at a range of different applications here. Some, [28:59] you know, deployed in the field, others, you know,  uh, developing the world models. But I'm curious [29:04] uh maybe if we can start with you Tim uh what  positives, what negatives, what bottlenecks are [29:11] you seeing when it comes to compute and how  is rationing compute or the ability to get [29:15] enough compute on the edge affecting your own  approach to world models? Yeah, so um in a lot [29:21] of [snorts] vehicle autonomous vehicles um things  have to be done at the edge, right? You don't have [29:27] a connected data center although that would be a  lot of fun um and very useful. So um in certain [29:34] applications as we're you know navigating through  and and and uh uh doing work um the other aspect [29:42] that I forgot to mention about what uh we are  challenged by and here when you talk about robots [29:47] everybody thinks about a single robot uh whether  it's a humanoid or some other kind of um uh thing [29:53] that does not change. Um, with tractors, there  are thousands of implements that get attached to [30:00] it. And so, you also have to train on all of those  things, and you have to train on what kind of, you [30:05] know, capabilities those have. But there's one big  advantage of some of these off-road um, you know, [30:12] industrial markets is that it can pause. And so,  we run out of bandwidth compute um, capabilities [30:20] on vehicle. And you also have to have all the  functional safety there. And you can't count on [30:24] GPS cuz sometimes you're under canopy. You can't  count on internet connectivity cuz even Starlink [30:29] and cellular, you know, don't work. Um, but if you  pause, you can actually kind of cheat the system a [30:35] little bit. And so we have these things called  self-reovering interrupts where we keep stats [30:42] of all of it, but they're usually just, you know,  [snorts] um, milliseconds to a half a second. And [30:49] now if it's doing that all the time, you're not  getting the job done efficiently and you're not, [30:53] you know, doing what the value that it's supposed  to bring. Unfortunately, we're not there yet. Um, [30:59] but what's what's great is that, you know, we  just like we're seeing real to sim or sim tore gap [31:05] closing, we're seeing a lot more compute available  on the edge. So you go from the orange to the [31:09] Thors to the, you know, whatever is next and  that's definitely helping a lot, but then you're [31:15] doing more complex jobs and so you just keep,  you know, moving the goal, if you will. But the [31:20] good news is there are ways to cheat the system  a little bit when you do run out of compute. All [31:24] right, you heard it from Tim. Cheating is good.  [laughter] Uh, [snorts] anyone else on the panel? [31:29] How is compute uh current compute environment  affecting your approaches to world models? [31:35] I I mean it is if you'd have asked  me this question six months ago, [31:38] I'd have probably had a very different answer.  Today it's a real strategic advantage to have [31:42] access to compute and having growing clusters.  Um I think it's undervalued just how much those [31:51] rounds that Anthropic and OpenAI did where they  raised 100 $120 billion 50 60% of that probably [31:58] went straight to compute supply right now and  in the next year or two. And so getting your [32:03] hands on tens of thousands of GPUs has become  extraordinarily difficult uh to do. So if you [32:09] can and you do have great partners to do it, then  you're going to ser uh going to have an advantage [32:15] as as a result. Just just a note, Telus does have  an AI factory. We have a sovereign AI factory [32:21] up in Canada. So if you need more compute,  just you know, come talk to me. [laughter] [32:27] So I'll go on the other side of this and say  that you know when you don't have the hundreds [32:31] of millions or billions of dollars on the hardware  side you know that uh that strict rationing causes [32:38] you to learn to do things and we've you know  seen some really good improvements obviously [32:43] in language models with ways to not use the entire  system um but with uh some of the models that I I [32:50] think I work on and maybe we work on is you know  feed forward and you know one one directional is [32:55] a far faster, cheaper way to operate both you know  on device and inference and also in training and [33:01] building that model. And so we you learn how to  work with the machines that you have and uh you [33:07] know the fact of the matter is to build a a you  know full geographic model of the world would be [33:12] physically impossible unless you had some way of  being much much more efficient. So I think I I'm [33:18] okay living living in this painful world for  a short more a short time longer. Okay. Very [33:23] short hopefully. Uh I'm curious, you know, a  lot of the discourse here is about um you know, [33:28] large language models. We're talking about very  different applications here. So from your vantage [33:33] point, what's one of the most misunderstood  aspects when people come to world models? They [33:37] think about you know AI in the physical world.  What's one of the most misunderstood aspects [33:41] potentially that people are porting over from  generative AI when it comes to LLMs? I can go [33:48] first. Um I think what maybe one is that world  models and language models may work together. [33:54] Um and that world models may serve as really  great learning environments for language [34:00] models in future. Um in fact maybe the ultimate  learning environment. You can imagine a world [34:06] model that allows multiple language models  to exist inside of it and to learn within [34:10] it to reason within it. Maybe that boosts the  performance of language models significantly. [34:15] Um and I I think often it has become  maybe this is thanks to Yan Lun but [34:20] very much a either or right like it's it's  either you work on language or you work on [34:25] world models but maybe there's a world  in which they they improve each other. [34:32] I mean I think one of the things that's very  different is that you know obviously with language [34:35] models you know you basically they hoovered up  all the world's text like literally all of it [34:40] they ran out of it and they started paying for  more to be created. Um but you know that doesn't [34:44] scale either. Um the with world models we have  both a you know huge amount of internet video [34:52] to take but we also have a very little amount  of structured organized accurately labeled data [34:58] ourselves and so creating that data and there are  several companies whose sole focus is generating [35:05] reasonable ground truth data at scale. I think  that's interesting, but it means that, you know, [35:10] for for those of us working on the models, you  have to figure out ways to work with the data [35:14] you have and find out ways to creatively create or  synthesize more data from it. Just to add to that, [35:22] yes, and I think one of the things that is also  very different is that large language models, [35:27] yes, you can definitely get into some critical  issues, right? Um, we've all seen kind of the [35:33] mental health problems that are potentially not  instigated but uh pushed more forward with large [35:40] language models. When you're talking about world  models in the physical AI space, it is critical [35:46] some of the safety guidelines, right? This is  why I asked about the critical failure modes [35:51] and making sure that there's ground truth around  that is because if you mess up in a physical space [35:58] and you know whether it's destroying crops very  valuable crops or in what we have teles health and [36:05] so we're also training in our health facilities  as well right and we also have factories and such. [36:12] So when you mess up in these areas in physical  environments then you have critical failure. It [36:20] is not just textual failure, right? And so this  is where I think in some ways it's more important [36:27] that there's a large variety of data coverage and  it's kind of that flywheel that you were talking [36:33] about Tim um more so than in just ingesting sorry  just internet videos right because there is so [36:41] much more at risk u when you're deploying into  the real world so that's where I think is the [36:47] biggest difference in what we've seen before to  where we need to go in the physical AI space. And [36:55] I'll just um add a little bit more, but I think  uh my colleagues covered it really well. And I'm [37:00] going to summarize it by saying, you know, the  the world model and [clears throat] the things [37:05] that it can benefit. It's just a much harder  space requiring so much more data, so much more [37:12] compute than LLMs. And I want to flip it to what  the benefit is of it though. If you just take, [37:21] you know, the physical AI opportunity, whether  you're Jensen Wang and says he says it's a 50 [37:29] trillion dollar opportunity, um, you know, all the  humanoids and the robots and the tractors. Um but [37:36] you know even even those industries that require  this uh kind of uh significantly harder problem to [37:44] solve and you know more data that is needed. They  too have significant uh TAMs the total available [37:53] market of for example agriculture output each  year is $15 trillion which dwarfs the entire [37:59] automotive industry construction $15 trillion. So  those two alone, you know, are worth going after, [38:06] especially if there's impact with things like  skilled labor. So these are things that are hard, [38:12] but they have to be solved and that's what, you  know, we're all trying to do and I think everyone [38:18] here at this conference and I'm very encouraged  by that. Wonderful. Uh I promised I would give [38:24] some time for questions. We've given you almost  40 minutes to brainstorm questions. So hopefully [38:28] you have some out there. Maybe while we just give  everybody a chance to get their thoughts in order, [38:33] just quick very rapid fire down the panel here.  Uh what's one thing that your company's working on [38:39] that you're just very excited about in we'll call  the next 3 6 months? uh Niantic Spatial's I think [38:47] uh making some you know very interesting strides  in this localization problem of saying you know [38:52] I'm standing somewhere I take a picture with my  camera with a phone or whatever exactly where am [38:58] I standing and this matters because that photo can  then become part of this model of the world but it [39:02] also matters if you're in a GPS denied environment  and you want to know where you are in the world [39:08] and so I think we are seeing that there's now the  ability to actually find yourself without GPS at [39:14] all and you know strictly from a visual model and  I think that's going to be very interesting. That [39:19] is cool. Um what's exciting for us is to train a  single world model that can drive a car, control [39:27] a robot, fly a drone, generate a game, teach  your kids the ABCs. That would be cool. Easy. [39:35] Easy. What about you? So again, kind of the data  standardization especially for um humanoid robots. [39:43] So working with a company called light wheel to  standardize how the data is collected to reduce [39:48] the waste um and optimize and so so that hopefully  we don't need as much compute uh to take that data [39:57] train the model takes hundreds of millions of  dollars to train to do a single training run so [40:02] hopefully we can actually optimize that through  some level of data standardization around all the [40:07] sensor configuration etc. And quickly to you, Tim.  Uh, little bit of a modest thing here, but um, we [40:14] just released multi-point turns, which opened up  like millions of acres. Um, but I'm really excited [40:19] about operating in reverse, which is, you know,  you might not realize all the impact that has, but [40:26] in these off-road industrial markets, it opens up  so many other opportunities. So, uh, driving cars [40:32] with trailers or driving vehicles with trailers in  reverse is, uh, is going to be a fun thing we're [40:37] we're working on. Awesome. All right, that's a  great teaser. Uh, does anybody have questions? [40:42] I think we have time. We have one down here in the  front. Yeah. Hi there, Chuck Hudson. Uh, so Brian, [40:49] you and I talked about this years ago with um  iRoot, in fact, and that was along my lines. So, [40:56] I spent several years at iRoot where we saw  firsthand that the technology wasn't always [41:00] the limiting factor, right? [snorts] uh consumer  trust and privacy perception often were. The mo [41:08] more intelligence a robot had about the home, the  more value it could deliver, but also the greater [41:12] the scrutiny from regulators and consumers. As  world models become richer and more persistent, [41:18] how do we avoid hitting that same adoption wall?  Is it is the answer stronger privacy guarantees, [41:24] ondevice intelligence, better governance, or  simply creating enough value that users are [41:29] willing to make this trade-off? I think there's  always been this transition between uh cloud and [41:35] ondevice, right? I mean, in the short term, you  know, when you don't have enough data and it's all [41:40] sort of PC kind of uh testing, you know, you let  that you send these things to cloud and you hope [41:45] that you have the data input from users that are  willing to do that. But long term, this has to be [41:51] on device. This model has to fit on the machine.  the photos, the data, the user information cannot [41:57] leave that machine because I think we are going to  struggle with having these robots rolling around [42:01] our home and uh telling us stuff and you having  opinions about us. Um and we don't want that [42:08] shared with the cloud and I so the only way to you  know to narrow the funnel down to the information [42:14] that you're willing to upload is to say all of  the visual model uh lives on device. Another [42:21] question down here in the front. So, our LLMs are  susceptible to what we call data poisoning. You [42:27] can edit some Wikipedia entries and then make them  spit out some interesting hallucinations. When [42:32] building world models, are there any points for  adversarial insertion or is it more robust because [42:39] so much of the raw data is so raw? Great question.  We had a a whole series of safety questions. So, [42:45] curious to hear what you think. Sorry. Was that  for anyone? Yeah. So I I think in that sense there [42:54] is an opportunity for data poisoning. I don't  think that that is not uh and we actually do see [43:01] this right now where um we try to weed out using  AI and sorry I can't see this questioner anymore [43:08] but we try to weed out any sort of AI generated uh  collection as well. So on the from our side when [43:17] we are doing this and this is a part of our whole  issue around the compute is that we are always [43:24] looking at every frigin [snorts] file to make sure  that it is not some sort of randomly AI generated [43:32] file that is trying to poison that data. And so  it's not even just poison the data from like a [43:38] actual antagonistic uh reason. It's could be just  for like to try to create less diversity or um [43:49] occlusion, confusion, etc. So, it's it's all over  the map on what we've seen as this like bad data [43:57] that we find from our gathering through the crowd  and through kind of global uh workforce. And I [44:06] think we have time for one last quick question if  anybody has one. over here. Thank you so much. Pal [44:13] walking over here. Yeah. Where is that? Where's  that hand? Oh, that one. Right. Okay. All right. [44:20] Thank you so much. Um I had a question kind of  regarding the case that Tim mentioned. So there's [44:26] certain things that are just not capable of being  captured by geometric sensor data. Like so for [44:31] example like wires. So there's kind of cases where  you kind of have to infer that like even though [44:36] your sensor information is not there, you have to  assume because you see certain things, you have to [44:42] kind of fill that unknown gap. Have you guys kind  of ran into these type of situations like how do [44:47] you handle this type of like lack of information  and like the guessing game in a will? Yeah. And [44:53] the repetitive nature of some of the things  we work in helps with that. So the scenario I [44:58] described, we have to do a better job of inferring  on that case and then we will get it because [45:03] you'll know that oh if 10,000 other rows come  to this point here, you should assume the others [45:09] should come to this point as well. And so um you  know we took a shortcut and [clears throat] we [45:14] but that's that's part of improving all the time.  But that's absolutely the case. Um there's uh yeah [45:21] there's a lot of um uh different uh objects that  we encounter like animals is a really big deal. [45:28] Uh we haven't encountered the elephants yet like  I mentioned earlier. Um but uh snakes are a really [45:34] um interesting one too and you have to make some  inferences about where they're going to go or what [45:39] they're going to do. [clears throat] Um but I have  instructed my CTO um not to avoid rattlesnakes. [45:46] So we we're okay with that. Surely they thank  you for that. Does anyone else have uh any any [45:51] quick thoughts on that last point? I mean just on  the wires thing in particular, I think it's very [45:54] interesting that you you can infer that a wire  should be there bas you see the telephone pole or [45:59] you see some other indicator or your sensor camera  catches some pixels over here and some pixels over [46:05] here but misses the step in between. You know, one  of the advantages of any kind of transformer model [46:11] is that with enough data, you deduce that those  lines are usually continuous and they do not have [46:16] floating bits in the middle. And so the model will  fill it in for you, which could cause, you know, [46:23] misinformation and you failing, you know, stopping  because you think there's a wire there. But most [46:27] of the time it'll do the right thing. Wonderful.  Well, with that, uh, we are at time. So, I want to [46:33] thank everybody for your time, for your attention,  for this worldclass world model panel. Uh, [46:38] I think we're all excited to keep the conversation  going. So, thank you so much and hope to see you [46:42] around the rest of the conference. Give him a  huge round of applause to our wonderful panel.