Niantic, Netradyne Ditch Web Video to Anchor World Models in Physics

Niantic与Netradyne放弃网络视频，转向以物理定律锚定世界模型 脱离物理实际的视频数据会引发复合误差并导致自主智能体失效，使确定性物理定律与专有边缘案例数据成为企业级仿真唯一可行的基石。 专家小组将世界模型定义为以物理规律为基础、受动作条件控制的系统，能够跨越从半导体热动力学到城市级空间映射的不同尺度来预测物理后果。嘉宾们否定了仅凭暴力计算与未经筛选的互联网视频就能产生真正物理智能的观点，强调必须依靠高保真空间数据和罕见的实际运行边缘案例，以克服不断累积的“仿真到现实”（sim-to-real）差距。他们分享了具体的企业效益回报，例如将炼油厂维护导航时间从两小时缩短至两分钟，以及将芯片架构设计周期大幅压缩至数天。最后，专家们概述了企业落地的实用步骤，敦促各类机构构建模块化物理技术栈，打造专有数据护城河，并为围绕模拟器认证即将出现的监管挑战做好准备。 世界模型通过物理因果律、确定性规律与三维空间几何超越了纯文本生成，但由于缺乏罕见边缘案例与热场、电磁等不可见物理属性，仅靠互联网通用视频的暴力训练难以满足工业需求。 (3:22) Niantic Spatial 的视觉定位系统（VPS）利用三维世界模型解决了城市峡谷中的 GPS 漂移难题，为配送机器人与现场作业人员提供厘米级的高精度定位。 (13:02) 虚拟到现实的差距（sim-to-real gap）仍是核心技术瓶颈，长尾边缘场景中的微小预测误差会随时间推移迅速累积，最终导致自主智能体完全偏离安全轨迹。 (18:29) 纯合成数据训练在关键物理领域存在产生危险幻觉的风险，企业在部署时必须遵循守恒定律等第一性原理进行验证，并将大模型大幅压缩蒸馏以实现边缘端实时推理。 (21:28) 工业应用已展现出显著的商业回报：炼油厂设备定位时间从数小时骤降至 2 分钟，半导体物理设计迭代周期缩短至数天，商用车队碰撞事故率减少了 50% 至 60%。 (30:57)

Industry panelists argue that building viable enterprise world models requires embedding deterministic physical laws, 3D spatial geometry, and rare edge-case data rather than merely scaling compute over raw internet video. While these systems demonstrate clear industrial returns in fleet safety, semiconductor engineering, and spatial localization, compounding errors in the sim-to-real gap remain a primary technical bottleneck. Grounding AI in Physics and Action Conditioning (3:22) Panelists reject the premise that world models can be constructed simply by scaling compute over vast collections of raw internet video. Sreekanth Annapureddy of Netradyne argues that routine driving video provides low "intelligence per sample," noting that meaningful models require rare edge cases coupled with action conditioning—understanding the precise consequence that follows a specific action. Hardik Kabaria of Vinci explains that visual cameras alone miss invisible phenomena governed by physical laws such as Maxwell's equations; an operational world model must enforce deterministic physical constraints, such as conservation of energy, across scales ranging from nanometer semiconductors to cities. Inhi Cho Suh of Niantic Spatial cautions against uncurated "video slop," noting that models trained purely on synthetic simulations frequently cause robots to fail in physical settings, such as crashing into transparent glass. To avoid operational hallucinations, Niantic Spatial grounds its models in real-world 3D geometry and computer vision across real-world infrastructure. Navigating Sim-to-Real Divergence and Edge Constraints (18:29) Annapureddy identifies the "sim-to-real gap" as the primary bottleneck in world model deployment. Because errors compound over extended time horizons, an autonomous agent making a minor sub-optimal decision in an edge scenario can rapidly diverge into total system desynchronization. Early driving models, for instance, hallucinated open highways when approaching roadside houses or continued driving unimpeded after major collisions because standard training datasets lacked crash and near-miss physics. To counter these failures, Kabaria stresses validating model inferences against first principles, enforcing strict energy balance before accepting simulation outputs. Suh emphasizes the danger of relying exclusively on synthetic approximations where human lives are at stake in industrial facilities, advocating instead for tight acceleration loops that reinforce synthetic data with physical ground truth. Annapureddy also highlights the challenge of model distillation, explaining that massive foundation models must be compressed to execute real-time inference on low-latency vehicle edge hardware. Quantifying Operational Returns Across Heavy Industries (30:57) Deployments of spatial and physics-informed models demonstrate quantifiable advantages over legacy engineering methods. In heavy industry, Suh reports that oil refinery turnarounds—where facility downtime can cost roughly $10 million per day—traditionally saw new workers spend up to two hours locating pipe "blinds" using 2D blueprints. Using mobile 3D spatial reconstructions reduced search times to two minutes flat, contributing to an overall 80% turnaround time reduction. For robotics, Niantic's Visual Positioning System resolves urban GPS drift—which can reach 10 meters or more—by providing centimeter-level visual localization against 3D world models. In hardware engineering, Kabaria notes that Vinci's physics-informed models evaluate multi-gigabyte spatial geometries across thermal and mechanical disciplines simultaneously, compressing semiconductor design iteration cycles from months to days. In fleet safety, Annapureddy reports that Netradyne’s real-time edge monitoring and spatial reconstructions have cut fleet collisions by 50% to 60% and reduced distracted driving behaviors, such as mobile phone usage, by 90%. Enterprise Playbooks: Data Moats and Reverse Engineering (39:26) As underlying model architectures become commoditized, Annapureddy and Kabaria advise enterprises to view domain-specific physical data and quantifiable evaluation benchmarks as their primary competitive moat. Kabaria recommends working backwards by identifying the precise physical action or engineering threshold—such as robotic manipulation or thermal dissipation limits—before selecting model architectures or assembling modular physics stacks. To lower barriers to adoption, Suh invites enterprises with complex physical data to explore spatial modeling, noting that visual models trained on aerial feeds have exhibited zero-shot transfer to underwater asset monitoring. She also points out that engineering teams are containerizing world models so organizations can safely simulate proprietary physical operations within secure, on-premises environments. As autonomous robots and vehicles edge closer to sharing environments with the public, the panelists point to an unresolved regulatory dilemma: determining who will independently certify the proprietary simulation environments companies use to validate their own systems. Over the next decade, the speakers anticipate world models will drive widespread structural transformation across semiconductor engineering, municipal planning, and physical autonomous systems.

Ai4行业研讨会的核心观点表明，世界模型正在超越传统语言模型与纯视觉视频生成，通过深度融合基础物理法则、三维空间感知与长尾动作条件，为硬件研发、工业运营与自主系统提供可信决策底座；但其实现工业级落地的关键前提，在于克服长尾场景中的“虚实鸿沟”（Sim-to-Real Gap）并建立严格的物理验证体系。 超越视频堆叠：物理法则与三维空间锚定 (3:22) 与会专家一致否定了仅靠海量互联网视频与算力暴力堆砌就能构建世界模型的假设。Vinci的Hardik Kabaria指出，真实世界由不可违背的物理定律支配，真正的世界模型必须在从纳米级半导体到城市级工程的跨尺度场景中严守能量守恒等物理公理，确保同一物理输入在任何地点均得出确定性结果，而非产生随机幻觉；此外，视觉感知无法直接捕捉麦克斯韦方程组所描述的电磁干扰、散热与力矩等隐性物理现象。 Niantic Spatial的Inhi Cho Suh则批评了未经筛选的互联网视频对工业仿真的潜在危险，主张模型必须扎根于高精度的真实三维物理空间。仅依托纯合成数据训练的机器人容易在真实世界碰壁（例如因无法将视觉转化为精确网格而撞向玻璃）；相比之下，利用真实工业点云、历史管线测绘以及如加州兰乔科尔多瓦全城高精3D扫描等现实数据，世界模型能够克服高楼遮挡导致的传统GPS漂移，实现厘米级精准视觉定位与空间资产重构。 虚实鸿沟与长尾故障：模型核心训练瓶颈 (18:29) 专家指出当前世界模型研发仍处于起步阶段，最关键的瓶颈在于“仿真与现实差距”（Sim-to-Real Gap）。Netradyne的Sreekanth Annapureddy解释道，常规行驶数据缺乏高价值信息增益，但在长尾与极端边缘场景中，模拟与现实的误差会随时间推移呈指数级累积发散，引发机器人控制理论中的彻底脱轨。早期训练中由于常规视频缺乏事故数据，模拟车辆在发生严重车祸后甚至会若无其事地继续行驶，或在驶向路边房屋时由于缺乏几何认知而幻觉出开阔的高速公路。 为消除幻觉，Netradyne通过专门注入事故与险兆数据，使模型理解具体驾驶操作所伴随的物理后果。Hardik Kabaria进一步说明，当把热力学、电磁学与机械应力等多物理场统一输入单一模型而非割裂训练时，模型反而展现出更快的学习速率与更强的泛化能力。Inhi Cho Suh同时提醒，由于重工业运营关乎人身与资产安全，不能依赖单一合成数据，必须构建合成数据与现实物理真值持续校验的加速飞轮，同时解决将大型模型高度蒸馏压缩至车载边缘硬件进行低延迟推理的工程难题。 交互范式演进与重工业量化投资回报 (30:57) 与文本对话类大语言模型不同，世界模型正深度嵌入专业工业工作流中。在车队安全运营上，Netradyne采用生成式空间提示推演反事实场景：当驾驶员闯红灯时，系统能推演生成可能发生的碰撞后果以辅助安全辅导，并为自动驾驶代理提供路测前的模拟沙盒，实测使车队交通事故减少了50%至60%，手机分心驾驶行为下降了90%。 在硬件工程与能源重工领域，世界模型展现出清晰的商业回报。Vinci将物理基础模型直接嵌入计算机辅助工程（CAE）流程，工程师输入空间几何与温度边界条件即可直接预测芯片热负荷与结构形变，将原本耗时数月、依赖超级计算机或实体物理打样的半导体设计周期压缩至数天之内。面对每日停工维护成本高达近1000万美元的炼油厂，Niantic Spatial利用3D重建与移动端空间定位技术引导工人快速查找检修盲板，将新员工原先长达2小时、资深员工约10分钟的设备排查时间统一缩短至2分钟，整体检修用时缩减约80%。 企业落地路径：逆向构建与模块化物理栈 (39:26) 针对企业切入世界模型的实施路径，专家建议避免直接套用通用大模型。Sreekanth Annapureddy提出，底层模型架构正在走向商品化，企业真正的核心壁垒在于自身专有的物理环境数据储备与清洗能力，并在规模化部署前建立可量化的物理保真度评估基准。 Hardik Kabaria进一步建议采取逆向工程策略：企业应首先明确最终的物理行动目标与成功标准（例如机械部件制造公差或机器人动作），据此倒推所需的力学、热学或电学等特定物理模态，进而拼装专用的模块化物理软件栈，而非寻求单一通用的整体模型。Inhi Cho Suh补充表示，研究表明轻量级空间嵌入（Spatial Embeddings）能大幅降低运算开销，且航拍模型在无需微调的情况下可泛化至水下检测场景；未来企业落地将倾向于利用容器化技术在受保护的本地环境中安全运行模拟，确保敏感工业数据不出园区。 尽管与会嘉宾一致预测世界模型将在未来十年深刻重构硬件设计、城市基础设施与具身自治系统，但行业仍面临严峻的治理与验证挑战：当各企业纷纷依赖自主构建的闭门仿真器来验证并认证自家机器人的物理安全性时，“谁来认证模拟器本身”构成了当前尚未解决的核心监管真空。

## Background 本内容由 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台上。视频记录了一场由 Techmeme 和 EPro Strategies 的 Emil Protalinski 主持的行业圆桌论坛，邀请了 Netradyne 首席技术官 Sreekanth Annapureddy、Niantic Spatial 首席执行官 Inhi Cho Suh 以及 Vinci 联合创始人兼首席执行官 Hardik Kabaria，共同探讨世界模型（World Models）如何通过理解物理因果关系来重塑自动驾驶测试、工业模拟与高价值硬件设计等商业场景。 ## TL;DR 三位行业专家深入剖析了世界模型的本质与落地路径，打破了“海量视频加算力堆砌即可炼出世界模型”的认知误区，强调物理守恒定律、动作因果映射与高保真 3D 空间数据才是其核心支柱。讨论涵盖了多源异构真实数据与合成数据的飞轮构建、克服模拟与现实差距（Sim-to-Real Gap）的挑战，以及在边缘端实现低延迟推理的工程实践。嘉宾们还结合大型炼油厂巡检、配送机器人视觉定位与微观半导体热力学仿真等具体案例，展示了世界模型带来的百倍 ROI 跃升，并预测其将在未来十年彻底重塑芯片研发、自动驾驶和城市基础设施规划。 ## Atmosphere & style 整场研讨会专业务实且极具洞见，主持人以幽默轻松的开场迅速打破“AI 流行词”的浮躁氛围，引导讨论切入硬核技术底层。三位嘉宾各自分享了在自动驾驶长尾场景、微观多物理场仿真以及复杂工业现场探索中的“踩坑”教训与突破，对话节奏紧凑、观点碰撞强烈，让听众深刻感受到世界模型从前沿概念迈向真实物理工业落地的巨大势能与技术严谨感。 ## Key takeaways - [03:22](https://www.youtube.com/watch?v=VupX7v6mocM&t=202s) 世界模型绝非纯文本生成或海量被动视频堆砌，其核心在于理解现实物理世界的空间因果推演、物理定律约束与动作条件化（Action Conditioning）。 - [08:35](https://www.youtube.com/watch?v=VupX7v6mocM&t=515s) 纯视觉传感器存在固有的物理盲区，电磁场分布、热力学、能量守恒等物理要素必须作为底层编译层内嵌到物理基础模型中。 - [11:06](https://www.youtube.com/watch?v=VupX7v6mocM&t=666s) 企业级空间世界模型能够利用低质历史影像或航拍/全景相机，在无 GPS 或弱纹理的复杂工业环境中实现厘米级高保真 3D 场景重建与导航。 - [13:02](https://www.youtube.com/watch?v=VupX7v6mocM&t=782s) 世界模型的交互模式已超越文本提示词框，工业端可通过输入 3D 几何与环境物理边界条件直接推演最高温度、形变等物理场测量结果。 - [16:01](https://www.youtube.com/watch?v=VupX7v6mocM&t=961s) Niantic Spatial 的视觉定位系统（VPS）API 解决了机器人在城市高楼峡谷中的“迷失机器人问题”，并推动模型迈向原生理解几何与深度的物理语义理解。 - [18:53](https://www.youtube.com/watch?v=VupX7v6mocM&t=1133s) 当前世界模型面临的关键技术瓶颈是虚实迁移差距（Sim-to-Real Gap）以及交互推演过程中随时间步不断累积的微小误差发散。 - [21:59](https://www.youtube.com/watch?v=VupX7v6mocM&t=1319s) 在高安全要求的工业场景中不能完全依赖合成数据，必须构建“真实世界数据引导 + 合成数据投影 + 物理守恒与实测校验反馈”的加速飞轮。 - [27:11](https://www.youtube.com/watch?v=VupX7v6mocM&t=1631s) 由于真实机器人无法在现实中进行破坏性测试，世界模型将成为物理机器人安全认证的唯一模拟测试场，未来将催生行业专属认证机构。 - [31:17](https://www.youtube.com/watch?v=VupX7v6mocM&t=1877s) 在大型炼油厂等超大工业场景中，空间世界模型将故障定位与找阀门时间缩短 80%（从数十分钟乃至 2 小时缩短至 2 分钟），极大降低停工损失与人力成本。 - [32:58](https://www.youtube.com/watch?v=VupX7v6mocM&t=1978s) 微观硬件设计具有极高数据复杂度，将物理多模态规律嵌入模型后可将原本依赖超算或数月实验的芯片与零件迭代缩短至数天内完成。 - [35:41](https://www.youtube.com/watch?v=VupX7v6mocM&t=2141s) 训练自动驾驶世界模型必须专门注入碰撞与险情（Near-miss）等极端长尾场景数据，否则模型会默认无事故倾向甚至生成虚幻道路。 - [39:26](https://www.youtube.com/watch?v=VupX7v6mocM&t=2366s) 企业应用世界模型的关键在于掌握专有高价值数据、建立明确的量化评估体系（Evals），并从最终业务评估指标倒推所需的物理模态。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=VupX7v6mocM&t=4s) 论坛开场与嘉宾自我介绍 主持人 Emil Protalinski 以轻松幽默的方式调动现场气氛，调侃世界模型（World Models）是当下最新的“AI 流行词”（AI buzzword），并指出本次讨论将深入剖析世界模型的本质、局限性以及具体的商业应用。随后，三位来自不同领域的行业专家依次介绍了各自公司正在推进的核心业务。 ```mermaid flowchart TD accTitle: 三位嘉宾所在企业的核心方向与世界模型构建基础 subgraph Netradyne["Netradyne（交通与自动驾驶）"] N1["车端实时边缘计算"] --> N2["车队安全与事故预防"] N2 --> N3["高价值长尾极端场景（Edge Cases）"] end subgraph Vinci["Vinci（物理基础模型）"] V1["物理底层规律（热力/电磁/力学）"] --> V2["Tier-1 硬件与芯片研发"] V2 --> V3["跨尺度物理仿真（纳米到卫星）"] end subgraph NianticSpatial["Niantic Spatial（空间与视觉智能）"] S1["真实物理世界计算机视觉"] --> S2["高精度 3D 地理空间建图"] S2 --> S3["机器/智能体定位、导航与空间检索"] end ``` - **Sreekanth Annapureddy（Netradyne 首席技术官）**：Netradyne 致力于利用运行在车辆上的实时边缘计算技术，使车队运营更安全、更高效并减少事故。在车队日常行驶中，系统收集了海量真实的驾驶数据，并重点提取各类极端长尾场景（edge cases），用于训练能够模拟真实道路交互与驾驶动态的世界模型。 - **Hardik Kabaria（Vinci 联合创始人兼 CEO）**：Vinci 专注于构建物理学基础模型（Foundation Model for Physics）。Hardik 强调，周围的一切物体无论是半导体芯片还是在轨卫星，其运行都严格遵循物理定律。Vinci 为一线（Tier-1）硬件企业提供工程仿真与设计软件，将物理规律融入芯片设计与机械/热力学分析全流程中。 - **Inhi Cho Suh（Niantic Spatial 首席执行官）**：Niantic Spatial 选择从物理地球本身出发构建世界模型。Inhi 指出，与大语言模型从互联网抓取文本数据不同，物理空间的数据无法仅靠网络爬虫获得。无论是拉斯维加斯大道、纽约、伦敦还是旧金山，各地的物理基础设施千差万别。他们以计算机视觉而非纯语言为核心，开发能够让人类、机器人与自主智能体在真实物理空间中实现实时定位、导航、空间检索和预测性维护的视觉世界模型。 --- ### [03:22](https://www.youtube.com/watch?v=VupX7v6mocM&t=202s) 什么是世界模型？多维度的核心定义 面对主持人关于“如何用一句话给出极简定义”的提问，三位嘉宾从视觉空间感知、物理定律约束与行为决策预测三个维度给出了更具技术深度的解答。Inhi Cho Suh 首先指出，人类理解世界依赖多感官输入，因此她的团队坚持以真实世界的视觉数据为基底，而非直接从虚拟仿真起步。 Inhi 解释道，如果直接依赖纯虚拟仿真训练机器人，当它们部署到真实环境中时，往往会因为摄像头无法正确理解反光介质而直接撞上玻璃幕墙。只有建立真实世界的高精度网格多边形表示，机器人才能真正理解物理障碍。她分享了与加州萨克拉门托附近的兰乔科尔多瓦市（Rancho Cordova）刚刚达成的合作：团队正在对整座城市进行高保真 3D 数字化建图，用于交通流量管控、市政规划以及未来机器人的实地部署训练。 ```mermaid flowchart LR accTitle: 构建高保真世界模型的三个核心要素 subgraph Perception["1. 真实空间感知 (Visual Base)"] P1["高保真 3D 视觉网格"] P2["防范幻觉与虚假仿真"] end subgraph Physics["2. 物理规律确定性 (Physics Laws)"] H1["能量守恒 / 麦克斯韦方程组"] H2["环境跨尺度一致性"] end subgraph Conditioning["3. 动作与因果演化 (Action Conditioning)"] A1["智能体动作输入"] A2["因果推断与未来推演"] end Perception --> WM["完整物理世界模型"] Physics --> WM Conditioning --> WM ``` Hardik Kabaria 则从物理决定论的角度定义了世界模型。他认为世界模型必须满足两大核心物理准则：一是能量守恒（"Energy in is equal to energy out"）；二是计算结果的确定性与一致性。无论是在加州圣克拉拉还是在拉斯维加斯，相同边界条件下的芯片热力学问题必须得出完全一致的物理演化结果。如果对相同的物理场景给出截然不同的推演结果，那就背离了现实物理法则，不能算作真正的世界模型。 Sreekanth Annapureddy 用人类大脑的日常决策机制作类比：当我们在会议中发言或转动魔方时，大脑会瞬间运行内置的世界模型，预演老板的反应或方块的旋转结果。经验越丰富的人，见过的极端案例（edge cases）越多，其内置世界模型的预判能力就越强。在自动驾驶中，世界模型的核心就是因果状态转移——当自车（ego vehicle）或其他交通参与者采取某一动作时，周围的世界环境与动态物料将如何随之演变。 --- ### [08:35](https://www.youtube.com/watch?v=VupX7v6mocM&t=515s) 破除迷思：世界模型绝非“海量视频 + 算力堆砌” 针对主持人提出的常见误解——“只要投喂海量视频数据并堆积庞大算力就能炼出世界模型”，三位嘉宾一致表示这是一种极其危险且片面的认知。Sreekanth 指出，日常驾驶中绝大多数里程都是平淡重复的“无聊数据”（boring data），单样本所含的信息增益极低。要训练出真正可用的世界模型，关键在于两点：高价值的极端场景分布，以及“动作条件化”（Action Conditioning）数据——模型必须明确记录“采取了何种动作”与“导致了何种物理后果”之间的因果映射，单纯播放被动的视频流无法学到交互因果。 Hardik 进一步指出，纯视觉传感器存在固有的物理盲区。许多支配物理世界的关键要素在视觉上是完全不可见的： - **电磁干扰与场分布**：移动机器人或自动驾驶车辆中的芯片决策模块会受到附近电磁场的严重干扰，这种物理现象严格遵循麦克斯韦方程组，但摄像头根本无法直接捕捉。 - **多物理量通道集成**：除了几何外观外，真正的物理世界模型必须融合温度场、地面附着摩擦力（traction）、动量平衡（momentum balance）以及电磁场等多维度测量通道。 Inhi 强调，互联网上充斥着毫无价值甚至具有误导性的“劣质视频”（video slop），将这些数据直接塞入仿真系统是非常危险的。工业客户最宝贵、最核心的资产在于真实的工业现场资产数据（例如炼油厂内带有精密几何与工程约束的 CAD/BIM 结构数据），基于高价值真实场景进行多维度建模，才是构建物理世界模型的正道。 ### [11:06](https://www.youtube.com/watch?v=VupX7v6mocM&t=666s) 真实世界数据的规模化重构与产业落地 在构建世界模型时，企业所面临的数据形态远比单纯的文本复杂得多。Inhi Cho Suh 阐述了当前空间计算与世界模型在极具挑战性的物理环境中的应用潜力。例如，企业常常手握点云数据（如 E57 格式）或测绘地图，目标是在一个占地五平方英里的复杂室内外混合场地上重构出完整的三维场景。在这样的场景中，脚手架林立、金属结构遍布，GPS 信号完全失效，如何让现场作业人员、机械或自主机器人实现精准导航并安全高效地作业，是亟待解决的核心难题。 除了数据规模庞大，企业还拥有大量的历史遗留资产。Niantic Spatial 曾利用一段长达 10 年之久、水质浑浊的水下历史视频影像，成功重建了高逼真度的 3D 场景，并生成可编程、可渲染的网格模型（Mesh），用于精密观测管道和集管结构上的藤壶生长情况。在传统操作中，这类水下巡检要么需要潜水员进行高危作业，要么必须派遣极其昂贵的自主水下航行器（AUV），不仅成本高昂而且巡检频次极低。通过世界模型，企业可以在数字世界中分析沉积物位移、机械部件连接脱落等潜在风险，显著优化运营效率。此外，在城市级场景中（如与加州 Rancho Cordova 市的合作），团队结合无人机航拍数据与地面消费级 360 度全景相机采集的影像，以低成本、易普及的设备实现了从空中到地面的大范围场景数字化。Inhi 强调，行业必须聚焦于真实物理世界中解决实际问题，远离网络上充斥的“视频垃圾”（video slop）。 ```mermaid flowchart TD accTitle: 异构物理数据到世界模型应用的全流程 subgraph DataSources["多源异构现实数据"] A["高精度测绘点云 / E57"] B["10年水下浑浊历史影像"] C["无人机低空航拍"] D["地面 360° 全景消费级相机"] end subgraph WorldModelCore["空间世界模型重构与物理表征"] E["高逼真度 3D 场景重建"] F["可编程可渲染 3D 网格 (Mesh)"] G["无 GPS 弱纹理空间定位"] end subgraph BusinessOutcomes["高价值工业与城市应用"] H["GPS拒止环境下的工人/机器人导航"] I["水下管网藤壶生长与沉积物监测"] J["城市级基础设施数字孪生巡检"] end DataSources --> WorldModelCore WorldModelCore --> BusinessOutcomes ``` --- ### [13:02](https://www.youtube.com/watch?v=VupX7v6mocM&t=782s) 超越文本提示词：世界模型的多模态交互方式 主持人 Emil Protalinski 抛出了一个核心问题：当大众习惯了 ChatGPT、Claude 或 Gemini 这种“在提示词框中输入文本，获取文本/图像/视频”的交互范式时，世界模型的企业客户究竟是如何与之交互的？未来这种交互模式又将如何演进？ Netradyne 的 Sreekanth Annapureddy 表示，他们目前正处于前沿研究阶段。一个直观的应用方向是商用司机的安全培训。例如，当司机发生闯红灯行为时，传统方式很难让司机直观理解“如果当时发生意外会怎样”。借助世界模型，系统可以输入特定场景并让其“想象并生成潜在的灾难性后果”，从而以高度可视化的方式为司机提供更强效的安全反思与强化教学。另一个重要交互场景是自主智能体的测试评估——在物理世界中大规模测试自动驾驶或机器人的安全性代价高昂且风险极大，而世界模型提供了一个高保真的模拟测试场。 Vinci 的 Hardik Kabaria 则指出，他们的产品直接嵌入到了工业工程设计的工作流之中： - **空间几何与布局输入**：客户提供零部件的 3D 几何外形、电子元件排布等空间数据； - **物理环境场设定**：设定工件的服役环境（例如 80 摄氏度的高温表面等外部物理场）； - **物理量推理提问**：工程人员向模型提问，如“该电子器件达到的最高温度是多少？”或“该部件在受力后会产生怎样的弯曲形变？”。 Hardik 强调，这彻底颠覆了以往在火箭或硬件开发中“设计—制造—实验室测温/测压—发现缺陷—推倒重来”的昂贵实体实验循环。在世界模型中，“输入”就是实验配置与物理边界条件，“输出”则是高精度的物理场测量结果，整个研发闭环得以全数字化快速推进。 --- ### [16:01](https://www.youtube.com/watch?v=VupX7v6mocM&t=961s) 视觉定位系统（VPS）与物理语义理解的前沿探索 对于 Niantic Spatial 而言，交互方式涵盖了从基础软件到高层 API 的全栈形态。Inhi 介绍了他们与无人机航拍重建团队 Specsy 的合作，通过模型快速重构大型城市、复杂仓储中心、研发实验室、建筑工地、炼油厂以及海军港口。此外，他们面向机器人领域提供了专用的 SDK 和 API 接口，其中最典型的是与最后一公里配送机器人公司 Coco Robotics 的合作，用于解决业内著名的**“迷失机器人问题”（Lost Robot Problem）**。 在城市密集的高楼峡谷中，传统卫星 GPS 定位的误差常常超过 10 米甚至更多，导致配送机器人无法导航。Niantic Spatial 提供的视觉定位系统（VPS）API，就像人类在迷路时抬头辨认路牌、餐馆招牌并对照地图一样，让机器人通过视觉特征进行厘米级精度的即时重定位，保证最后一公里配送任务的稳定执行。 Inhi 进一步指出，世界模型正在迈向比语言语义更深一层的“物理语义理解”（Physics Semantic Understanding）： 1. **隐式物体分割与物理属性感知**：模型不仅要在场景中识别并分割物体，更要直接在底层代码表示中原生理解物体的物理尺寸、三维几何形态及空间深度，而无需依赖大量手工标注； 2. **长周期时序变化检测**：捕捉现实场景在间歇性时间跨度下的微小形变与状态演迁。 她坦言，目前行业对于世界模型的探索“甚至连第一局比赛的第一棒打者都还没打完”，一切才刚刚开始。 --- ### [18:53](https://www.youtube.com/watch?v=VupX7v6mocM&t=1133s) 当前世界模型的核心技术局限与挑战 面对主持人关于“世界模型当前局限性”的提问，三位嘉宾从不同维度剖析了技术发展中的深层瓶颈： Hardik Kabaria 认为，物理学是硬件与真实世界的“底层编译层”（compilation layer）。人类生存于世并未发明物理定律，而是学会了遵守并利用物理定律。世界模型的核心起点在于将物理定律内嵌至模型中，只有这样，未来训练出的设计智能体或城市级机器人集群在做出决策、执行群体协同动作时，才绝不会违反客观物理规律。 ```mermaid flowchart LR accTitle: 模拟到现实差距与误差随时间累积示意图 subgraph SimTimeline["模拟推演时间轴"] T0["初始状态: 真实度极高"] T1["单步决策产生微小偏差"] T2["长尾/复杂动作交互"] T3["误差指数级发散 (Sim-to-Real Gap 扩大)"] end T0 --> T1 T1 --> T2 T2 --> T3 ``` Sreekanth Annapureddy 指出，当前最致命的瓶颈在于**“模拟到现实的差距”（Sim-to-Real Gap）**： - **长尾场景的失真**：在常规、单调的场景中模拟精度尚可，但在长尾、极端边缘场景下差距会被无限放大； - **时间步累积误差（Drift & Divergence）**：世界模型推演的时间越长、执行的交互动作越多，单次细微的决策偏差就会如同机器人控制误差一样不断叠加，最终导致模拟轨迹严重偏离现实，彻底失去参考价值。 Inhi Cho Suh 补充道，现实规模化部署对精度的要求极度苛刻——在某些工业场景中，1 毫米、1 厘米或 1 米的定位偏差都会导致完全不同的系统性后果。要真正攻克因果推断与物理仿真的难题，其最大的障碍依然是**极高品质的真实世界多模态数据**。她当场呼吁各企业、市政机构展开开放合作，共同贡献具有挑战性的真实场景数据以推动世界模型的底层突破。 ### [21:59](https://www.youtube.com/watch?v=VupX7v6mocM&t=1319s) 真实数据与合成数据之辩：构建闭环加速飞轮 在复杂的工业运营与物理工程中，数据质量关乎生命安全，容不得丝毫差错。Inhi Cho Suh（Niantic Spatial CEO）指出，工业领域的操作流程高度复杂，且经过了机器学习算法和时间序列的精细调优；面对这类高强度工业场景，“人命关天，你绝不能仅仅依靠合成数据来做可能会产生数量级偏差的决策”，真实世界数据是不可替代的基石。为此，Niantic 推出了可在 iOS 和 Android 商店下载的应用 Scaniverse，让用户能够通过便捷的 3D 扫描对空间和物体进行测量与空间定位（Localization），从而转变人们审视工作设施与工作方式的思维模式。 对于合成数据（Synthetic Data）的角色定位，嘉宾们展开了深入探讨。Inhi 认为，纯粹从合成数据起步去预测下一个画面的模型往往“过于狭窄且存在偏差”，真正能够带来质的飞跃的是“真实数据增强 + 合成数据投影 + 真实数据强化反馈”的加速飞轮。Hardik Kabaria（Vinci 联合创始人兼 CEO）对此深表赞同，他强调 Vinci 在训练物理模型时不使用客户的私有数据，而是依靠符合底层物理规律的合成数据，并在每次推理时进行物理守恒验证（例如“输入能量必须等于输出能量”），将有效信号反哺到检查点（Checkpoint）中，以极快速度为半导体和硬件公司提供复杂物理场景的推理。而 Sreekanth Annapureddy（Netradyne CTO）则从自动驾驶视角提出了另一种见解：他更倾向于将合成数据视为世界模型的“输出端”（用世界模型生成长尾极端场景视频去训练下游感知模型），因为现实中始终存在巨大的“虚实迁移差距”（Sim-to-Real Gap），模型必须牢牢扎根于现实边缘场景。 ```mermaid flowchart LR accTitle: Real-Synthetic Data Flywheel subgraph Loop["数据闭环加速体系"] direction TB R["真实世界多维数据<br/>(Real-World Data)"] -->|"引导与微调"| W["世界模型 / 物理模型<br/>(World Model)"] W -->|"生成长尾与未知场景"| S["合成数据 / 模拟仿真<br/>(Synthetic Data)"] S -->|"物理约束与实测校验<br/>(如能量守恒/现场验证)"| V["置信度校验与强化反馈<br/>(Validation & Feedback)"] V -->|"迭代模型权重"| R end ``` - **Scaniverse 移动应用**：支持在手机端进行便捷的 3D 扫描、尺寸测量与空间定位，帮助用户快速体验空间数字化的工作范式。 - **物理守恒约束推理**：Vinci 在模型推理（Inference）阶段实时执行物理规律校验（如能量守恒：Energy in = Energy out），确保无客户数据泄露的前提下输出高精度物理预测。 - **虚实迁移挑战（Sim-to-Real Gap）**：在自动驾驶等涉及长尾效应（Long-tail）的领域，直接用纯合成数据训练世界模型存在失真风险，必须以真实边缘案例为基础进行锚定。 --- ### [26:36](https://www.youtube.com/watch?v=VupX7v6mocM&t=1596s) 世界模型落地的核心技术瓶颈与边缘端挑战 谈及世界模型当前面临的最大障碍，Sreekanth Annapureddy 总结了两个核心痛点：长尾边缘案例（Corner Cases）的现实拟合，以及庞大模型体量的压缩与实时化。在 Netradyne 的实际业务中，AI 算法必须直接运行在车载边缘计算设备上，这就要求动辄数十亿参数的庞大世界模型必须经过极其高效的知识蒸馏（Distillation）与模型量化，才能在有限算力的车载硬件上实现低延迟的实时推理。 - **极端边缘场景拟合**：真实世界的物理动态与道路状况极其多变，模型需要具备对低概率、极端长尾情景（Corner Cases）的精准理解和物理常识推理能力。 - **模型轻量化与端侧实时推理**：车载设备与工业边缘硬件的算力和功耗极为有限，必须将大模型压缩到可实现毫秒级实时响应的尺寸。 --- ### [27:11](https://www.youtube.com/watch?v=VupX7v6mocM&t=1631s) 应用场景展望：从机器人安全认证到芯片物理模拟 当讨论转向当下最佳落地场景与未来愿景时，三位行业领袖展示了从微观芯片到宏观实体世界的广阔图景。Sreekanth 展望道，在不久的将来，大量实体机器人将进入人类日常生活与儿童玩耍的物理空间；鉴于机器人部署后无法直接在现实中进行破坏性安全测试，基于世界模型的仿真模拟将是唯一的安全测试与认证途径。这进而引发了更深层次的行业治理讨论——"如果每家公司都开发自己的模拟器来认证自家机器人，那么谁来认证这些模拟器本身？" 现场讨论指出，未来可能会出现针对特定垂直行业（如核聚变设施与自动驾驶汽车分别由不同机构验证）的专业认证机构和基于物理基准的通用评估标准。 Hardik Kabaria 则聚焦于硬件加速与半导体领域。他指出，随着 AI 被全面嵌入到各类边缘设备、数据中心和自动化硬件中，人类正在以前所未有的速度设计制造新硬件，而这背后充斥着复杂的电磁学、热力学等物理规律。Vinci 正在通过将物理规律嵌入工程工作流，帮助半导体客户实现极速的微观物理仿真。Inhi Cho Suh 则描绘了人机协作的未来：未来世界中的机器人数量可能会超过人类，而人类也将超越传统智能手机形态，佩戴全新的穿戴式设备（Wearables）与现实环境深度交互。Inhi 分享了自己登顶珠峰及在喜马拉雅山脉徒步的个人经历，强调人类在亲身体验自然的同时，渴望以全新方式捕捉记忆、感知环境并在世界模型的实时赋能下获得更沉浸的支持。 - **机器人仿真与认证生态**：模拟测试是实体机器人入场前的必经之路，未来各行业（如交通、工业、能源）将建立独立的仿真器安全认证标准与物理基准。 - **芯片与硬件设计加速**：AI 无处不在的趋势倒逼半导体与电路硬件呈指数级增长，利用世界模型加速物理仿真成为突破半导体设计周期的关键抓手。 - **下一代空间穿戴设备**：超越手机屏幕交互，基于空间视觉栈的可穿戴设备将使人类在旅行、勘探和复杂现场工作中获得实时情境感知与智能辅助。 --- ### [31:17](https://www.youtube.com/watch?v=VupX7v6mocM&t=1877s) 工业级落地实战：炼油厂维护导航的百倍 ROI 跃升 在具体企业落地案例与投资回报率（ROI）方面，Inhi Cho Suh 详细分享了 Niantic Spatial 在大型炼油厂（Oil Refinery）的落地成果。大型炼油厂内部管线纵横交错、空间极其庞大，现场维护人员寻找目标阀门和管道盲板（Blinds）如同“在拉斯维加斯大海捞针”。在传统作业模式下，维修人员只能依赖纸质蓝图、粗糙的 2D/3D 地图以及大致方位推测进行寻路，导致即使是工作了 20 年的老员工通常也需要耗费约 10 分钟才能找到指定点位，而对厂区不熟悉的新员工寻找目标甚至需要 2 个小时以上。 通过利用世界模型技术对复杂厂区进行高精度空间重建，并结合手机端厘米级空间定位，Niantic Spatial 能够为现场工人提供如同 AR 导航一般的直观路线指引，将原本需要数十分钟乃至数小时的寻路过程直接缩短至仅需 **2 分钟**。这种空间定向能力的跃升不仅极大降低了新员工的培训门槛，更直接重塑了工业维护流程的时间经济学与运营 ROI。Hardik 随后也展开了微观层面的类比：在芯片制造与内存微架构中，工程师同样需要在纳米级别的密集导线网络中排查物理异常，世界模型在微观世界的精准寻路与仿真同样带来了不可估量的效率飞跃。 | 评估维度 | 传统工业运维导航方式 | 基于空间世界模型的实时 AR 定位导航 | |:--- |:--- |:--- | | **导航工具** | 纸质图纸、粗略 2D 地图、经验估算 | 手机/移动设备高精度空间定位、3D 实景导引 | | **资深工人耗时** | 约 10 分钟（依赖长期记忆与经验） | 缩短至 **2 分钟** 以内 | | **新员工耗时** | 往往高达 **2 小时** 以上（极易迷失方向） | 缩短至 **2 分钟**（零经验平滑上手） | | **核心商业价值** | 停机维护周期长、人力成本高、存在安全盲区 | 极大压缩检修停机时间，消除寻路误操作风险 | ### [32:58](https://www.youtube.com/watch?v=VupX7v6mocM&t=1978s) 物理计算的精细尺度与硬件设计的 ROI 在探讨微型物理尺度的计算难题时，Hardik Kabaria（Vinci 联合创始人兼 CEO）将手机内部极精密的结构比喻为“在纽约市地图上辨识人类头发的丝缕”。例如芯片或电子元件内部的散热与电信号传输，其复杂程度堪比厘米级别的管道与道路系统。这类硬件设计的几何与空间数据占据了数个 GB 的存储空间。在过去，想要找出设计瓶颈并迭代方案，往往需要数百名工程师耗费巨量时间，因为没有任何单个人脑可以完全合成并理解如此庞大的数据，最终大家不得不采用近似法，拉长了分析周期。 现在，通过引入嵌入物理规律的模型，客户能够在几天内完成原本需要超算或物理实验才能解决的分析与设计变更，彻底颠覆了传统的开发流程。Hardik Kabaria 指出，衡量这种技术的投资回报率（ROI）不能仅看消耗了多少 Token，关键在于如何帮助客户做出更好的物理决策、加快研发迭代，并让“物理智能”（Physics Intelligence）变得触手可及。 与硬件设计的高精尖类似，Sreekanth Annapureddy（Netradyne CTO）探讨了车队管理中的实用投资回报。通过实时监控与预防，系统能够将安全事故减少 50% 至 60%，并将驾驶员不系安全带、边开车边发短信等危险行为降低 90%。车队日常运营产生的庞大“数据废气”（Data Exhaust）则反哺给世界模型进行训练。随着行业向自主智能体（Autonomous Agents）和自动驾驶车辆转型，世界模型将释放出更巨大的商业价值。 - 芯片及微型电子元件的空间几何与物理数据规模可达数个 GB。 - 传统方法依赖数百名工程师手动分析和近似估算，耗时且容易出错。 - 引入物理世界模型后，原本极其耗时或依赖超算的迭代分析可缩短至数天内完成。 - 车队安全系统可将交通事故率降低 50%~60%，并将驾驶员违规行为减少 90%。 --- ### [35:41](https://www.youtube.com/watch?v=VupX7v6mocM&t=2141s) 训练世界模型过程中的意外发现与突破 谈到训练世界模型时的意外惊喜，Hardik Kabaria 表示，他们最初以为训练物理模型必须为电子元件和机械零件分别建立独立的起点。然而在实际探索中，他们发现现实世界天生就是“多物理场”（Multiphysics）交织的，人类人为地将热学、电学等学科拆分开来，但物理现象本身同时发生。当团队将多种物理模态融合进同一个模型时，模型的学习速度反而更快，且泛化能力显著增强。只要将底层物理规律嵌入其中，多物理场模型就能展现出超乎预期的泛化效果。 Sreekanth Annapureddy 则分享了自动驾驶世界模型训练中的“踩坑”经历。他们训练的第一代世界模型在模拟车辆发生碰撞后，竟然像什么都没发生一样继续往前开。原因在于绝大多数日常驾驶数据都是平淡无奇的，模型默认倾向于无事故状态。为了让模型真正理解事故及其后果，团队必须专门引入事故和险情（Near-miss）数据。此外，模型在预测偏离道路时，有时会因为没把侧边的房屋识别为障碍，而直接“梦幻般”地生成一段高速公路。这些现象反向指明了数据清洗与强化的方向。 Inhi Cho Suh（Niantic Spatial CEO）提到，他们最惊喜的突破在于从“最差的数据”（如低端手机拍摄的劣质、模糊或角度偏移的数据）起步，最终成功开发出能够重建几何精准、具备物理参考系（如 6 自由度 6DoF 方位）场景的独特算法。他们将地面数据、卫星图与无人机航拍数据相融合，使模型能够快速重新定向。在研究阶段，团队还在探索空间数据的嵌入与编码（Embedding & Encoding），力求以更少的资源和更低的数据成本，打造强大的“空间智能层”（Spatial Intelligence Layer）。 ```mermaid flowchart TD accTitle: 多源空间数据融合与物理世界模型构建流程 subgraph Data_Collection["多源异构数据采集"] A["移动设备低质数据\n(手机/劣质镜头)"] B["空地空天多模态数据\n(卫星/无人机/地面)"] end subgraph Processing_Engine["空间智能与多物理场融合"] C["6DoF 定位与几何重建"] D["多物理场嵌入\n(热学/电学/力学)"] E["数据编码与空间嵌入"] end subgraph Output_Layer["世界模型应用场景"] F["高精度物理仿真与硬件设计"] G["空间智能与自主系统决策"] end A --> C B --> C C --> E D --> E E --> F E --> G end ``` --- ### [39:26](https://www.youtube.com/watch?v=VupX7v6mocM&t=2366s) 企业应用世界模型的落地建议与炼油厂案例 对于希望开始应用世界模型的企业和从业者，三位嘉宾提出了具体的可操作建议： 1. **重视数据与评估（Evals）：** Sreekanth Annapureddy 强调，尽管基础模型正变得越来越便宜和标准化，但企业的核心竞争壁垒依然在于**数据**。企业需要深度理解并精细化清洗自己的环境数据。同时，必须建立清晰的评估体系（Evals），尽可能量化想要达成的目标。 2. **从应用场景逆推（Reverse Engineering）：** Hardik Kabaria 建议企业从终极目标倒推——无论是设计零件、部署设备还是控制机器人，先要想清楚“如何评估这是一个好产品或好系统”。搞清楚评价标准后，自然就能明白需要引入哪些物理模态（如摩擦力、热能或空间几何），进而选择合适的技术栈。 3. **寻找高价值落地切入点：** Inhi Cho Suh 举了一个极其震撼的炼油厂案例：一个中型炼油厂如果停工一天，造成的损失可能高达 1000 万美元。而在日常维护中，寻找故障点就像“在针堆里找一根针”。通过引入空间世界模型，技术人员定位故障的时间缩短了 80%，一位现场技术人员直言：“如果我必须在针堆里找针，这就是唯一可行的方法。” Inhi Cho Suh 补充道，甚至原本用于空中航拍的数据，在迁移到水下场景后依然能够有效工作。她鼓励企业带着自己独特的 2D 图像或 3D 视频数据（无论来自工厂、施工现场还是运营区）积极尝试与探索。同时，为了保护企业的数据资产，未来的趋势还将包括模型的容器化部署（Containerization），让世界模型能够在受保护的私有环境中安全运行。 - 炼油厂每日停工或周转成本可达约 **1,000 万美元**。 - 引入世界模型后，故障定位与定位排查耗时降低了 **80%**。 - 即使只有普通的 2D 图像或视频，也能通过世界模型构建出可用于规划、模拟和运营的空间模型。 --- ### [43:12](https://www.youtube.com/watch?v=VupX7v6mocM&t=2592s) 展望未来：十年内将被世界模型彻底重塑的行业 在研讨会落幕之际，主持人邀请三位嘉宾各用一句话预测未来十年内将被世界模型彻底重塑的行业。嘉宾们给出了各自的终极展望： - **Hardik Kabaria：** 自动驾驶以及每一个硬件零件的设计与部署方式。 - **Sreekanth Annapureddy：** 自动驾驶与智能交通领域。 - **Inhi Cho Suh：** 城市规划与基础设施。她表示：“如果世界上每一个城市的城市规划和基础设施都能因此变得更好，那将是最理想的未来。” ## Notable quotes > [03:22](https://www.youtube.com/watch?v=VupX7v6mocM&t=202s) Energy in is equal to energy out — Hardik Kabaria > [16:01](https://www.youtube.com/watch?v=VupX7v6mocM&t=961s) 甚至连第一局比赛的第一棒打者都还没打完 — Inhi Cho Suh > [21:59](https://www.youtube.com/watch?v=VupX7v6mocM&t=1319s) 人命关天，你绝不能仅仅依靠合成数据来做可能会产生数量级偏差的决策 — Inhi Cho Suh > [27:11](https://www.youtube.com/watch?v=VupX7v6mocM&t=1631s) 如果每家公司都开发自己的模拟器来认证自家机器人，那么谁来认证这些模拟器本身？ — Sreekanth Annapureddy > [31:17](https://www.youtube.com/watch?v=VupX7v6mocM&t=1877s) 在拉斯维加斯大海捞针 — Inhi Cho Suh > [32:58](https://www.youtube.com/watch?v=VupX7v6mocM&t=1978s) 在纽约市地图上辨识人类头发的丝缕 — Hardik Kabaria > [39:26](https://www.youtube.com/watch?v=VupX7v6mocM&t=2366s) 如果我必须在针堆里找针，这就是唯一可行的方法。 — Inhi Cho Suh > [43:12](https://www.youtube.com/watch?v=VupX7v6mocM&t=2592s) 如果世界上每一个城市的城市规划和基础设施都能因此变得更好，那将是最理想的未来。 — Inhi Cho Suh ## Mentioned Emil Protalinski, Sreekanth Annapureddy, Inhi Cho Suh, Hardik Kabaria, Netradyne, Vinci, Niantic Spatial, Techmeme, EPro Strategies, Rancho Cordova, Specsy, Coco Robotics, Scaniverse, ChatGPT, Claude, Gemini ## Video description Emil Protalinski - Editor and Consultant, Techmeme and EPro Strategies Sreekanth Annapureddy - CTO, Netradyne Inhi Cho Suh - CEO, Niantic Spatial Hardik Kabaria - Co-founder and CEO, Vinci World models allow AI to do more than just generate text, they allow it to understand cause and effect in the physical world. This panel features industry leaders discussing how they are leveraging world models to create photorealistic video content, test autonomous systems, and optimize supply chains through advanced simulation. Learn how the ability to simulate reality is becoming a critical competitive advantage across industries. [Source](https://www.youtube.com/watch?v=VupX7v6mocM)

[3:22] Hardik Kabaria: Energy in is equal to energy out [16:01] Inhi Cho Suh: 甚至连第一局比赛的第一棒打者都还没打完 [21:59] Inhi Cho Suh: 人命关天，你绝不能仅仅依靠合成数据来做可能会产生数量级偏差的决策 [27:11] Sreekanth Annapureddy: 如果每家公司都开发自己的模拟器来认证自家机器人，那么谁来认证这些模拟器本身？ [31:17] Inhi Cho Suh: 在拉斯维加斯大海捞针 [32:58] Hardik Kabaria: 在纽约市地图上辨识人类头发的丝缕 [39:26] Inhi Cho Suh: 如果我必须在针堆里找针，这就是唯一可行的方法。 [43:12] Inhi Cho Suh: 如果世界上每一个城市的城市规划和基础设施都能因此变得更好，那将是最理想的未来。

[0:04] Emil Protalinski: Who wants to hear about world models? Oh, come on. You can do better than that. Who wants to hear about the latest AI buzzword? [0:04] Inhi Cho Suh: there's one world that we all live on called Earth. [3:22] Hardik Kabaria: energy in equals energy out [8:35] Inhi Cho Suh: video slop [21:28] Inhi Cho Suh: People's lives are at stake and you don't want to make mistakes at an order of degree based on just synthetic data. [21:28] Hardik Kabaria: energy in is equal to energy out [27:11] Sreekanth Annapureddy: who's going to certify the simulator? [36:37] Sreekanth Annapureddy: start dreaming about a highway. [36:37] Inhi Cho Suh: worst data possible [41:22] Inhi Cho Suh: finding a needle in a stack of needles

[0:04] All right, AI4. Who wants to hear about world  models? [laughter] Oh, come on. You can do better [0:10] than that. Who wants to hear about the latest  AI buzzword? Yeah, world models. All right. So, [0:16] today you're going to hear about what world models  are. You're going to hear about their limitations, [0:21] and you're going to hear about their applications.  Um, but that's enough for me. Why don't you guys [0:27] introduce yourselves one by one and give us a  little taste of what you do at your respective [0:31] country uh respective companies starting with  you Sri K. Hi uh so I'm Sri Kant uh and I'm a [0:38] CTO at Natur basically what we do is uh we provide  technology for fleets to become safer efficient [0:44] uh and smarter um and we deploy technology um  that runs in real time on the vehicles to reduce [0:51] accidents and so on. uh as part of that we collect  like lot of driving data and with that we're [0:55] training world models with lot of color cases that  can kind of simulate how the driving works. Uh [1:02] my name is Hardik. I'm co-founder CEO of WinI. Uh  we built foundation model for physics. Everything [1:09] around us is governed by laws of physics.  We want to understand it to deploy products, [1:14] how semiconductors behave, how satellites behave.  Uh so we deploy uh software for tier one hardware [1:21] companies so their engineering um teams can  utilize and make physics accessible throughout [1:28] the design process helping them understand thermal  problems and the mechanical problems and so on. Um [1:35] I've been in the same space of understanding  making physics accessible for more than 10 [1:39] plus years and we started this journey now three  years ago and starting to pick up as you can all [1:44] imagine. World is governed by laws of physics and  that's where we fit into the system. Hi everyone, [1:51] my name is Inhy. I am the CEO of Niantic  Spatial and we're building world models as well. [1:59] uh we're actually starting with the real  world because there's one world that we all [2:02] live on [laughter] called earth. Um and when I  think about world models uh what's fundamentally [2:09] different especially for the physical world is we  start with computer vision not language. That's [2:14] point one. Second is we think about real world  spaces at scale. You can't scrape data from the [2:21] internet. It like the conference room setups,  Las Vegas strip, you know, New York City. I [2:28] happen to be based in San Francisco. We also have  a headquarters in London. Each of these cities, [2:33] towns, communities around the world have  very different physical infrastructure. [2:38] And as we train, we're thinking of how to navigate  people, machines, agents within these spaces for [2:48] real time operations, simulation, training. It  could also be intelligent predictive maintenance. [2:54] It it's a number of aspects. So, we're building  essentially vision models that allow you to [2:58] position, navigate these spaces, localize in  them, and eventually be able to do even more [3:04] interesting things like search. Um, and today  most search technologies are really predicated [3:10] on language. It doesn't understand the physics  of things, like how things move in spaces. Um, [3:17] and that's the intersection that we're working on.  Awesome. Thank you. I would like to actually hear, [3:22] so there's been a ton of different ways of  describing world models. I'd love to hear [3:27] each of your definitions. um if you could if  you could give me kind of just like a really [3:31] simple definition maybe like a one sentence how  you think of a world model starting with you and [3:36] let's go the other way. Oh okay. So I think about  it at a couple levels. First is um the real world. [3:44] So meaning can I see it visually? Like as a human  we don't just speak we hear we listen we there's [3:51] more sensory. This specific stack we've got is a  confus computer vision stack. So um part of that [3:58] is rather than starting with simulation there's  a theory of starting at that angle we actually [4:04] start with real world data. I like that because it  creates the foundational base on top of which you [4:09] might want to generate simulations. Main reason  is we want to make sure that you have also less [4:15] likelihood of hallucination that you're going to  have better precise deployments where today when [4:22] you see robots trained on simulation and you  deploy them actually in the physical setting [4:27] they're running into you know anything that is  glass. Why? because the [laughter] camera doesn't [4:32] understand like there's some basic elements of  understanding the translation of what you see to [4:39] then a mesh with a number of right polygons for  a robot to then comprehend because the robot has [4:45] to be trained so that degree of precision and  partnership is part of what we're working on [4:50] um at one scale is in robotics the other scale we  actually launched this week Tuesday a significant [4:56] announcement with the uh town of Rancho Cordova  it's a city right outside of Sacramento, [5:01] California. We are mapping that entire city in  a 3D highfidelity resolution space. It's the [5:09] very first scenario. We're then doing um and in  partnership with a couple other partners rendering [5:16] like helping them render traffic management. Uh  imagine how they would train robotics deployments, [5:22] train other types of city planning. It's a  completely different way of working. That's a [5:27] that's quite a quite a long definition. Hardock,  can you give a shorter one? [laughter] Uh I'll [5:32] go back to the same thing you started with which  is everything around us is spatial. So there are [5:37] different materials at different resolution and  all of these things whether it's at the sort of [5:43] city scale or within semi semiconductor nanometer  scale is gone by laws of physics. That's the world [5:49] we live in. We can't really violate it. That's  what the world we live in. So the world model [5:53] would be something that understands it and can  predict on it given some set of materials and some [6:00] environmental condition whether it's at the city  scale or semi semiconductor scale it understands [6:06] and provides an answer and I'll tack on the other  stuff which is it does have to satisfy certain [6:11] criterias. Imagine you have a thermal problems  in Santa Clara or thermal problem in Las Vegas. [6:19] they kind of have the same answer. If you have  the same conditions and that world models will [6:24] also have to behave by that. If you ask the same  question, you get a different answer. Then it's [6:29] not really world model because you are diverging  from how the world behaves around us. So those [6:35] would be my definition of like the modality is  different. Energy in is equal to energy out. [6:40] We'll have to be satisfied. And lastly, there is  a level of determinism that it needs to provide. [6:46] Once we get there then that's the world model  and of course that's the sort of the world we [6:50] are living in and that's where we deploy for our  customers. So I think of like world models with [6:58] uh so with the human analogy. So pretty much all  of us have like a world model like running in our [7:02] brain. So say you saying something in a meeting  and you would run like a simulation of how your [7:07] boss may uh like may react or how like your  team may react and based on that you would kind [7:12] of adapt to that right and say I'm solving like a  Rubik's cube. If I turn it this way then how would [7:18] it change and so on right so pretty much like  you know we as humans run world models and the [7:23] more experienced people essentially gather more  experiences of more and more common cases so the [7:29] more of edge cases that you see then the better  world model that you have and the better world [7:35] model that you have you can make better decisions  because you can better predict what may happen if [7:41] I make this decision what may happen if I take  this action and so on Right. So in the context [7:47] of driving so for example like so the driver  specifically may take some action or some other [7:53] agent on the road can take a different action. So  they the world model is like you know by kind of [7:58] the ego person or other agents on the kind of  in the world if they take a particular actions [8:03] or particular decisions how would that change the  world right like that's kind of how I look at it. [8:08] Okay. So, in you started with the visual what  what we see around us making sure that everything [8:13] that we see and I think you mainly focused on the  physics. So, making sure that the the everything [8:20] that we do see that it interacts with the law  of physics that we as we know them and then [8:24] you focused more on edge cases and every single  action and kind of building up all the actions [8:30] that contribute to that world model behaving as  as we would expect. Um, one way of thinking about [8:35] world like a very oversimplistic way I would think  of getting to a world model would be taking a ton [8:41] of video data and adding a ton of compute. That's  the oversimplistic way of looking at it. Now, [8:46] please dispel that and explain to the audience  how it actually works for when you're training [8:50] a world model starting with let's say use three  con. So, I mean that's largely the premise of [8:55] ours as well. We have like a ton of video data  but again like most of the driving is let's say [9:00] very boring. So if you think of like intelligence  per sample lot of samples don't really add value [9:06] it's really the edge cases that add a lot of value  but it's basically you throw in lot of data lot of [9:11] compute plus also you also need to understand the  action conditioning so for example if the agent [9:18] take this action then what was the outcome and so  on so it's not just only video data you also need [9:24] to throw in other action and outcome kind of a  data and throw a lot of compute and I get like a [9:30] hard to I'm saying I'm guessing you want to just  add a bunch of physics formulas. Yeah, because [9:35] we can easily think of cases. Let's say you have  electronics component on a robot or a car. It's [9:40] moving around and it interfaces with something  else. You can easily have signal inference. Now [9:46] the moment you have interference between the  two things, it's going to behave differently. [9:50] your chips or um whatever electronics box you have  that is making decisions is going to be affected [9:56] by electromagnetic field of something nearby.  The reason I give this example, it's very hard [10:01] to see electrical field and magnetic field yet  it will be behaved by Maxwell's equation. So yes, [10:07] you can capture geometry around you with visual  sensing but to really get to the world model [10:14] you'll have to amend it with bunch of other  stuff. Some of those are physical things that [10:18] we can measure like temperature, the traction, the  momentum balance, electrical field and magnetic [10:24] field. So the world model has lot more channels  very similar to what you said is a lot more sens. [10:34] I think we are getting closer to the world model  and then we can say okay we have thoughts and we [10:38] know how they'll behave and they can be operated  by agents not humans. I agree with that. um know [10:47] there's a lot of video slop that's just not  valuable actually and quite frankly dangerous [10:53] if you are trying to put into simulations. So  I'll start there. Uh most customers have probably [10:58] the most precious data sets. So we're working with  for example um an oil refinery with existing Navis [11:06] maps or E57 point clouds right so they've captured  it. Can you recreate entire 3D scenes of you know [11:14] five square mile of a very complex indooroutdoor  site with a lot of scaffolding a lot of metal GPS [11:22] doesn't work and how do you get workers navigating  to where they need to be whether it's human or a [11:26] machine or a robot and and um and operate in an  intelligent way in a safe way right so that's one [11:33] piece the the scale of the data set too is not  just one type so not only do enterprises have a [11:40] lot of data. They also have old data. So, we even  took this 10-year-old footage of underwater murky [11:48] data to recreate um uh understanding in a 3D high  photorealistic way that is also in a programmable [11:59] renderable mesh to look at barnacle growth on  manifolds and pipes. And historically, you'd have [12:06] to do very dangerous work or send an autonomous  vehicle and it's in incredibly expensive. You [12:13] can't do it frequently. And what you're trying to  do is look at shift in sediment. You might want [12:18] to see if there's a a disconnect in certain parts  of the mechanical pieces. But those are real data [12:23] sets that exist. Can we do something meaningful in  a way that your operations are improving and more [12:29] effective and better? Um, there's data sets that  we're thinking about at the drone level. I mean, [12:34] the partnership with Cordova is is the Rancho  Cordova city is fantastic because it's a very [12:39] simple way that we're able to take, you know,  drone level data and then be on the ground [12:44] in the city with like 360 cameras, you know,  not expensive equipment, but things that are [12:50] applicable available for everybody. So, these are  just a few examples. I I think you have to keep it [12:56] real because there's a lot of, you know, bad video  uh video slop on the internet. [laughter] Yeah. [13:02] No, for sure. Um, I'd love to hear more a little  bit more. So, I think we're starting to understand [13:08] world models. I'd love to hear one more thing.  So, currently everyone in the audience, I'm sure, [13:13] has interacted with, you know, Chad GBT, Claw,  Gemini, what have you. The way you interact with [13:18] them is there's a prompt box and you, you know,  put something in, you get something out. But then, [13:21] whether that's text, whether that's image,  whether it's video, how are your customers [13:24] interacting with your world models? What what is  the modality they're doing that in today? And how [13:29] do you see that changing in the future? How are  they going to interact with them in the future? [13:33] Let's let's start with you sir. So we're currently  in a uh like a research phase. We haven't deployed [13:38] to production but one of the immediate use cases  for our customers is we coach commercial drivers [13:43] to be safer while driving and so on. So one thing  that's very visual is sometimes if you run like [13:48] a red light um you don't really understand like  what could have happened like you know had things [13:54] gone wrong. So you could now like you know prompt  it saying that like you know hey this situation [14:00] imagine something bad that could have happened and  that's like a much more powerful way of coaching [14:05] and kind of uh like reinforcement and so on.  The other natural application is basically you [14:10] know as you develop like autonomous agents world  models is how you essentially test it and kind of [14:15] uh at a large scale you can't test the autonomous  agents kind of like in the physical world kind of [14:21] engineering safety you have to test it in the  simulation world u we deploy our product into [14:28] engineering workflows so they have a understanding  of a spatial data like how my part's going to [14:35] behave what is the design of my part and And the  spatial data can come in many form like real the [14:41] geometry of the part. It could be the layout of  your electronics component and so on. So there [14:46] is a spatial data then there is environmental  data. How this part is going to be deployed? Is [14:51] it going to be in 80 surface environment or whatn  not and then they ask a question like okay what [14:56] is going to be the maximum temperature for this  electronics or how this part is going to bend or [15:01] deform. So the input modality is very similar to  what we experience spatial data the physical field [15:09] that is around it and then you ask a questions  like okay help me understand what's happening [15:13] in the part what it replaces is a really expensive  physical experiment because that's how we measure [15:20] we have been dealing with hardware design or  deploying rockets before we had AI how we how [15:25] did we do that like we'll design a part we will  manufacture a part we'll measure it and they're [15:31] like oh that didn't it let's fix it. Right? So the  one way of adding world model into the engineering [15:38] workflow is do that all digitally. Right? And it  should be as accurate as you know measuring the [15:44] temperature sensor in the lab. If you get there  then of course your cycle will be faster and [15:49] so on. So basically how you would do a physical  experiment but those are that's the prompt going [15:54] in and the outcome is what you would measure. It's  how our customers are using the product today. Uh [16:01] I will give a few examples. We're a foundation  research plus software and we're still at the [16:07] very early stages. We're only a year old. So  [laughter] give us a little bit more time. But [16:12] um one of our best partnerships is actually with  Specsy who's also here. Uh they are in the think [16:18] of it as in the drone space and thinking aerial  capture. And so we're able to in partnership with [16:24] him be able to quickly reconstruct and create a  viewing capability to see large cities at scale or [16:33] complex indooroutdoor facilities, a warehouse, a  research lab, um construction site, oil refinery, [16:41] a warehouse, a naval port. So that's starting  point one. So we can do that. We also have [16:47] um SDKs and SDKs and APIs that we're working on  that allow you to then localize within a place and [16:56] one of our partnerships is with Koko Robotics. Uh  we provide a visual positioning system and there's [17:01] a real problem called the lost robot problem.  Okay, what's a lost robot problem? Anyone's ever [17:06] been lost in a city where GPS doesn't work? What  do you do as a human? You look up at the street [17:11] sign. You look at the restaurant names. You double  check Google Maps and you're like, where am I? So [17:18] when last mile delivery robots are trying to get  to where it needs to get to and GPS doesn't work, [17:24] you would use a visual positioning system is what  we provide as an API that then allows Coco or any [17:32] other robot to kind of reorient themselves and  get back on track and do it with a high degree [17:37] of precision. So GPS satellite based is often like  off by 10 meters, even more than that between like [17:46] large skyscrapers and downtown cities. We're  doing that at centimeter levels. You know, [17:51] centimeter is like a huge degree of precision  when you're trying to do last mile delivery. Okay, [17:57] so there's there's quite a bit, right? There's  more than just text, image, and video. There's [18:01] watching the worst case scenarios. There's  simulation, and there's just going directly [18:05] in the with the API. There's like new frontier  stuff too around what I would consider the next [18:10] category of semantic understanding. There's  language semantic understanding but there's [18:14] not actual physics understanding of semantics. So  physics understanding would be like one can you [18:19] segment the objects in the scene and what is an  object? Well some of it and can you do it not with [18:24] labeling each item but can you do it inherently  embedded in the codebase in such a way that can [18:29] be trained and learned and you know the dimensions  and size and depth as one example. Another is like [18:35] change detection in the scene because change over  like intermittent periods or a lot of new research [18:41] will be need to need to be developed for this to  happen. So I I think we're just barely scratching [18:48] the surface. We're literally not even in the first  inning. We're like in the you know first batter [18:53] up. Okay. So speaking of which, I'd love to hear a  little bit more about where we are today. So what [19:00] are the current limitations of world models? Like  for example, heartache. I'm guessing you say we're [19:04] done. We figured out physics and everything else  goes from there. What are you saying for what they [19:08] like where are we today? What can you not do with  world models? Then of course you're hoping to be [19:13] able to do. I mean physics is the first layer. We  think about physics as sort of like a compilation [19:18] layer of hardware, right? You want to design  something whether it's a building or anything [19:22] you have to abide by the laws of physics and  it's you're never going to be able to violate it. [19:27] So you understand physics to me is the starting of  the world model because what we do as humans we've [19:32] been living here we didn't invent physics we just  learned how how to take advantage of it. So we are [19:39] the agents acting in the world that is abiding the  laws of physics and now created we are the agents [19:45] like you said and we are either making decision  creating intelligence system or together creating [19:50] better and better systems. So the from here on  if we can add the understanding of physics into [19:55] a model that starts off a possibility of creating  agents and those agents now will take actions that [20:04] will not violate the physics. So I think it's just  a starting point. We understand that we'll be able [20:10] to do much better um decision making whether it's  training agents to do design, training agents to [20:16] make robots to make decisions in the real world  or at city scale you have various different robots [20:22] and they are pretty much swarming around trying to  create an action will not be able to violate these [20:27] things. What about you Sri Gant? What limitations  are you seeing right now? I think the sim tore gap [20:32] is huge. Um essentially like if the simulation is  perfectly close to reality then I think like you [20:39] know to a large extent like you know you can  train like agents very fast but I think the [20:43] gap is huge. Um and again like a lot of mundane  scenarios the gap is less. The more you get into [20:49] the longtail scenarios the gap is like huge right?  So so the longer the world model is running and [20:54] the more actions are happening the further away  from reality you get right it it ra it ramps up [21:01] over time and you make one decision slightly off.  It's just like robotics control and at some point [21:06] you are so off it kind of doesn't make sense  to come back. Any other limitations that we're [21:11] missing? Yeah, I maybe just building on that is  just the scale of deployment in the real world [21:16] and that's the practice and it's if you're off by  like a millimeter or centimeter it can have or a [21:22] meter it it's all relevant you know depending  on the use case. Uh so combination of data as [21:28] well as scaled deployment and as in with regards  to some of the research you were mentioning what [21:34] what is kind of the biggest hurdle or obstacle  you you're trying to overcome to to move world [21:39] models forward so that they that you know cause  and effect is actually happening how we would [21:44] want really high quality data and I I would say I  welcome any enterprise city organization here in [21:54] the room that wants a partner just to say, "Hey,  you've got some interesting data you'd love for [21:59] us to help you look at differently." I'm open to  doing that because we're still kind of learning [22:05] and testing and building. So that first is that  is being really thoughtful about the quality of [22:11] the data because we're talking about making not  just efficient operations. I mean there these are [22:19] complex processes that have been so finely tuned  with machine learning algorithms and time series [22:25] that it's because it impacts people. It's like  you know some of these uh jobs are incredibly [22:31] intense jobs um in the industrial sector space  in particular and so people's lives are at are [22:37] at stake and you don't want to make mistakes at an  order of degree based on just synthetic data. You [22:44] really need real world data. So, I welcome anyone  who wants to come partner with us if you if you [22:49] just want to even try something. Um, we have a  um an app that's actually available in on the [22:57] Apple Store and Android. It's called Scanverse and  it's a very simple way to start to 3D scan spaces [23:04] as well as things just to get you to practice.  And then within it, you can measure and you can [23:08] localize. And it's just a it starts to change your  mindset about how you're going to start to look at [23:15] the things uh uh not only in the facilities that  you work in, but the way you might start to work. [23:22] What about synthetic data? So, I like I like that  you you flagged that the main thing you need is [23:27] real high quality data. How are you thinking about  synthetic data and are you using it and where is [23:32] its place? Yeah, we it's a great question. I  do think there's a lot of value in synthetic [23:38] data. I think my concern is when training models  start only with synthetic data and continue to [23:42] generate like the next predictive, you know, like  image or scene uh next frame. I it's it's a too [23:49] narrow. It's off. I I do think the synthetic data  that's augmented with real data and then you can [23:57] project and then you reinforce by learning back  into the real data and then generate additional [24:02] synthetic data. I actually think that is the loop  for acceleration. I I mean these two are also may [24:08] have strong opinions as well but it if we can  tighten that I I think the possibilities of the [24:15] way we work and and do things and how things are  built fundamentally changes. What do what do you [24:21] think Artic where do you see kind of the biggest  obstacles and is the synthetic data play a role? [24:25] I mean we definitely use synthetic data but the in  the color in which you use is absolutely important [24:31] like we are training physics models and we don't  use data from customers to deploy our product. So [24:38] there is the essence of synthetic data and how  we have trained our model and yes how we have [24:43] deployed product but the physics that it answers  is real. So we checked the other side is that okay [24:50] it really means at any given inference energy  in is equal to energy out. So that balance is [24:56] performed at the time of inference and checked  is performed produces quality. You take that [25:01] signal back to create a better loop. So the next  iteration of the checkpoint of the model that goes [25:06] out continues to provide the faster inference. So  I would definitely there is a enough data that you [25:12] will start seeing a momentum but you cannot just  add synthetic data without checks about what you [25:19] are trying to predict and some level of spatial  like okay you want to capture all different [25:24] versions of colors and polygons and shapes like  you are talking about for us is different variety [25:29] of physics. make sure we capture all the different  combination of physical field that can happen [25:35] but don't lose sight of the underlying thing that  you're trying to do you keep in that side I think [25:40] we are definitely seeing success that you can  train a model that can be useful for extremely [25:47] complex physical scenarios where the data is not  available outside the bounds of like hardware [25:53] companies semiconductor companies it can be  useful for them but you have to be grounded in the [25:57] product are is noddone using synthetic data. Uh  I actually see synthetic data as uh as an output [26:04] of the world models in some sense especially  in the driving context like you know the world [26:09] models actually generates the videos and stuff  so that we can use it as training data for other [26:16] downstream things. Um I haven't really thought  about using synthetic data to train world models [26:22] because like the biggest gap is a syntoal gap. So  it's more about grounding to real edge scenarios. [26:28] So I think if you use synthetic data, you should  train the world models and then generate synthetic [26:32] data. I'm not sure how that loop might work. Yeah.  Well, and you should try. [laughter] What are what [26:37] are you seeing as as the biggest obstacles  to world models today? Um I think like it's [26:42] mainly really just um kind of grounding with the  reality of all the color cases and the long tail [26:48] like you know that's that's one main thing and  the second one is these models are really big. [26:53] In our scenario, we run AI in real time kind of  on the vehicles. So kind of trying to compress [26:59] them down to like you know like uh like dist them  to uh like a size that can be run kind of a bit [27:05] uh real time. That's also one more uh bit. Okay.  So we we have a definition of world models. We [27:11] have their limitations. I'd love to hear now about  their applications. So let's start again with you [27:17] sir. Can you give us one use case that you see  world models as the best use case today and [27:23] then use case that you're most excited about over  the next few years? So I mean like in a few years [27:29] essentially you'll have robots operating in the  physical world. So how do you know that this robot [27:34] is safe to kind of operate the world where we live  in uh and our kids live in uh it's not possible [27:41] to test them after deployment. The only way to  test them is through simulation world models. So, [27:46] and the other challenge we're talking about is so  I guess each company would develop their simulator [27:52] model and then they would certify the robot then  who's going to certify the simulator right like [27:57] you know so that's that kind of things will pop up  in the future yeah yeah we were talking about that [28:01] backstage I I think it's very interesting because  I think you'll have validators per industry right [28:07] you don't want the same people validating  you know nuclear fusion sites to validating [28:12] self-driving cars so I think you'll have different  organizations and different government agencies [28:16] is validating their specific industries. Um, but  I could also see certain val certain, you know, [28:20] physics based validators that are kind of just the  standard or benchmarks for for world models. Um, [28:25] but Hardik, same question for you. Top use case  today and top use case that you're most excited [28:30] about in the future. Uh, top use case for us  today is definitely in embedding physics in [28:35] the engineering workflow. But I'm going to go to  the other side is what's more exciting. We are I [28:42] believe are going to create a lot more hardware  at an accelerated pace. Why? Because we want to [28:46] embed AI everywhere. Once you start embedding  AI, you have to need a chip to do some level of [28:52] inference whether it's in data center or on the  edge. You're going to have electronics. You're [28:56] going to have all the physical laws in the things  that are behaving. Whether they are complete [29:01] autonomous robots or human guided doesn't matter.  Man-made nature made objects are going to live [29:06] in this world. And we want to understand them at  every level of detail. So what's exciting for us [29:12] is as we grow through these level of technology,  we'll be able to understand it in full detail and [29:18] design systems to create better life for ourself.  Where do we start? We start in semiconductors. So [29:23] that's what we are helping our customers  understand physics faster. But that's the [29:28] part that excites us. And any top use case today  and top use case you're most excited about. Yeah, [29:33] our top use case today is really in industrial  spaces. So I use the um oil refinery example, [29:40] warehouse, logistics, large indoor outdoor complex  spaces where you've got a lot of assets and people [29:47] and it's just complex. Um I think future you  know the in the future I expect there to be [29:56] u more robots than humans. So let me just kind of  envision a future for a second and then the use [30:01] case. I I think we will have more wearable type  devices that are different beyond the phone. I [30:06] think the phone is um we do quite a lot of things  with the phone and it's great and that's why we [30:11] optimize for the phone even for our vision stack.  Uh but I can imagine there's more capabilities [30:18] that then actually allow us as humans to interact  and operate and live in this world in new ways. [30:23] I shared a little bit on on stage that I  love to hike and travel around the world in [30:29] different places including went to the Himalayas  and Everest. Um, and when you experience things in [30:35] the real world, you want to be able to capture  it in ways and memorize it and remember it but [30:41] interact with the world and be present but also  have like, you know, enablement and support in [30:47] different ways to make that experience great.  So that's my point. I I think every company [30:52] has the opportunity and every individual has an  opportunity to play in this new new world. Can [30:57] you give an a specific example of of maybe your  favorite or any an enterprise customer or just [31:03] a a project that is you were working on or you've  already deployed and what were the results of the [31:08] world model? What was the ROI there? Well, oh,  go for it. Oh, well, one of the use cases we're [31:17] um proving out was in oil refineries because  think about if someone said go find the needle [31:24] in the hay stack in in Vegas and we had one of the  scenarios was reconstructing an or refinery space [31:32] that's quite large and enabling field workers that  are going to maintenance and turn over blinds. [31:40] lines are very specific targeted areas where  they're turning on and off um uh the pipe systems [31:46] to to replace and update and do maintenance fix.  Now, if you're hired because you happen to be [31:53] a specialist on X system, typically the way the  person navigates is by looking at a paper, maybe [31:59] they have a digital twin of some map and they have  some rough estimates on the direction they should [32:04] be going. Uh on average it takes somewhere on the  order of like 10 minutes if they know the facility [32:10] quite well because they've worked there for 20  years or it can take upwards of 2 hours if it's a [32:16] new worker doesn't know the space because they're  trying to we can actually not only create the [32:21] space localize it on your mobile device and allow  you to navigate to to that space in two minutes. [32:27] So the economics are very different depending on  how you decide you want to change a number of the [32:35] ways you work. But just that orientation of being  able to get to what you need to get to to do the [32:41] work you need to do is also huge. Can you give an  example of a specific project you deployed or a [32:47] customer getting actual ROI from a world model?  Yeah, we're very similar like needle and haste. [32:52] So I'll say like if you look at any memory or a  chip you have nanometer level wires pretty much [32:58] like plumbing or roads on a centimeter scale. It's  sort of hard for us to kind of imagine what it is. [33:04] It's like resolving human hair on the map of New  York. So that's how complex the part is inside [33:10] every one of our phone. So our customers are  interested in like can we identify where we want [33:15] to change this design pretty much like changing  the plumbing of heat transfer or electronic signal [33:22] transfer so they can get a better performance on  it. The memory footprint is large as gigabytes of [33:27] geometric or spatial data and their ask was can  you help us identify where the bottleneck is so [33:33] we can do the design iteration faster. Today  it takes hundreds if not more engineers with [33:40] ours because you cannot synthesize all of it  in head. In one person's head it's that much [33:44] data and you end up making approximations and  it it's a longer analysis. With our product [33:51] uh our customers have been able to identify make  design changes perform this analysis in matter of [33:58] days what otherwise would have taken supercomput  if at all or they would have made approximations [34:04] and found out the issue in physical experiments.  So the ROI again is is harder to measure in terms [34:10] of how much tokens did we consume. It's more about  helping our customers ship better parts faster by [34:18] enabling them to create better physics decision.  Physics intelligence is accessible. So very much [34:24] the same scale but in in absolute scale at a  different place. same level of data but now the [34:30] users are making decisions of where do I change  stuff so I get a better thought and I know that [34:36] you're still at more of the research stage there  isn't ROI necessarily but where do you see the ROI [34:41] and where or is there a project that you deployed  I walk us through the you know the input and the [34:45] output there sure so essentially our core product  is making the fleet safer and smarter so in terms [34:51] of the core product like we see like uh kind of  the accidents reducing by 50% 60% and some of [34:57] the unsafe behaviors uh like while texting while  driving and so on are not wearing seat belt those [35:03] go down by like 90% and so on so that even if  there's an accident like you know people kind of [35:09] uh don't face uh too many injuries and so on so  that business generates data and the world models [35:14] train on this data exhaust um so but I think  like ROI would come as the world goes more into [35:20] autonomous agents and like self-driving vehicles  and so so I think that helped those all those [35:25] examples kind of help the audience think about  the applications of world models. Another way of [35:29] asking the same question really is if you when you  guys were training these world models and you've [35:37] been interacting with them, was there any point  that you saw is like where you were surprised by [35:41] the intuition or the understanding of these world  models if you can think back through all the all [35:45] the time you spent with them. Was there a specific  time where you you know you saw it do something [35:49] that you didn't expect it to know? I mean in  when we are training physics model often we start [35:56] thinking by oh we'll have to train it for certain  type of parts and then when we go to electronics [36:02] or mechanical parts we'll have to completely start  from a different starting point and we'll have to [36:07] do it. What has been surprising to us is um that  the world we live in is multifysics. So I would [36:14] say we have created arbitrary discipline that you  should study thermal and I'll study electrical. [36:18] That's not how the world is. world is multifysics  all at the same time. So when we started adding [36:24] more physical modality into a single model in  some ways it started learning faster and the [36:31] generalization was brought very easy to say is  not the insight we had before. So I would say [36:37] that level of generalization that it can really  understand it once you embed the physics in it. [36:42] Um and the multifysics model is in some ways is  better generalized is definitely a surprising [36:50] happy uh um insight that we got. Either of  you Amy or Sagan did you have any surprises? [36:57] So the driving context so kind of uh so our first  W model that we trained uh it would go into an [37:03] accident and then it would just keep driving  as if nothing has happened right and I'm like [37:09] oh like what happened and to his point there's lot  of like AI slot video in some sense lot of driving [37:15] data is just mundane doesn't have anything kind  of interesting and then we realize oh okay we need [37:20] to actually throw in the accident data and the  nearness data for the word models to understand [37:25] like you know that like an after often an accident  like you just don't keep driving because most [37:29] of the time there's not an accident so it was  indexing on that but it needs to index on the [37:33] accident because that's what you're focused on  right at the same point of time like the vehicle [37:36] stays towards the right uh like where there's a  house uh it hasn't really indexed on the house [37:43] um uh like driving towards the house so it  would just kind of start dreaming about like [37:48] uh about a highway after that right so like these  are the things that you have to keep fixing any [37:53] any surprises yeah probably the best surprise  has been uh we don't we because the models were [38:01] created actually starting with what I would say  the uh worst data possible which is mobile device [38:08] data on poor quality phones and data that might be  captured in the wrong places and wrong distances [38:15] and we actually discovered unique algorithms to  recreate scenes that are geometrically accurate [38:24] physics referenced uh in terms terms of certain  positions like six degrees of freedom is the [38:29] orientation of how we think about position and and  views of things. So, and then fusing ground data [38:35] with satellite data and aerial drone data, we also  learned that the model can actually reorient just [38:42] as fast. And then the last piece we're um thinking  about, we're working in research right now. We [38:48] don't have any product because it's research. So,  you have to wait until we technically engineer it [38:52] and then can put it into a format. But we're I'm  super excited much like in the language world of [39:00] of embed embedding and encoding we are actually  doing that with data about space. Uh and what [39:09] that ultimately means that we could do it cheaper,  faster, better with less data. And it means you [39:16] you can use you're going to be um we're going  to be able to do things that were not possible [39:21] before around that spatial intelligence layer.  Okay. So, we're nearing the end already. So, [39:26] just there might be a couple more questions. I'd  love for you to hear to let me think about this. [39:33] Could you actually let's say there in the audience  there are individuals and companies looking to [39:39] start applying world models? What is the first  step they should take? So I think mainly like [39:46] uh so the models are getting kind of cheaper  and like uh like standardized but I think like [39:52] like your differentiating would be uh like data  so understand like your environment understand [39:58] your data like curate the data well I think like  focus like a lot on data and the second thing is [40:03] like evals uh understand quantify as much as  possible what you're trying to accomplish and [40:10] uh set up evals around it uh and then world models  can kind of like help you with that. Um I'll start [40:17] from like what is it that you're trying to  do like if you're thinking about world model [40:21] you're trying to do something in the physical  world whether it is to design a part or deploy [40:25] a part or make actions on some other robots or  whatnot. So start back from there and that will [40:31] help you understand before data at least in my  opinion evaluation like how would you evaluate [40:36] this to be a good product or a good system and  that's when you are going to start discovering [40:40] the type of intelligence you want to add in  some of it is going to be I want to care about [40:46] uh traction forces or you know thermal problems  you some of it is going to be have I taken into [40:51] account all the spatial stuff so that's when the I  would say that tacking back from reverse from that [40:55] angle you're going to figure out which modality  is important to for your product and where you [41:00] can embed that intelligence. Absolutely. I don't  think there is one product that takes care of all [41:04] of it. It's going to come in stacks because we are  in the early days of that understanding and then [41:10] you a enterprise can figure out a workflow for  themselves like how do we create that system and [41:15] ship it faster in the modality that matters the  most. Yeah. Start from the use case. I mean the [41:22] oil refinery use case was a simple one. I mean the  average cost uh just a turnover per day cost is [41:29] like almost $10 million. If the blinding process  i.e. finding that needle I was talking about is [41:35] 20% of that process every day. Okay. Um and all  of a sudden we're able to find one of one of the [41:43] um uh essentially technician said if I had  to find a needle on a in a stack of needles, [41:50] this is the way for me to do it. which is a huge  compliment uh because we were able to reduce that [41:56] time frame by 80%. So we don't know what we don't  know and I say that as an invitation actually is [42:04] if you have really unique 2D data and or 3D uh  videos uh you don't need 3D but 2 2D data images [42:15] andor videos around your operations facility  sites that are physical uh and you want some [42:24] way of starting to understand that area potenti  potentially plan in that area, simulate in that [42:30] area or operate in that area, then I I would say,  you know, do a bit of research. We're welcome [42:36] uh to have any of you come and say, "Hey, we've  got some unique data. Would you mind just testing [42:41] if this might even be possible?" We really did  not know that data that we had captured in the air [42:47] could actually work for data that is underwater. I  mean, that is just think about what that implies, [42:53] right? So, we don't know and we're we're  thoughtful about it. We're thinking about um your [42:59] ability to bring your own data protecting you and  your data sets as well and how also to potentially [43:04] this is the last piece like containerize the model  in some way or create some sub formats of it so [43:10] you could also have it in protected environments.  Okay, we're just over time so let's keep this last [43:16] one very very succinct. Uh obviously we've  talked about various use cases. We've talked [43:22] about oil and gas. We've talked about hardware.  We're talking about driving 10 uh 10 years from [43:29] now. Pick one industry that you think will be  completely transformed by world models. Going [43:38] how we design deploy every hardware part one  industry. [laughter] I would love it if every city [43:47] planning every [laughter] city in the world would  like have better infrastructure. Awesome. So that [43:55] would be ideal. Okay. So self-driving hardware  and city infrastructure. Thank you so much.