Waymo Distills Giant AI Models to Scale Driverless Fleets Safely

Waymo蒸馏巨型AI模型，安全扩展无人驾驶车队规模 Waymo通过将庞大的AI基础模型蒸馏为轻量级学生模型来扩展其自动驾驶车队，使模型能够在严格的车载延迟限制下实时处理多模态传感器数据。 Srikanth Thirumalai阐述了Waymo如何利用其第5代多模态传感器套件与Waymo基础模型，应对现实世界中具身智能面临的挑战。他详细介绍了公司的双路径视觉-语言方法以及教师-学生蒸馏流程，该流程可将庞大模型转化为低延迟车载系统组件和云端仿真器。演讲重点介绍了Waymo如何利用严格的仿真测试平台对安全性进行数学量化，并通过显著降低事故率的指标与现实极端边缘案例，证明该系统已具备大规模落地的成熟度。 Waymo 自动驾驶里程在全美 5 座城市已突破 1 亿英里；其第 5 代硬件搭载 29 个摄像头、5 个 LiDAR、6 个雷达与 4 个音频接收器，用以解决具身 AI 在多变路况下的亚秒级实时决策挑战。 (0:28) Waymo 基础模型采用编码器-解码器架构并集成驾驶 VLM，通过“快慢思考”双路径范式，将多传感器时空数据与高级逻辑推理相结合，直接输出底层控制动作。 (4:55) 该模型被微调为驾驶、仿真器和评估器三大 Teacher 模型，并通过知识蒸馏压缩为轻量级 Student 模型，从而平衡车载实时低延迟执行与云端大规模仿真成本。 (11:02) 通过量化残留风险的安全就绪框架，Waymo 在 7100 万英里无人驾驶数据中验证了系统可靠性，相比人类驾驶员将严重伤亡事故减少了 88%，总体人伤事故减少了 78%。 (15:26)

Waymo head of onboard engineering Srikanth Thirumalai argues that deploying safe autonomous driving at scale requires more than standalone AI algorithms; it demands an integrated ecosystem combining multi-modal sensing, domain-adapted foundation models, knowledge distillation, and mathematical risk quantification. Embodied AI and Multi-Modal Perception (0:28) Addressing a global crisis where over one million people die annually in road accidents—primarily due to human error—Waymo operates "embodied AI" directly inside its vehicles. Thirumalai notes that unlike cloud-based generative AI, which can take seconds to respond, autonomous driving software must navigate messy, safety-critical environments and compute decisions within milliseconds. It must handle unpredictable edge cases ranging from toddlers and stray cyclists to falling truck debris, fallen trees, flooded roads, and distracted human drivers. To perceive this environment, Waymo's fifth-generation vehicle pairs AI with 29 cameras, five LiDARs, six radars, and four audio receivers to build a real-time 3D view. Thirumalai illustrates the value of this multi-modal suite through several real-world encounters: braking safely for two dogs and two humans chasing a ball into the street; using LiDAR in pitch darkness to spot two pedestrians jumping over a concrete K-rail partition when cameras and the human eye could not see them; and detecting the sparse LiDAR reflection of a pedestrian underneath an occluding San Francisco Muni bus to predict their movement and slow down proactively. Adapting Foundation Models for Physical Control (4:55) While standard Vision-Language Models (VLMs) offer broad world knowledge and chain-of-thought reasoning—such as deciphering a complex street sign to confirm parking is permitted for one hour at 4:00 PM on a Wednesday—they cannot process raw LiDAR or radar inputs, output low-level steering and braking controls, or meet real-time onboard latency requirements. To overcome these limits, Waymo developed the Waymo Foundation Model using an encoder-decoder architecture. A sensor fusion encoder converts spatial and temporal data from cameras, LiDAR, and radar into tokenized embeddings, which an action decoder translates into physical driving sequences while predicting the behavior of surrounding road agents. Alongside this network, an integrated driving VLM operates in a "thinking fast and slow" dual-path paradigm, providing high-level reasoning in complex situations—such as recognizing a stalled vehicle with hazard lights as being in distress and deciding to safely nudge around it rather than wait indefinitely. Distilling Teacher Models for Car and Cloud (11:02) Thirumalai emphasizes that scaling autonomous driving requires not only a driver, but also a simulator to test edge cases and a critic to evaluate performance. Waymo fine-tunes its foundation model into three specialized, high-capacity "Teacher Models." The Driver Teacher uses behavioral cloning and reinforcement learning on human demonstrations and curated logs from 100 million autonomous miles to generate smooth action sequences. The Simulator Teacher acts as a closed-loop world model to generate challenging scenarios like near-misses, highway spin-outs, and severe weather, while the Evaluator (Critic) Teacher uses supervised fine-tuning on human feedback, internal documentation, and bug reports to flag improper driving. Because these massive teacher models are too computationally heavy to run in real time inside a car or across hundreds of millions of simulated cloud miles, Waymo relies on knowledge distillation. The company compresses the teachers into compact "Student Models" optimized to meet strict onboard latency constraints and off-board simulation budgets. Quantifying Residual Risk and Real-World Safety (15:26) Having begun as a Google project 15 years ago before spinning out into an independent company eight years ago, Waymo has logged over 100 million fully driverless miles on public roads. It currently serves paying customers in San Francisco, Los Angeles, Phoenix, Austin, and Atlanta, with further expansion planned next year alongside international and extreme-weather testing in heavy snow, torrential rain, and Las Vegas desert heat. Before deploying any software release, Waymo uses a data-science readiness framework developed over more than a decade to mathematically quantify "residual risk." By running software candidates through standardized simulation test beds and extrapolating failure rates to real-world frequencies, Waymo evaluates collision and near-miss probabilities, improper stops, strandings, software crashes, cybersecurity vulnerabilities, and passenger comfort. Across 71 million evaluated driverless miles, Thirumalai reports that the system achieved an 88% reduction in crashes with serious injuries, a 79% reduction in airbag-deployment crashes, and a 78% reduction in overall injury-causing crashes compared to human drivers. Thirumalai concludes that because human lives are at stake on public roads, embodied AI cannot rely solely on scaling raw compute and data. Proving that an autonomous driver is both fundamentally safe and commercially scalable requires unifying foundation models, domain-specific adaptation, cost-efficient student distillation, and continuous mathematical safety validation into a single operational pipeline.

Waymo 车载工程负责人 Srikanth Thirumalai 指出，实现规模化全自动驾驶的关键在于构建从基础模型训练、教师-学生模型蒸馏到严谨数学化验证的完整工程体系。Waymo 依托多模态感知与海量真实行驶数据，旨在以超越人类水平的安全性解决全球高发且主要由人为失误导致的道路伤亡问题。 具身智能的现实挑战与多模态感知布局 (0:28) Thirumalai 表示，与允许数秒响应时间的云端生成式 AI 不同，直接部署在车端的“具身智能”（Embodied AI）必须在毫秒级延迟内应对混乱且高度不可预测的物理世界。行驶环境充斥着路面落物、脱缰儿童、倒塌树木及注意力分散的人类司机等极端边缘案例，任何系统微小失误都可能带来致命后果。为在严苛时间窗口内实现全方位感知，Waymo 第五代硬件系统配备了包含 29 个摄像头、5 个激光雷达（LiDAR）、6 个毫米波雷达和 4 个音频接收器的多模态硬件架构。 该传感器组合能够实时构建连续的三维全景图景。在实际案例中，多模态阵列不仅成功在两只狗追逐皮球冲入街道并伴随行人追赶的突发场景下实现安全平稳制动，还能在暗夜与视觉盲区发挥关键作用。例如，激光雷达系统甚至能够识别大型车辆底部的微弱反射信号，提前预测被完全遮挡的行人走向并主动减速避让，克服了单纯依赖人类视觉或普通摄像头易受光线和死角阻隔的技术瓶颈。 基础模型架构与快慢双路径决策融合 (4:55) 面对长尾复杂路况，Waymo 开发了编码器-解码器架构的“Waymo 基础模型”（Waymo Foundation Model）。Thirumalai 解释称，通用视觉语言模型（VLM）虽具备强大的常识逻辑（如理解特定时段的复杂停车标志），但无法直接解析原始激光雷达或毫米波雷达的点云数据，亦不能直接输出底层方向盘与油门制动指令，且运算延迟无法满足实时行车需求。为此，Waymo 基础模型的编码器负责将多传感器时空数据融合为标记化嵌入表示，动作解码器则将这些特征转化为具体的物理轨迹，并同步预测周边交通参与者的行为路径。 为兼顾深层常识理解与即时行车反应，该系统构建了类似人类“快思考与慢思考”的双路径集成机制。驱动型 VLM 与传感器融合网络协同运作，在上层专注于解析高语义复杂场景。例如，当检测到前方车辆开启双闪灯停滞时，驱动型 VLM 能够推断出该车处于抛锚故障状态，从而指挥车辆主动借道绕行，避免机械地在障碍车后方无限期等待，显著提升了自动驾驶在非结构化场景下的决策合理性。 教师-学生模型蒸馏与大规模闭环仿真 (11:02) 由于超大规模基础模型无法直接在车载硬件上以极低延迟运行，Waymo 建立了由三大特化“教师模型”向轻量“学生模型”迁移的知识蒸馏流水线。三大教师模型分工明确：“驾驶员教师”（Driver Teacher）基于人类示范与 1 亿英里全无人行驶日志进行微调，结合行为克隆与强化学习生成安全顺畅的行驶轨迹；“仿真器教师”（Simulator Teacher）作为闭环世界模型，模拟真实世界的动态演进，生成险情逼近、失控打滑与恶劣天气等合成极限场景；“评估者教师”（Evaluator Teacher）则结合标注的人类反馈与缺陷报告，自动审查并标记不当驾驶行为。 鉴于教师模型计算开销极为庞大，无法直接部署于实车或在云端运行数亿英里仿真测试，Waymo 采用知识蒸馏技术，将其核心推理能力压缩至更紧凑的学生模型中。Thirumalai 指出，这种架构既保障了车端学生模型满足亚秒级的严格延迟指标，也使得云端大规模安全模拟能够在受控预算内运转，构筑起从海量离线训练到高效在线执行的技术闭环。 数据驱动的安全验证框架与真实降险成效 (15:26) 为了客观论证软件版本的实际上路安全性，Waymo 放弃了主观判断，依托一套历时十余年构建的数据科学安全就绪框架。该框架通过量化模型评估每次软件更新的“残留风险”，监测维度涵盖碰撞与险情发生频率、异常停滞或软硬件宕机、网络安全漏洞以及乘客平稳舒适度。Waymo 将候选版本置于标准化仿真基准中运行，并将虚拟环境中的失效率推导至真实行驶频率，以此决定软件是否达到发布门槛。 Thirumalai 披露的统计数据表明，在累计超过 7100 万英里的全无人真实运营里程中，Waymo Driver 的安全表现显著优于人类驾驶员基准：导致严重人身伤害的事故率降低了 88%，安全气囊弹出的严重碰撞事故降低了 79%，整体伤害性交通事故发生率降低了 78%。从在极暗环境下依靠激光雷达识别越过混凝土隔离栏的行人，到在复杂路口规避从遮挡公交车后方走出的行人，实测数据与场景印证了硬件冗余与预测算法在消除人身伤亡方面的实际效益。 Srikanth Thirumalai 总结强调，自动驾驶具身智能的核心使命在于挽救生命并赢得公众信任，这需要依赖基础模型训练、轻量化车端蒸馏与定量安全评估构成的全链路生态。目前 Waymo 已在全美五大主要城市开展商业运营并推进全天候及跨国测试，但其能否在更复杂的极端气候与不可预知的全球道路环境中持续稳定运行，仍是其全面扩大商业化落地时面临的长期验证课题。

## Background 该视频由 YouTube 频道 Ai4 于 2025 年 8 月 24 日发布，主讲人为 Waymo 车载工程副总裁 Srikanth Thirumalai。视频简介指出，本场演讲深入展示了支撑 Waymo 快速增长的底层技术，阐述如何通过多模态感知、先进 AI 模型与前沿仿真技术，打造超越人类水准且值得信赖的自动驾驶系统。 ## TL;DR Waymo 工程副总裁 Srikanth Thirumalai 深入剖析了 Waymo 达成超过 1 亿英里完全无人驾驶里程背后的技术架构。针对具身智能在现实世界中面临的高混乱度与毫秒级时延挑战，Waymo 研发了结合快慢思考机制的 Foundation Model，融合多模态传感器网络与驾驶专用 VLM。为了在车端与云端低成本高效运行，团队通过知识蒸馏将基础模型提炼为驾驶员、模拟器与评估器三大教师-学生体系。结合十余年打磨的量化安全就绪度框架，Waymo 在 7,100 万英里的实际运营中实现了严重人身伤害事故率降低 88% 的卓越安全表现，验证了其规模化落地的可行性。 ## Atmosphere & style 整场演讲去除了繁复的客套与宣传包装，直奔硬核工程技术与底层系统架构展开，透着顶级科技巨头工程师沉稳、严谨且务实的叙事基调。演讲者以客观理性的数据开篇，随后层层递进地剖析模型拓扑与工程挑战，在穿插展示夜间行人避障与公交车盲区微弱激光雷达点云预判等震撼现实录像时，情绪从严谨推演自然升华至对具身智能关乎真实生命责任的敬畏。听众能够深刻感受到团队在面对极高安全门槛时，那种以深厚数据科学与量化指标为支撑的笃定与底气。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=4s) Waymo 在公共道路上的完全无人驾驶里程已突破 1 亿英里，并在旧金山、洛杉矶、凤凰城、奥斯汀和亚特兰大五大城市实现了常态化商业运营。 - [02:30](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=150s) 自动驾驶作为物理世界交互的具身智能（Embodied AI），面临不可预测的混乱现实环境、安全攸关容错率极低，以及决策必须在毫秒级内完成的三大核心挑战。 - [02:30](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=150s) Waymo 第五代硬件系统配备 29 个摄像头、5 个激光雷达、6 个毫米波雷达与 4 个音频传感器，构建全方位高精度的 3D 多模态环境感知。 - [04:55](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=295s) 通用视觉语言模型（VLM）具备常识推理优势，但因缺乏激光雷达/雷达模态、缺少底层物理控制映射且原生推理延迟过高，无法直接套用于自动驾驶。 - [08:19](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=499s) Waymo 基础模型采用“快慢思考”双通道架构：多模态传感器融合网络处理高频时空信息，驾驶专用 VLM 负责复杂语义与常识推理。 - [08:19](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=499s) 基于 Transformer 的动作解码器融合快慢双路径 Token，同步输出底层的车辆执行序列以及对其他交通参与者的运动轨迹预测。 - [11:02](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=662s) 通过知识蒸馏技术，Waymo 将庞大的驾驶员、模拟器与评估器三大教师模型压缩为轻量级学生模型，以兼顾车端低延迟运行与云端百亿英里级超大规模仿真。 - [11:02](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=662s) 模拟器教师模型（Simulator Teacher）充当高保真物理世界模型，利用闭环训练生成车祸险情、极端恶劣天气等长尾危险场景。 - [15:16](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=916s) 安全就绪度评估框架以“量化残余安全风险”为核心，利用数据科学方法将在高保真仿真中捕获的不良驾驶事件发生率外推至真实道路风险区间。 - [18:18](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1098s) 7,100 万英里真实道路行驶数据表明，Waymo Driver 相比人类基线将严重人身伤害事故降低 88%，总体致伤事故降低 78%。 - [20:25](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1225s) 激光雷达感知展现出超越人类视觉的极限能力，能有效应对无照明夜间突发翻越隔离带行人，或通过巴士底部稀疏反射点云提前化解盲区“鬼探头”。 - [20:25](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1225s) 实现超级人类驾驶水准必须依赖全链路闭环工程体系：涵盖基础模型研发、驾驶场景适配微调、学生模型蒸馏与全流程量化安全验证。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=4s) 自动驾驶的使命与 Waymo 的规模化运营进展 Waymo 车载工程团队负责人 Srikanth Thirumalai 开场直接切入主题，省去了宣传片播放，将时间留给核心技术干货。他指出，自动驾驶之所以是一项紧迫的事业，源于一个令人痛心的数据：全球每年有超过 100 万人因道路交通事故丧生，而绝大多数事故完全是由人为失误造成的。15 年前，Waymo 作为 Google 自动驾驶项目起步，并在大约 8 年前独立拆分为子公司，始终秉承着“打造全球最值得信赖的自动驾驶系统”（World's most trusted autonomous driver）这一宏大愿景。 ```mermaid flowchart LR accTitle: Waymo 商业化落地与规模化版图 subgraph Operations["商业运营城市 (Paying Service)"] SF["旧金山 (SF)"] LA["洛杉矶 (LA)"] PHX["凤凰城 (Phoenix)"] ATX["奥斯汀 (Austin)"] ATL["亚特兰大 (Atlanta)"] end subgraph Testing["极端工况与拓展测试 (Global Testing)"] Snow["暴雪极寒"] Heat["沙漠干热 (拉斯维加斯等)"] Rain["暴雨洪涝"] Global["国际化新城市"] end Operations --> Expansion["计划明年拓展至更多城市"] Testing --> Scale["超过 1 亿英里全无人驾驶里程"] ``` 目前，Waymo Driver 在公共道路上的完全无人驾驶（Driverless）里程已突破 1 亿英里大关。全付费乘客服务已经在旧金山、洛杉矶、凤凰城、奥斯汀和亚特兰大五大城市常态化运营，并计划在下一年拓展至更多城市。与此同时，Waymo 正将测试拓展到包括海外在内的更广泛区域，直面拉斯维加斯酷热、极端暴雪以及倾盆暴雨等严苛天气的考验。 - **核心里程碑**：公共道路完全无人驾驶里程已超 1 亿英里（100+ million miles）。 - **已落地商业运营城市**：旧金山（SF）、洛杉矶（LA）、凤凰城（Phoenix）、奥斯汀（Austin）、亚特兰大（Atlanta）。 - **极端气候拓展**：暴雪、极端干热、强降雨及国际化多城市测试。 --- ### [02:30](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=150s) 具身智能的现实挑战与第五代硬件感知系统 自动驾驶不同于云端运行的传统生成式 AI，它属于在现实物理世界中直接交互的“具身智能”（Embodied AI），面临着三大本质难题： 1. **现实环境极度混乱且不可预测**：街道充满突发状况，例如乱窜的骑行者、脱离看护的幼儿、货车掉落的货物碎片、倒伏的树木、积水路段及分心的司机。 2. **安全攸关（Safety-Critical）**：系统的决策容错率极低，犯错成本极其高昂。 3. **极苛刻的实时性要求**：云端大模型可以消耗数秒生成回答，而车端具身模型必须在“数毫秒”（milliseconds）之内做出攸关生死的操控决策。 为了应对真实世界的复杂性，Waymo 采用了多模态传感器系统。当前的第五代硬件套件（5th-gen car）装备了 29 个摄像头、5 个激光雷达（LiDAR）、6 个毫米波雷达（Radar）和 4 个音频传感器，共同构建出车辆周围精准而立体的 3D 环境感知。现场展示了一段极具挑战性的街景视频：两只狗在马路中央追逐一颗球，后面还有两个人紧追着狗；系统通过激光雷达与摄像头的多模态感知，迅速捕获动态异常并指挥车辆稳稳停下。 ```mermaid flowchart TD accTitle: Waymo 第五代多模态感知硬件配置 subgraph Sensors["第五代感知硬件架构"] CAM["29 个摄像头 (Cameras)"] LID["5 个激光雷达 (LiDARs)"] RAD["6 个毫米波雷达 (Radars)"] AUD["4 个音频接收器 (Audio Receivers)"] end Sensors --> 3D["全方位精准 3D 动态感知"] 3D --> Safety["毫秒级实时安全避障决策"] ``` - **第五代硬件配置清单**：29 个高分辨率摄像头、5 个激光雷达、6 个毫米波雷达、4 个音频接收传感器。 - **具身 AI 决策时延**：毫秒级响应，区别于秒级响应的云端生成式大模型。 --- ### [04:55](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=295s) 视觉语言模型（VLM）在自动驾驶中的潜力与局限 行业内常有一种观点认为，只需遵循 AI 的“苦涩教训”（The Bitter Lesson），直接借助前沿大模型并灌入海量数据，自动驾驶问题就能迎刃而解。但事实远比想象复杂。当前爆火的视觉语言模型（VLM）通过互联网海量数据预训练，具备广泛的世界知识与思维链（Chain of Thought）推理能力，在解析复杂语义（例如看懂极其繁杂的停车指示牌并判断“周三下午 4 点能否停车 1 小时”）上展现出惊人能力。 然而，现成的商业 VLM 根本无法直接用于自动驾驶，必须经过深度适配与微调： - **感知模态缺失**：通用 VLM 仅接受图像和文本，未曾融合激光雷达点云与毫米波雷达信号。 - **缺乏底层控制映射**：模型无法直接将感知环境转化为车辆可执行的底层动作序列（如加速度、转向角与制动曲线）。 - **实时性能不足**：原生长文本/长视频推理延迟过高，无法满足实时行车决策。 --- ### [08:19](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=499s) Waymo Foundation Model 架构：快慢思考结合的端到端体系 为了攻克这些瓶颈，Waymo 构建了专用的“Waymo 基础模型”（Waymo Foundation Model）。该模型采用 Encoder-Decoder 体系，融合了多模态感知网络与专门调优的自动驾驶 VLM，形成了类似于人类大脑“快思考与慢思考”（Thinking Fast and Slow）的双通道架构。 ```mermaid flowchart TD accTitle: Waymo Foundation Model 双通道架构 subgraph Inputs["多模态输入源"] C["摄像头视频流"] L["LiDAR 点云"] R["Radar 雷达信号"] end subgraph DualPath["快慢思考处理通道"] SFN["传感器融合网络 (Sensor Fusion Network)\n[多模态编码器 / 快思考]"] DVLM["驾驶专用 VLM (Driving VLM)\n[高级语义推理 / 慢思考]"] end subgraph Output["动作生成与交互预测"] DEC["动作解码器 (Action Decoder)\n[Transformer 架构]"] ACT["底层执行序列 (转向 / 制动 / 加速)"] PRED["道路他车与行人行为预测"] end Inputs --> SFN Inputs --> DVLM SFN -- "时空 Token 嵌入" --> DEC DVLM -- "高级推理 Token" --> DEC DEC --> ACT DEC --> PRED ``` 在上方路径（快思考）中，多模态传感器融合网络（Sensor Fusion Network）对摄像头、激光雷达和毫米波雷达的跨时间流信息进行融合编码，输出高频的 Token 嵌入表示；在下方路径（慢思考）中，驾驶专用 VLM 实时解析复杂语义情境（例如前方停靠打着双闪的故障车，VLM 能推理出“该车辆抛锚受困，应当借道绕行而非死等”）。两大分支的 Token 最终汇入基于 Transformer 的动作解码器（Action Decoder），同时输出车辆动作控制序列以及对道路上其他交通参与者的行为轨迹预测。 - **双系统协同**：快路径（时空传感器融合）保证毫秒级低延迟响应；慢路径（Driving VLM）负责高层级语义常识与因果推理。 - **输出能力**：高精度车辆纵横向运动控制、他车及弱势道路使用者（VRU）行为预测。 --- ### [11:02](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=662s) 教师-学生大模型蒸馏体系：驾驶员、模拟器与评估器 基础模型虽然强大，但体积过大，无法直接在车端算力平台或云端数百亿英里的仿真测试中运行。Waymo 将基础模型适配并微调为三大高精度“教师模型”（Teacher Models），随后利用知识蒸馏（Knowledge Distillation）技术压缩为小巧高效的“学生模型”（Student Models），以满足实车运行与海量云端评测对延迟和成本的苛刻要求。 ```mermaid flowchart TD accTitle: 教师模型微调与学生模型蒸馏架构 WFM["Waymo 基础模型 (WFM)"] subgraph Teachers["三大教师模型体系 (Teacher Models)"] DT["驾驶员教师 (Driver Teacher)\n• 人类示范 & 1亿英里错误修正微调\n• 行为克隆 & 强化学习"] ST["模拟器教师 (Simulator Teacher)\n• 世界模型预测时空演变\n• 闭环训练 & 危险边缘场景生成"] ET["评估器教师 (Evaluation/Critic Teacher)\n• 监督微调检测危险驾驶行为\n• 结合内部 Bug 库与文档认知"] end subgraph Students["学生模型落地 (Student Models)"] OS["车载端部署 (Onboard Latency)"] CS["云端超大规模仿真 (Cloud Simulation)"] end WFM --> DT & ST & ET DT & ST & ET -- "知识蒸馏 (Distillation)" --> OS & CS ``` 1. **驾驶员教师模型（Driver Teacher）**：以生成安全、合规、平顺舒适的行车动作序列为目标。使用海量人类驾驶示范数据进行微调，并融合了以往 1 亿英里无人驾驶里程中被算法与人工标记的错误工况修正数据，在后训练（Post-training）阶段结合了行为克隆（Behavior Cloning）与强化学习（RL）。 2. **模拟器教师模型（Simulator Teacher）**：作为高保真“世界模型”（World Model），任务是准确预测物理世界在每一个时间步的演进。它负责生成逼真多元的危险工况，包括事故、险情（Near misses）、恶劣天气、复杂路口甚至是高速公路上失控打滑的车辆，采用闭环训练（Closed-loop training）和强化学习进行优化。 3. **评估器/裁判教师模型（Critic / Evaluation Teacher）**：专注于分析、理解和自动捕捉驾驶日志中可能表征不当驾驶的异常事件。模型结合了人类标注数据以及 Waymo 内部的海量技术文档与 Bug 库，通过监督微调（SFT）精准识别行车瑕疵，判断系统在面对特定场景时原本应采取的正确动作。 --- ### [15:16](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=916s) 安全就绪度评估框架与量化残余风险 仅仅拥有强大的 AI 模型还不足以支撑安全的大规模扩展，Waymo 建立了一套历经十余年打磨的“安全就绪度框架”（Readiness and Safety Framework）。系统每一次软件发布与更新，都必须通过该流程对其上路就绪状态进行严格量化。 Thirumalai 强调，“量化”（Quantify）是该体系的核心。工程团队不能停留在定性分析，而是必须明确在特定软件版本中系统蕴含的“残余安全风险”（Residual safety risk）到底有多少： - **明确评估标的**：系统不仅排查碰撞事故（Collisions）、险情、不当停车与死锁停滞（Strandings），还会量化软件崩溃、安全漏洞以及整体驾乘品质。 - **构建高保真仿真数据集**：设计极具代表性的综合仿真驾驶场景，在云端大规模回放与变异测试。 - **前沿数据科学外推**：借助深厚的数据科学方法与专家经验，将在仿真环境中捕获的不良驾驶事件发生率，科学外推至现实道路运营规模下可能出现的真实概率区间。 --- ### [18:18](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1098s) 真实道路安全数据与极端危险场景避险实录 基于 7,100 万英里（71 million miles）的真实道路行驶数据比对分析，Waymo Driver 的安全表现远超人类驾驶基线： - **严重人身伤害事故率降低 88%**（Reduced crashes involving serious injuries by 88%）。 - **导致安全气囊弹出的事故率降低 79%**（Reduced airbag deployment crashes by 79%）。 - **总体致伤事故率降低 78%**（Reduced overall injury-producing crashes by 78%）。 | 事故严重程度分类 | 相比人类驾驶基线的降低幅度 | |:--- |:--- | | **涉及严重人身伤害的事故**（Serious Injury Crashes） | **-88%** | | **触发安全气囊弹出的事故**（Airbag Deployment Crashes） | **-79%** | | **总体导致人员受伤的事故**（Overall Injury Crashes） | **-78%** | 除了宏观统计数据，演讲中还展示了具体的现实避险案例： - **黑夜行人跨栏翻越隔离带**：在深夜无光照环境下，两名行人突然翻越道路中央的 K-Rail 水泥隔离墩横穿马路。人类肉眼与车载摄像头在黑夜中均难以看清，但激光雷达（LiDAR）敏锐捕获到了两人的三维轮廓与移动趋势，系统提前微调航向远离隔离栏，从容规避了潜在伤亡事故。 - **旧金山交叉路口 Muni 公交车视线遮挡**：在旧金山一个繁忙路口，绿灯亮起时前方一辆旧金山城市铁路（Muni）公交车挡住了视线，系统依靠周边感知与盲区预判，在公交车驶离的瞬间提前探知被完全遮挡的行人，化解了十字路口的“鬼探头”隐患。 ### [20:25](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1225s) 具身智能的安全边界与规模化落地闭环 在实际行驶场景中，Waymo 的激光雷达（LiDAR）传感器展现出了超越人类视觉的感知能力，例如能够直接捕捉到从巴士底部反射出的稀疏扫描信号。Waymo Driver 借助这些微弱且稀疏的点云反射信息，精准预测出巴士另一侧正有一位行人可能正要绕行出来。基于这一前瞻性预测，车辆会主动提前减速，从而稳妥避开行人。这些真实的行驶案例生动展示了具身智能（Embodied AI）在开放道路上面临的根本考验——这关乎真实的人与生命（"It's people, it's lives."）。 仅凭单一的 AI 模型并不足以打造出真正值得信赖的自动驾驶系统。要实现超级人类水准的安全性，必须构建一套完整的全链路闭环体系：首先打造强大的前沿基础模型（Foundation Models），将其针对自动驾驶特有场景进行定向适配与微调；随后提炼并训练出轻量高效的学生模型（Student Models），使之能以更低成本在车端硬件和云端高效运行；最终再配合一整套严谨的工程评估流程，实现对车辆安全性的量化验证。正是依靠这套体系，Waymo 能够向业务的核心拷问给出确凿的肯定答案：“Waymo Driver 足够安全吗？我们能否实现规模化落地？——这两个问题的答案都是肯定的（Yes）。” ```mermaid flowchart LR accTitle: Waymo 自动驾驶安全与规模化闭环架构 subgraph ModelPipeline["模型构建与部署"] A["强大的基础大模型<br/>Foundation Models"] --> B["自动驾驶领域适配<br/>Domain Adaptation"] B --> C["蒸馏训练轻量学生模型<br/>Student Models (车端/云端降本)"] end subgraph SafetyLoop["量化与验证"] C --> D["全流程安全量化体系<br/>Safety Quantification"] D --> E["业务核心目标实现<br/>安全 (Safe) + 规模化 (Scale)"] end ``` - **超视距/遮挡感知**：利用穿透巴士底部的稀疏激光雷达（LiDAR）反射信号，提前预判盲区行人动向并执行主动减速。 - **构建可信驾驶的核心支撑**：涵盖大模型研发、自动驾驶任务适配、低成本车端/云端学生模型蒸馏，以及严密的安全性量化验证流程。 - **两大战略问题的定论**：Waymo 明确验证了其系统的“安全性（Safe）”与“可规模化能力（Scalable）”。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=4s) 打造全球最值得信赖的自动驾驶系统（World's most trusted autonomous driver） — Srikanth Thirumalai > [02:30](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=150s) 现实环境极度混乱且不可预测 — Srikanth Thirumalai > [02:30](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=150s) 云端大模型可以消耗数秒生成回答，而车端具身模型必须在“数毫秒”（milliseconds）之内做出攸关生死的操控决策。 — Srikanth Thirumalai > [04:55](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=295s) 苦涩教训（The Bitter Lesson） — Srikanth Thirumalai > [08:19](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=499s) 快思考与慢思考（Thinking Fast and Slow） — Srikanth Thirumalai > [08:19](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=499s) 该车辆抛锚受困，应当借道绕行而非死等 — Srikanth Thirumalai > [15:16](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=916s) 工程团队不能停留在定性分析，而是必须明确在特定软件版本中系统蕴含的“残余安全风险”（Residual safety risk）到底有多少 — Srikanth Thirumalai > [18:18](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1098s) 严重人身伤害事故率降低 88%（Reduced crashes involving serious injuries by 88%） — Srikanth Thirumalai > [18:18](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1098s) 导致安全气囊弹出的事故率降低 79%（Reduced airbag deployment crashes by 79%） — Srikanth Thirumalai > [18:18](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1098s) 总体致伤事故率降低 78%（Reduced overall injury-producing crashes by 78%） — Srikanth Thirumalai > [20:25](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1225s) 这关乎真实的人与生命（"It's people, it's lives."） — Srikanth Thirumalai > [20:25](https://www.youtube.com/watch?v=jrSNcEPWuhE&t=1225s) Waymo Driver 足够安全吗？我们能否实现规模化落地？——这两个问题的答案都是肯定的（Yes）。 — Srikanth Thirumalai ## Mentioned Srikanth Thirumalai, Waymo, Google, Waymo Driver, Waymo Foundation Model, Driver Teacher, Simulator Teacher, Evaluation/Critic Teacher, Muni, Transformer ## Video description Srikanth Thirumalai, VP of Engineering, Waymo Abstracts Get a peek under the hood of the Waymo Driver as we share about the technology that enables our rapid growth. Autonomous driving is uniquinely challenging, and this presentation will show how multi-modal sensing, advanced AI models, and cutting-edge simulation deliver superhuman driving performance. At Waymo, our mission is to build the world's most trusted driver. Discover how we go about earning that trust by pushing the boundaries of AI for the future of transportation. [Source](https://www.youtube.com/watch?v=jrSNcEPWuhE)

[0:04] Srikanth Thirumalai: 打造全球最值得信赖的自动驾驶系统（World's most trusted autonomous driver） [2:30] Srikanth Thirumalai: 现实环境极度混乱且不可预测 [2:30] Srikanth Thirumalai: 云端大模型可以消耗数秒生成回答，而车端具身模型必须在“数毫秒”（milliseconds）之内做出攸关生死的操控决策。 [4:55] Srikanth Thirumalai: 苦涩教训（The Bitter Lesson） [8:19] Srikanth Thirumalai: 快思考与慢思考（Thinking Fast and Slow） [8:19] Srikanth Thirumalai: 该车辆抛锚受困，应当借道绕行而非死等 [15:16] Srikanth Thirumalai: 工程团队不能停留在定性分析，而是必须明确在特定软件版本中系统蕴含的“残余安全风险”（Residual safety risk）到底有多少 [18:18] Srikanth Thirumalai: 严重人身伤害事故率降低 88%（Reduced crashes involving serious injuries by 88%） [18:18] Srikanth Thirumalai: 导致安全气囊弹出的事故率降低 79%（Reduced airbag deployment crashes by 79%） [18:18] Srikanth Thirumalai: 总体致伤事故率降低 78%（Reduced overall injury-producing crashes by 78%） [20:25] Srikanth Thirumalai: 这关乎真实的人与生命（"It's people, it's lives."） [20:25] Srikanth Thirumalai: Waymo Driver 足够安全吗？我们能否实现规模化落地？——这两个问题的答案都是肯定的（Yes）。

[0:04] Srikanth Thirumalai: over one million people die in road accidents every year, with the vast majority caused by human error. [0:04] Srikanth Thirumalai: Logged over 100 million miles on public roads in fully driverless mode. [0:28] Srikanth Thirumalai: The driving environment is inherently messy, unpredictable, and cluttered with edge cases [0:28] Srikanth Thirumalai: Unlike low-stakes software, the cost of an error in physical traffic is extremely high. [0:28] Srikanth Thirumalai: Cloud-based generative AI systems can take seconds to generate answers, but embodied driving AI must compute safe driving decisions within milliseconds. [4:55] Srikanth Thirumalai: while popular AI relies on the "bitter lesson" (scaling compute and data), achieving autonomous driving at scale requires more than just a driver [4:55] Srikanth Thirumalai: Off-the-shelf VLMs cannot handle raw LiDAR/radar inputs, cannot natively output low-level control actions (steering, acceleration, braking), and are far too slow for real-time onboard driving. [11:02] Srikanth Thirumalai: Because these massive teacher models are too computationally heavy to run real-time in the car or across hundreds of millions of simulated miles in the cloud, Waymo uses knowledge distillation. [15:26] Srikanth Thirumalai: Rather than guessing, Waymo mathematically quantifies the "residual risk" of any software update [15:26] Srikanth Thirumalai: 88% reduction in crashes resulting in serious injuries compared to human drivers. [15:26] Srikanth Thirumalai: 79% reduction in crashes where airbags were deployed. [15:26] Srikanth Thirumalai: 78% reduction in overall injury-causing crashes. [20:25] Srikanth Thirumalai: LiDAR sensors pick up a person's reflection from underneath a bus. Even with only sparse scan information available, the Waymo Driver is able to predict that a pedestrian is obscured on the other side of the vehicle [20:25] Srikanth Thirumalai: Achieving the vision of building the world's most trusted driver requires far more than standalone AI algorithms.

[0:04] Hello everyone. Uh my name is Shriant [0:06] Dumalai and I lead the onboard [0:08] engineering team at Whimo. My team [0:11] develops and improves the Whimo driver [0:13] which is what we call our fully [0:16] autonomous driving system. And today I'm [0:18] here to talk about how we're using AI to [0:22] power um driving at scale safely and [0:26] effectively. [0:28] I'll skip this marketing video so I can [0:31] spend more time on the actual content. [0:33] So, let's start with this number. Why [0:36] are we solving the self-driving car [0:38] problem? Because this number tells us [0:42] it's urgently needed. Every year, over a [0:45] million people lose their lives in road [0:49] accidents around the world, and the vast [0:52] majority of them are due to human error. [0:55] Now this is a saddening and staggering [0:58] statistic and we should do better. We [1:01] think we can do better and that's why 15 [1:05] years ago we started the Google we [1:08] started off as the Google project before [1:11] spinning off into a separate company [1:14] about eight years ago with the lofty [1:16] mission of building the world's most [1:20] trusted autonomous driver. Now in this [1:23] talk I'll share some details about how [1:25] we do that at large scale using AI. [1:31] So what kind of scale are we talking [1:32] about? As of today, the Whimo driver has [1:35] logged over 100 million miles on public [1:38] roads [1:40] in driverless mode and paying customers [1:44] across five cities are able to hail [1:47] rides in a Whimo in SF, in LA, in [1:51] Phoenix, Austin, and Atlanta. Now, we [1:54] have plans to expand our service to many [1:56] more cities next year. And we're also [1:59] testing our driver in even more cities, [2:03] including internationally, under some [2:05] really extreme weather conditions. Heavy [2:08] snow, dry heat of the kinds we have here [2:11] in Vegas and torrential rains. Now, the [2:15] scale is huge as you can see from the [2:17] number, but it's not daunting because we [2:20] think we have the right approach to [2:22] building our product to scale safely. [2:24] And today, I'd like to share that [2:26] approach with you. [2:30] So first what makes this AI problem so [2:33] challenging? [2:35] Our AI first off our AI runs in the car [2:40] in the real world. In that sense it's [2:42] embodied AI and that presents three [2:46] challenges that I've listed here. First, [2:49] the real world is complex. It's [2:51] unpredictable and it's just downright [2:53] messy. [2:54] Second, this autonomous driving problem [2:58] is a safety critical problem. The cost [3:00] of a mistake, as you can well imagine, [3:03] is very high. And third, unlike [3:06] generative AI that runs in the cloud, we [3:09] don't get seconds to make driving [3:11] decisions. Our embodied AI has to make [3:15] decisions in a matter of milliseconds. [3:19] And here are some examples of the [3:22] messiness that we have to deal with on [3:24] the roads. You know, stray cyclists, [3:27] toddlers on the loose, debris falling [3:30] off of trucks, fallen trees, flooded [3:34] roads, [3:35] distracted drivers, you name it. So, how [3:39] do we deal with all of this complexity? [3:43] The short answer is with powerful [3:46] sensors and state-of-the-art AI. You [3:49] know, our cars are outfitted [3:52] with sensors of multiple modalities. The [3:55] current generation Whimos, our fifth [3:57] generation car, have 29 cameras, five [4:02] LARs, six radars, and four audio [4:06] receivers that give us a comprehensive [4:09] and accurate 3D view of the world around [4:12] the car. And here's a scene [4:16] of two dogs chasing after a ball and two [4:18] humans chasing after the dogs. And all [4:21] of this is happening in front of us, but [4:23] our sensors are able to pick all of this [4:25] up and give the software the information [4:28] which leads the car to stop. [4:31] Now, the video on the bottom shows you [4:34] what our cameras pick up, and the video [4:37] on the top kind of shows you what our [4:40] lightars and radars pick up. But just [4:43] having good, powerful multimodal sensing [4:46] isn't enough. We need great software. [4:50] And I'll go over that in the upcoming [4:52] slides. [4:55] Now, a lot of people think, well, how [4:57] hard can this be? Just build a great [5:00] driver using the latest AI and throw a [5:03] lot of training data at it and we'll be [5:05] done, right? After all, isn't that what [5:08] the bitter lesson in AI is all about? I [5:11] mean, I'm here to say, yes, you need a [5:13] great driver. You need a great AI [5:15] driver, but there's a whole lot more [5:17] that you need. You need a good simulator [5:20] to test the driver's performance in [5:23] challenging situations. You need a good [5:25] critic to evaluate how well the driver [5:28] performed. And finally, you need a way [5:32] to tell if your driver is ready for full [5:36] autonomy. And to scale successfully, you [5:39] need to have solved all these problems. [5:41] Well, in the next slides, I'll share [5:45] with you how we're leveraging AI to [5:47] solve these problems. [5:50] Now, let's start off with vision [5:52] language models. You know, everyone [5:54] knows them. They're all the rage these [5:56] days. You probably use it in your [5:59] personal lives every day. They're [6:00] amazing and they're getting better by [6:03] the day. [6:04] Many of you in this room are also [6:06] building products powered by them and [6:09] they're trained on all of the data [6:11] available on the internet and as a [6:14] result they have broad world knowledge [6:17] and some of these frontier VLMs are also [6:19] very good at chain of thought reasoning. [6:23] These VLMs are multimodal in that they [6:26] take videos, images, and text as input [6:30] and produce those modalities as outputs. [6:32] And they're foundation models. They're [6:34] incredibly flexible and can be adapted [6:37] to various tasks. So, we decided to look [6:40] at vision language models and see how [6:43] good they are and whether we could use [6:44] them for self-driving. Um, and so here's [6:47] an example [6:49] uh that showcases their abilities. [6:52] Now, let's say it's Wednesday at 4 p.m. [6:56] and you want to know whether you can [6:58] park at this sign. For humans, you have [7:01] to read each sign carefully, parse what [7:03] it's trying to say, reason across all of [7:05] the information present in all of the [7:07] signs before you actually decide whether [7:08] you can park. But these VLMs are pretty [7:10] good. You know, they can take that image [7:12] and they can tell you that, yeah, it's 4 [7:14] p.m. on a Wednesday. You can park here [7:15] for one hour. That's pretty impressive. [7:19] Now, now what? So, can we use this for [7:24] self-driving? The answer is not quite. [7:28] Not without a lot of um me a second. Not [7:32] without a lot of fine-tuning to adapt [7:35] them to the problem of self-driving. [7:38] Now, some of these, you know, of course, [7:40] some of these are obvious, like what are [7:42] the what do we need to do? As I said, [7:45] Whimo uses multiple sensing modalities. [7:48] We use lightars and radars. Certainly, [7:50] these VLMs have not been trained with [7:51] that data. Um, they certainly can't [7:54] right out of the box [7:56] convert that sensory information about [7:59] the environment into action sequences [8:02] like acceleration, steering, and braking [8:04] that the car can execute. And most [8:07] importantly, they can't do this in real [8:09] time. Like I just mentioned now, they [8:12] need a lot of work to adapt them to the [8:14] complexities of real world driving. [8:19] So the first step is to do precisely [8:22] that adapt them to the self-driving [8:24] task. So we built a foundation model to [8:28] solve all of the challenges that I told [8:29] you about which is that you need a good [8:31] driver, you need a good simulator, and [8:34] you need a good evaluator if you want to [8:37] use cutting edge AI to solve the [8:38] self-driving problem. And so we built [8:41] this foundation model which we call the [8:43] Whimo foundation model. And basically [8:46] it's an encoder decoder model. On the [8:48] encoder side, you know, we have the [8:51] sensor fusion network, which is really [8:53] an encoder, a multimodal encoder that [8:56] takes camera, LAR, and radar as three [8:59] inputs and then fuses information across [9:02] those three modalities and also fuses [9:04] that information over time to produce [9:08] tokenized embeddings, which is then I [9:11] mean these tokenized embeddings are then [9:12] sent to an action decoder, which is very [9:16] much like a regular transformer decoder [9:19] that you have in some of these vision [9:21] language models. And what that action [9:23] decoder does is it decodes, you know, [9:26] that input into action sequences that [9:30] the car can execute. Um the decoder also [9:34] predicts the behavior of other agents on [9:36] the road and makes a few other [9:39] predictions. Now these two models, the [9:41] sensor fusion network and the action [9:43] predict action decoder, they have access [9:46] to some but not all the world knowledge [9:49] that a VLM might have. And they also [9:53] don't have the reasoning abilities, [9:54] right, that VLMs have. And so the thing [9:58] to do is to add a driving VLM, you know, [10:01] to this to the setup. And what this [10:04] driving BLM is able to do because it's [10:06] adapted to the driving problem is that [10:08] it's able to understand complex [10:09] scenarios that unfold on the road and [10:11] also reason about highle actions that [10:14] the car needs to take in response. For [10:17] instance, let's say the car sees another [10:20] vehicle stranded on the roads with its [10:22] blinkers on. Now it might reason the [10:25] vehicle is in distress and can't move [10:28] and what it has to do instead is to [10:31] rather than wait for the car to proceed, [10:33] it has to nudge around it and make [10:35] progress. Now that's what these driving [10:38] VLMs can do with their reasoning [10:39] ability. And so these BLM tokens when [10:42] used along with the sensor fusion [10:44] network tokens improve the quality of [10:48] the Whimo Foundation model and its [10:50] predictions. sort of like thinking fast [10:53] and slow where the upper path is [10:55] thinking fast the lower path is thinking [10:57] slow it's thinking slow but doing that [10:59] simultaneously [11:02] now having built this Whimo foundation [11:05] model and it is a foundation model it's [11:08] versatile it's flexible um we'd like to [11:11] build our tech stack on top of it to [11:13] solve the challenges that I alluded to [11:15] earlier a good driver a good simulator [11:19] and a good evaluator and the way we do [11:21] that is we adapt the Whimmer Foundation [11:23] model to these three specific tasks to [11:26] produce large highquality teacher models [11:30] that are adept at their respective [11:32] tasks. And the teacher models of course [11:35] are too big to run in the car or in the [11:38] cloud to simulate um and evaluate the [11:42] hundreds of millions of miles of driving [11:44] that we need to. And so we use [11:46] distillation. We distill those teacher [11:49] models into smaller models that we call [11:52] student models uh that are practical for [11:56] us to run in the car or in the cloud in [12:00] our simulation and evaluation [12:03] so that they meet our latency and cost [12:05] needs. [12:06] So here it is at a 30,000 ft level. This [12:10] is how we're leveraging cutting edge AI [12:13] to solve our problem. [12:19] Now, in the next three slides, I'll deep [12:21] I'll dive deeper into the three teacher [12:24] models and what they do. [12:26] This is our first teacher model, the [12:28] driver teacher. And the goal of this [12:31] teacher model is to generate safe and [12:33] compliant action sequences for the car [12:36] to execute so that the car can move [12:39] smoothly and comfortably as it makes [12:42] progress towards its destination. [12:46] And this model is fine-tuned with human [12:49] driving demonstrations. We have human [12:51] drivers that drive the car that provide [12:52] it with training data. And so it's [12:54] fine-tuned on those driving [12:56] demonstrations. It's also fine-tuned on [13:00] the 100 million miles that I just [13:02] referred to in the past of autonomous [13:04] driving that we have annotated with [13:06] humans and algorithms [13:09] um for when the car got it wrong and [13:12] what it needs to do instead if it did. [13:15] Now we're leveraging training techniques [13:18] like behavior cloning and reinforcement [13:20] learning as post-training steps. uh when [13:22] we produce this uh teacher. [13:26] The second teacher model is a simulator [13:28] model and this model's goal is to [13:31] generate realistic diverse scenarios in [13:35] challenging situations [13:37] uh including accidents, you know, near [13:39] misses, inclement weather, complex [13:42] intersections, [13:44] even vehicles losing control, you know, [13:47] on freeways. [13:49] And this teacher model is also [13:52] fine-tuned on all our driving logs, but [13:55] it's fine-tuned on the simulation task [13:58] of predicting how the world evolves with [14:01] every time step. You may have heard the [14:02] term world model. That's what this is [14:04] trying to achieve. [14:06] Now, this model is post-trained with [14:10] very similar sort of, you know, post- [14:12] training techniques, state-of-the-art [14:14] techniques, closed loop training and [14:15] reinforcement learning. [14:19] And the final model, the final teacher [14:20] model is the critic model or the [14:22] evaluation teacher. And this model's [14:25] goal is to analyze, understand, and [14:29] detect notable events that may indicate [14:33] bad driving. [14:35] Among other things, you know, this model [14:37] is trained with driving runs that humans [14:40] have annotated with information on what [14:42] the car did poorly or other events of [14:45] note, what the car should have done in [14:48] those situations. [14:50] You know, other sources of information [14:52] like way more internal docs and bugs [14:54] also provide rich information to these [14:57] models. And the teacher is kind of [14:59] fine-tuned on that evaluation task which [15:01] is different from the simulation task [15:03] and the driver task which is detecting [15:06] bad behavior uh when you analyze [15:08] simulation logs or driving logs and [15:11] again it uses supervised fine-tuning. [15:16] So now you've seen the three teachers [15:18] for the three different tasks and we [15:20] also talked about how these teachers are [15:22] distilled into smaller models that run [15:24] in the car or in the cloud. Let's talk [15:27] about the last challenge which is how do [15:29] we tell if the Whimo driver is ready for [15:33] the road [15:35] because scaling safely needs more than [15:39] just great AI models. We need a [15:42] framework to determine the readiness of [15:45] the driver. And we've developed a very [15:49] comprehensive readiness and safety [15:52] framework over the years thanks to our [15:56] more than a decade long uh experience in [16:00] this space and we follow this readiness [16:03] and safety framework [16:05] for any software update to the Whimo [16:07] driver. We put it through this process [16:09] to quantify the driver's readiness. [16:13] Now the word quantify is really [16:16] important because you can build a [16:18] powerful AI. You can to drive the car. [16:21] You can run it through a great [16:22] simulator. You can build a powerful [16:24] critic to detect poor performance. [16:28] But how do you tell how much safety risk [16:30] you're carrying in any given driver [16:33] version? [16:35] In other words, how often do you expect [16:38] this model to perform poorly in the [16:41] normal course of driving on the roads? [16:44] That's a really important question. And [16:46] in order to quantify the residual risk [16:48] in the driver, first you need to know [16:51] what to look for. collisions, [16:54] rear uh near misses, improper stops and [16:58] strandings, software crashes, [17:01] security vulnerabilities, [17:04] overall drive quality, and a whole lot [17:06] more which I can't cover here. And then [17:09] you need to build driving sets that you [17:11] can simulate and use the occurrences of [17:14] poor driving in simulation and [17:18] extrapolate from that [17:20] um to get some sense of the frequency [17:23] with which these events are likely to [17:25] occur on the road. And all this needs [17:30] expert judgment [17:32] and years of experience to know what to [17:34] look for, how to construct these driving [17:37] sets, and how to extrapolate this [17:40] information to rates that you're likely [17:42] to observe, right, in real world [17:44] driving. And this requires cutting edge [17:46] data science. It's a rigorous and [17:49] comprehensive process that quantifies [17:52] the residual safety risk in the car at [17:55] any given operating scale. So let's [17:57] reflect on the whole picture. Foundation [18:00] model teachers, foundation models, [18:03] teachers that adapt adapted from those [18:05] foundation models, students that are [18:07] distilled from those teachers and this [18:09] readiness framework and process. We do [18:12] all this to create that safe um and [18:15] scalable driving experience. [18:18] So, let's see if we're actually doing [18:20] it. Now, I'm happy to say that the Whimo [18:23] approach to scaling safely is working. [18:26] I'm very proud of our safety record and [18:29] the positive impact we've made on public [18:31] roads. We analyzed 71 million miles of [18:36] driving data [18:38] and found that the Whimo driver reduced [18:41] the amount of crashes involving serious [18:44] injuries by 88%. It reduced the number [18:48] of crashes where airbags were deployed [18:51] by 79% [18:53] and it reduced the overall injury [18:56] producing crashes by 78%. Now these just [19:00] these are just statistics. Let me [19:02] actually show you actual events of the [19:05] Whimo driver protecting people like you [19:07] and me. [19:09] Now, in this scene, you'll see our LAR [19:12] sensors see two people jumping over a K [19:16] rail partition on the road onto the road [19:18] in the dead of night. It's dark and you [19:22] can't see them with a naked eye and our [19:24] cameras also don't quite pick them up, [19:25] but our light are able to see them. And [19:28] using that information, the car nudges [19:29] away from the car rails, anticipating [19:31] the trajectory of these two humans, and [19:33] safely avoids them. And you'll see our [19:36] LAR picking up those two people that are [19:38] trying to jump over the the KRL. [19:41] Now, this is how our LAR [19:44] helping us improve the safety of uh the [19:46] Whimo driver. Now, here's another [19:48] example in broad daylight. You know, [19:51] there's this is an this is an image of a [19:54] a busy a street intersection in San [19:56] Francisco, and you see this MUN bus um [20:00] blocking the intersection. And so the [20:02] question is what will the Whimo driver [20:04] do as it approaches this intersection? [20:06] You know, our light turns green and we [20:08] have to proceed. So let's see how this [20:10] unfolds. [20:12] As you can see, you know, as the bus as [20:15] the bus proceeds, we detect a a a person [20:21] oluded by the bus. And the way we detect [20:23] this person is because our peripheral [20:25] LAR sensors pick up this person's [20:27] reflection uh from underneath the bus. [20:30] And so we use that sparse scanned [20:33] information to predict that there's a [20:35] human on the other side of the bus who's [20:37] likely to come around. And that causes [20:39] us to slow down um and avoid the human. [20:42] Now these videos are powerful reminders [20:45] of what's at stake for embodied AI on [20:48] the roads. It's people, its lives. And [20:51] AI alone is not enough to build a [20:54] trusted driver. You need this whole [20:56] process. You need to build powerful [20:58] foundation models. adapt them to the [21:00] self-driving problem, build students so [21:02] they can run in the car and in the cloud [21:05] at at lower cost and then put a process [21:07] around it to ensure you can quantify the [21:10] safety of the car. And using this [21:12] process, we can answer the two most [21:15] important questions for our business. Is [21:17] the Whimo driver safe? And can we scale [21:20] it? And the answer to both questions is [21:24] yes. [21:26] That's my talk. Thank you.