May Mobility Pairs Reasoning With Deep Learning to Scale AVs
Ai4
Hybrid architecture using on-car real-time simulation enables autonomous vehicles to handle unseen scenarios safely with just 20,000 miles of training data.
## Background
This presentation was published on YouTube by the Ai4 channel on August 18, 2026. It features Jacob Crossman, Senior VP of Autonomy at May Mobility, discussing why traditional rule-based and end-to-end autonomous vehicle (AV) models struggle with unfamiliar edge cases and how pairing deep learning with real-time reasoning models offers a safer, more scalable alternative.
## TL;DR
Jacob Crossman challenges the industry's reliance on brute-force data scaling and multi-billion-dollar data centers for autonomous driving, arguing that humans become proficient drivers with only tens of thousands of miles of experience. He introduces May Mobility's Multi-Policy Decision-Making (MPDM) architecture, which combines learned transformer models with real-time online reasoning and mathematical fallbacks evaluated 5 times per second. By interleaving machine-learned policies with deterministic rules, the system navigates complex edge cases, handles narrow passes, and safely executes rule violations like passing double-parked vehicles using models trained on just 20,000 miles of human driving data.
## Atmosphere & style
The presentation is authoritative, technical, and pragmatically grounded, delivered by an industry veteran with decades of AI and AV experience. Crossman uses light humor—such as referencing *The Matrix* when discussing data center energy consumption—to make high-level architectural concepts engaging and accessible. The atmosphere feels like a clear-headed, candid industry reality check that thoughtfully moves from grand technological critiques to precise visual walkthroughs of real-world driving scenarios.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=4s) Jacob Crossman highlights the immense resource demands of AI scaling, noting that facilities like xAI's Colossus 2 cost over $30 billion and draw more power than Ann Arbor, Michigan.
- [00:04](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=4s) Data centers are projected to consume 10% of total U.S. electricity by 2030, with half of that dedicated specifically to AI workload acceleration.
- [00:14](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=14s) End-to-end deep learning models suffer from unpredictable black-box behavior when encountering unfamiliar scenarios, driving an inefficient race for hundreds of millions of training miles.
- [00:14](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=14s) Humans learn to drive safely within a few hundred to tens of thousands of miles, proving that brute-force data collection is not required for real-world intelligence.
- [05:27](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=327s) May Mobility's Multi-Policy Decision-Making (MPDM) brings real-time chain-of-thought reasoning to autonomous driving, evaluating trajectory options on the fly.
- [05:27](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=327s) MPDM runs directly on the vehicle at 5 Hz, simulating hundreds of potential candidate options per fraction of a second to select safe actions.
- [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) The hybrid architecture seamlessly interleaves machine-learned imitation models with traditional mathematical and rules-based fallback models.
- [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) By observing all surrounding road traffic rather than relying solely on fleet miles, May Mobility trained its core driving models on just 20,000 human driving miles.
- [07:33](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=453s) During high environmental uncertainty, classic deterministic models temporarily override machine learning predictions to guarantee safety in complex yield or nudge scenarios.
- [08:59](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=539s) Visualized trajectory lines and indicator bars show the real-time balance between machine-learned execution for smooth interactions and rules-based policies for safety checks.
- [08:59](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=539s) The reasoning-first approach enables complex behaviors, including narrow space passing, yellow light stop-or-go decisions, and safely breaking standard operating procedures like crossing double yellow lines.
- [12:30](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=750s) Reasoning-first architecture allows rapid deployment across dozens of cities without needing location-specific retraining or heavy HD mapping dependencies.
- [12:30](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=750s) The vehicle's L4 sensor suite relies on a combination of LiDAR for depth, cameras for semantics, and radar for instant velocity detection.
## Detailed walkthrough
Here is a detailed, faithful walkthrough of the presentation transcript:
[00:04](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=4s) Introduction and the Energy Crisis of AI Scaling
Jacob Crossman, Senior VP of Autonomy at May Mobility, introduces himself, drawing on 30 years of experience across various AI fields—including 15 years dedicated to off-road and on-road autonomous vehicles (AVs). Having led May Mobility's autonomy efforts for five years, he addresses what it truly takes to make AV technology scale effectively.
He highlights the staggering footprint of modern AI infrastructure, pointing to xAI's launch of Colossus 2 to accelerate full self-driving (FSD) and AI capabilities. The facility alone carries an estimated price tag exceeding $30 billion, consuming more power than the entire city of Ann Arbor, Michigan (where May Mobility is headquartered). Driven by theoretical scaling laws and real-world results, the industry race has centered on acquiring bigger models, more compute, and vast amounts of data. Projected trends indicate that by 2030, approximately 5% of all U.S. electricity will power AI data centers, with another 5% going to traditional data centers—humorously leading Crossman to joke that *The Matrix* is coming true, with humans serving as mere power plants for AI.
- Jacob Crossman: 30 years in AI, 15 years in AVs, 5 years at May Mobility.
- xAI's Colossus 2 facility cost exceeds $30 billion and draws more power than Ann Arbor, Michigan.
- By 2030, an estimated 10% of total U.S. power will be consumed by data centers (5% specifically for AI data centers).
[00:14](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=14s) The Hidden Trap of End-to-End AV Models
Turning to autonomous driving, Crossman examines end-to-end deep learning—a popular trend where sensor data or camera pixels feed directly into a "black box" neural network to output steering, gas, and braking commands. The main appeal of this approach lies in shifting the burden onto massive data centers to learn driving behaviors from immense fleets.
However, the major drawback is cost and safety risk. If an end-to-end vehicle encounters a scenario it has never trained on (or something dissimilar), its behavior becomes unpredictable. To compensate, companies constantly attempt to collect hundreds of millions of driving miles. Crossman challenges this paradigm by asking how many miles a human actually needs to learn to drive:
> "In Michigan... you have to drive for a few hundred miles before you can get a driver's license. Now, how much longer do you need to drive in order to become a good driver? Say maybe 10,000 miles, maybe 20,000 miles."
This comparison raises fundamental questions about whether building gigawatt data centers to brute-force driving skills represents true human-like intelligence or if a more efficient, reasoning-first path exists.
[00:327] Multi-Policy Decision-Making: Bringing Chain-of-Thought to AVs
Crossman draws a parallel to early chatbot LLMs, which suffered from two major flaws: an inability to answer deep reasoning/problem-solving questions and an inability to incorporate recent real-world information. AI developers solved these limitations by introducing Retrieval-Augmented Generation (RAG) for up-to-date context and chain-of-thought reasoning to evaluate multiple potential answers before responding.
May Mobility applies a similar architecture to autonomous vehicles, termed **Multi-Policy Decision-Making (MPDM)**. Rather than relying purely on end-to-end black boxes, MPDM performs "thinking on the fly" as the car drives. While transformer models handle perception, policy, and prediction, MPDM augments these learned models with real-time online reasoning, avoiding the need for millions of training miles.
```
+-----------------------------------------------------------------+
| Information-Rich Perception Model |
| (Transformers for Tracks & World Objects) |
+-----------------------------------------------------------------+
|
v
+-----------------------------------------------------------------+
| Candidate Policies |
| (Proposes actions from ML, physics, & rules-based models) |
+-----------------------------------------------------------------+
|
v
+-----------------------------------------------------------------+
| On-Car Real-Time Simulator |
| (Simulates 100s of options/fraction of a sec @ 5 Hz) |
+-----------------------------------------------------------------+
|
v
+-----------------------------------------------------------------+
| Policy Selection |
| (Selects safe, progress-making action) |
+-----------------------------------------------------------------+
```
- MPDM runs directly on the vehicle's computer at 5 Hz (evaluating options five times per second).
- Simulates hundreds of trajectory options per fraction of a second and thousands per second.
- Combines machine-learned imitation policies with mathematical and rules-based fallback models.
[06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) Real-Time Interleaving and the 20,000-Mile Fleet Model
The primary hurdle in autonomous driving is solving rare edge cases, which traditional machine learning attempts to cover with endless training data. May Mobility overcomes this by seamlessly interleaving learned models with classic mathematical/rules-based models. When a learned model faces an unfamiliar situation, classic models take over for a split second to navigate through safely. Conversely, when classic models reach their structural limits in complex urban interactions, learned models take over to ensure smooth execution.
This hybrid approach allowed May Mobility to train its models on a mere **20,000 miles** of human driving data—a fraction of competitor requirements—by leveraging driving data not just from their own vehicles, but from all surrounding traffic observed on the road. Additionally, because explicit policies are evaluated in simulation, every driving action is transparent and explainable during post-drive analysis.
[07:33](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=453s) Real-World Example: Nudge and Yield Mechanics
To illustrate MPDM in action, Crossman presents a scenario involving a May Mobility vehicle (a white minivan) approaching a double-parked delivery truck with a delivery worker nearby. Simultaneously, an oncoming purple car and a bicyclist approach from the opposite direction.
- **Standard ML Prediction:** An end-to-end model assumes the cyclist will hug the curb and the oncoming purple car will stay in its lane, prompting the AV to smoothly nudge around the delivery truck.
- **MPDM Execution:** The May Mobility system simulates plausible alternative behaviors, including an aggressive maneuver where the purple car swings wide into the shared center space. Realizing the learned imitation policy could risk a collision in that specific branch of the simulation, the system momentarily overrides the ML model, electing to stop and wait. Once the spatial uncertainty clears, the system seamlessly switches back to the learned policy and proceeds around the truck.
[08:59](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=539s) Visualizing Uncertainty and Handling Complex Driving Scenarios
Demonstrating a video drive from earlier in the year, Crossman highlights how the software operates in real time. Visualized on screen, green and orange trajectory lines fan out during periods of high environmental uncertainty (such as busy intersections or multi-agent interactions) and cluster tightly together during straightforward driving.
A colored indicator bar beneath the vehicle displays the relative contribution of machine-learned (ML) models versus traditional rules-based models:
- **Higher Blue Bar:** ML policies take primary control to execute nuanced, smooth multi-agent interactions that are difficult to program manually.
- **Lower Blue Bar:** Traditional rules-based policies take over for standard operational tasks (such as stopping at a stop sign) or to bridge brief gaps where learned policies behave suboptimally.
Crossman reviews several complex driving behaviors mastered by this system:
- **Narrow Space Passing:** Predicts oncoming vehicle yielding behavior to pass tightly parked cars with human-like comfort.
- **Breaking Standard Operating Procedures:** Accurately determines when to strictly adhere to driving rules versus when to safely violate them (e.g., crossing a double yellow line to bypass a permanently blocked lane or squeezing into a tight merge window).
- **Yellow Lights & Unexpected Obstacles:** Combines instant reasoning with deterministic safety checks to make immediate "stop or go" decisions at yellow lights or during sudden road hazards.
[12:30](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=750s) Deployability Across Cities and Audience Q&A
Because the underlying core architecture relies on real-time reasoning alongside compact 20,000-mile models, May Mobility has successfully deployed its systems across dozens of cities without needing extensive site-specific retraining or customization.
During the Q&A session, Crossman fields questions regarding cross-climate capability, sensor suites, retrofitting, and mapping dependencies:
- **Winter Conditions vs. Warm Climates:** Driving logic transfers effectively between locations (e.g., operating in both Minnesota and Michigan). While snow requires specific handling, core driving interaction models remain identical. Crossman humorously dispels the notion that Michigan is permanently covered in snow, noting it only experiences heavy snow on select days.
- **Sensor Suite:** The platform utilizes a multi-sensor payload consisting of **LiDAR, Cameras, and Radar**:
- *Cameras:* Provide rich semantic interpretation of the scene.
- *LiDAR:* Delivers precise spatial depth for physical obstacle avoidance.
- *Radar:* Provides immediate velocity detection, essential for tracking fast-moving cross-traffic.
- **Vehicle Retrofitting:** While current operations focus on specific platform integrations, the core software-based architecture can be adapted to various vehicle types in the future.
- **Localization and Mapping:** Operating as a **Level 4 (L4)** autonomous system, the vehicle utilizes a medium-definition map containing basic road network topology, avoiding the need for heavy LiDAR-dense HD maps.
## Notable quotes
> [00:14](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=14s) In Michigan... you have to drive for a few hundred miles before you can get a driver's license. Now, how much longer do you need to drive in order to become a good driver? Say maybe 10,000 miles, maybe 20,000 miles. — Jacob Crossman
## Mentioned
Jacob Crossman, May Mobility, Ai4, xAI, Colossus 2, Ann Arbor, Michigan, Multi-Policy Decision-Making (MPDM), Minnesota
## Video description
Jacob Crossman - Sr. VP of Autonomy, May Mobility
Humans don’t need millions of miles of experience to drive safely, so why do we keep building AVs that do? As autonomous vehicles expand into more cities and edge cases multiply, chasing data is not enough to make our roads safer. I’ll explain how traditional rule-based and end-to-end AV models share a critical flaw: they can’t reliably handle situations they’ve never seen before. I’ll also show why pairing deep learning with reasoning models is the architecture that handles any situation and why that matters at scale. If you want to see how AVs will truly create a safer world, this presentation will take you through the how and why of building AVs responsibly.
May Mobility结合推理与深度学习推进自动驾驶规模化
利用车载实时仿真的混合架构，仅凭2万英里训练数据即可让自动驾驶汽车安全应对未知场景。
Jacob Crossman质疑了自动驾驶行业对暴力数据扩展和耗资数十亿美元数据中心的依赖，指出人类仅需数万英里的经验就能成为熟练驾驶员。他介绍了May Mobility的多策略决策（MPDM）架构，该架构将训练好的Transformer模型与每秒评估5次的实时在线推理及数学兜底机制相结合。通过将机器学习策略与确定性规则相交织，该系统仅基于2万英里人类驾驶数据训练的模型，即可应对复杂的极端工况、处理狭窄通行，并安全执行绕行双排停放车辆等违背常规规则的操作。
May Mobility 自主研发高级副总裁 Jacob Crossman 指出，AI 算力扩张面临巨大的资源与能耗挑战，例如 xAI 的 Colossus 2 耗资逾 300 亿美元且能耗超过密歇根州安娜堡全城，预计到 2030 年数据中心将消耗全美 10% 的电力。 (0:04)
端到端深度学习自动驾驶模型在遇到未知边缘场景时具有不可预测的黑盒缺陷，导致行业陷入收集数亿英里数据的低效竞争；而人类仅需一两万英里即可熟练驾驶，证明纯暴力堆积数据并非最优解。 (0:14)
May Mobility 提出多策略决策（MPDM）架构，将实时思维链（Chain-of-Thought）推理引入自动驾驶，直接在车端以 5 Hz 的频率运行，每秒模拟评估数千个候选轨迹以确保安全决策。 (5:27)
该架构将机器学习模仿模型与经典规则及数学兜底模型实时交织，通过观察道路上所有交通流而非仅依赖自身车队里程，仅用 2 万英里人类驾驶数据便完成了核心模型训练，并成功跨多座城市部署。 (6:10)
Reading summary
Jacob Crossman, Senior VP of Autonomy at May Mobility, contends that scaling autonomous vehicles requires on-vehicle, reasoning-first decision-making rather than the industry's prevailing brute-force approach of collecting hundreds of millions of driving miles. He argues that combining lightweight learned models with real-time trajectory simulation allows vehicles to resolve edge cases safely with vastly lower computing and data overhead.
The Energy and Data Limits of Brute-Force Scaling (0:14)
Crossman highlights the mounting infrastructure costs of end-to-end deep learning, citing xAI's Colossus 2 facility—estimated at over $30 billion and consuming more power than Ann Arbor, Michigan. Projected industry estimates suggest that by 2030, data centers will consume roughly 10% of total U.S. electricity, with AI data centers accounting for 5%. In autonomous driving, relying solely on end-to-end neural networks shifts the development burden to massive data centers, requiring hundreds of millions of miles to account for rare scenarios.
Crossman challenges this paradigm by noting that human drivers achieve competence within 10,000 to 20,000 miles of driving experience. He asserts that end-to-end black-box models yield unpredictable behaviors when encountering unfamiliar inputs, creating safety risks that brute-force data collection cannot efficiently solve.
Multi-Policy Decision-Making and Real-Time Reasoning
To overcome the limitations of black-box models, May Mobility implements Multi-Policy Decision-Making (MPDM), an architecture analogous to Retrieval-Augmented Generation and chain-of-thought reasoning in large language models. Rather than relying entirely on pre-trained imitation models, MPDM conducts online evaluations directly on the vehicle's onboard computer at a frequency of 5 Hz.
While transformer models handle perception, prediction, and policy generation, the MPDM framework passes candidate actions through an on-car simulator. The system evaluates hundreds of options per fraction of a second and thousands per second across machine-learned, physics-based, and deterministic rule-based options to select the trajectory that balances safety with forward progress.
Interleaving Rules and Machine Learning with 20,000 Miles (6:10)
May Mobility trains its foundational models on 20,000 miles of human driving data by capturing observations from both the host vehicle and surrounding traffic. To resolve unfamiliar edge cases without massive datasets, the system seamlessly interleaves learned policies with classical mathematical and rules-based fallbacks.
When a learned neural network faces unfamiliar edge scenarios, deterministic rules take control momentarily to maintain safety. Conversely, when deterministic models hit structural limits in dense urban interactions, learned models take over to ensure smooth operation. Crossman notes that this hybrid simulation approach keeps every driving action transparent and auditable during post-drive evaluations.
Dynamic Trajectory Evaluation and Safe Rule Breaking (7:33)
In a demonstrated real-world example, a May Mobility vehicle approached a double-parked delivery truck alongside oncoming traffic and a cyclist. While standard imitation models might nudge forward under the assumption that other road users will hold their lines, MPDM simulated worst-case deviations—such as an oncoming car swinging wide—and commanded the vehicle to stop until spatial uncertainty cleared before resuming.
Crossman explains that real-time trajectory fans widen during high uncertainty and cluster during routine driving. This framework allows the vehicle to safely navigate narrow spaces, handle stop-or-go choices at yellow lights, and determine when to intentionally breach standard rules, such as crossing a double yellow line to bypass a permanently blocked lane.
Cross-City Deployment, Sensors, and Lightweight Mapping (12:30)
Because core driving behaviors are governed by real-time reasoning rather than localized data memorization, May Mobility has deployed its Level 4 platform across dozens of cities without needing site-specific model retraining. Crossman states that driving interaction logic transfers across regions, requiring adaptation for seasonal factors like heavy snow rather than core driving rules.
The hardware architecture combines cameras for semantic context, LiDAR for accurate spatial depth, and radar for instant velocity tracking of cross-traffic. Operationally, the platform avoids heavy, LiDAR-dense high-definition maps, relying instead on medium-definition maps that provide basic road network topology.
Crossman concludes that on-vehicle reasoning resolves the safety and cost roadblocks of autonomous driving without requiring gigawatt-scale data centers or endless mileage collection. While core interaction logic generalizes across cities and vehicle platforms, systems still require balanced multi-sensor hardware payloads and specific engineering adjustments for extreme conditions such as heavy winter snowfall.
## Background
该内容由 Ai4 频道于 2026 年 8 月 18 日发布于 YouTube 平台。根据页面简介，演讲者为 May Mobility 的自主研发高级副总裁 Jacob Crossman，他阐述了传统规则型与端到端自动驾驶模型的缺陷，并介绍了将深度学习与推理模型相结合的全新架构，旨在说明如何打造更安全、更具扩展性的自动驾驶系统。
## TL;DR
Jacob Crossman 质疑了仅靠堆叠数亿英里训练数据和庞大算力来解决自动驾驶问题的传统路线，指出端到端模型在应对未见过的边缘场景时存在隐患。May Mobility 提出了多策略决策系统（MPDM），通过在车辆端结合 Transformer 模型与实时在线仿真推理，实现了“即时思考”与透明可追溯的决策。该架构仅用约 2 万英里的人类驾驶数据即可实现强悍的零样本泛化能力，并已成功部署至全美多座城市。在问答环节，Jacob 还进一步解答了关于跨气候数据迁移、传感器三元融合方案及地图依赖度等实际应用问题。
## Atmosphere & style
演讲气氛专业且兼具批判性思考，Jacob Crossman 以沉稳且自信的姿态剖析了当前 AI 算力竞赛的狂热现状。他通过将复杂的自动驾驶架构与日常驾考、聊天机器人进化路径进行生动类比，使高深的技术路线之争变得通俗易懂。现场氛围兼具学术严谨性与产业落地的前瞻感，配合实测可视化的直观展示，传递出一种对技术理性与负责任创新（Responsibly Building AVs）的坚定信念。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=4s) 随着自动驾驶扩展与边缘场景增加，盲目追逐模型规模、数据量与算力的传统路线存在瓶颈，算力与能耗成本极高。
- [00:04](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=4s) 传统端到端学习依赖黑盒神经网络与海量行驶数据，但在遇到未见过的边缘场景时行为不可预测，缺乏泛化能力。
- [00:32](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=32s) 借鉴大语言模型引入 RAG 与思维链（Chain of Thought）的思路，May Mobility 提出了多策略决策系统（MPDM）。
- [00:32](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=32s) MPDM 架构在 Transformer 感知预测的基础上融合实时在线推理，可在车辆行驶中每秒运行数百至数千次仿真并即时决策。
- [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) MPDM 结合了机器学习与传统数学/规则模型：面对罕见路况由传统模型接管兜底，面对复杂互动则由机器学习无缝平滑处理。
- [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) 该混合架构大幅降低了数据需求，示范模型仅需约 2 万英里人类驾驶数据即可达到同等甚至更优的安全性能。
- [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) 相比黑盒模型，MPDM 架构的决策过程完全透明，事后可进行精准追溯与责任归因。
- [09:02](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=542s) 系统运行逻辑根据场景确定性动态调整：高不确定性场景由机器学习主导，规则明确场景由传统算法主导。
- [09:02](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=542s) 凭借强泛化能力，May Mobility 已将同一套通用自动驾驶模型直接部署至全美数十座城市，无需大规模重新训练。
- [13:23](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=803s) 核心驾驶决策逻辑具备跨区域与跨气候迁移能力，雪地等特殊环境积累的数据和数学模型可无缝复用。
- [13:23](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=803s) 硬件上采用激光雷达（精确定位）、摄像头（语义感知）与毫米波雷达（横向测速）的三元融合方案。
- [13:23](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=803s) 系统属于 L4 级自动驾驶，仅依赖中等精细度路网拓扑地图，无需激光雷达点云级别的超高精地图，利于快速部署。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=4s) AI 算力竞赛的瓶颈与自动驾驶的路线之争
May Mobility 的自主研发高级副总裁 Jacob Crossman 登台致辞。他拥有约 30 年的 AI 领域研究经验，其中 15 年专注于自动驾驶（涵盖早期的越野车辆到如今的载人上路车辆），并在 May Mobility 主导自主驾驶研发近五年。他以今年 xAI 推出的 Colossus 2 超级计算机集群切入话题，指出该设施造价估计超过 300 亿美元且仍在攀升，其耗电量甚至超过了 May Mobility 总部所在地密歇根州安阿伯市（Ann Arbor）的整座城市。当前 AI 领域的竞争盲目追逐“更大的模型、更多的数据、更强的算力”，据预测，到 2030 年美国约 5% 的电力将仅用于 AI 数据中心。
针对自动驾驶（AV）领域，当前最热门的技术路线之一是端到端学习（End-to-End Learning）——将传感器像素输入一个黑盒神经网络，直接输出油门、刹车和转向指令。然而这种路线极其昂贵且存在隐患：黑盒模型必须穷尽车辆可能遇到的所有场景，一旦面对未曾见过的边缘场景（Edge Cases），其行为便不可预测。这也导致业内不断追逐数亿英里的行驶数据训练。对此，Jacob 抛出了一个核心质疑：人类在密歇根州考取驾照只需开数百英里，成为熟练驾驶员也仅需一万到两万英里，为何自动驾驶却要耗费如此巨大的计算资源与数据量去模拟“人类智力”？
- **xAI Colossus 2 算力成本**：单个算力中心投资额已逾 300 亿美元，能耗超越安阿伯市全城用电量。
- **全国能耗预估**：预计到 2030 年，仅 AI 数据中心就将消耗全美 5% 的电力总量。
- **端到端模型痛点**：缺乏泛化能力，依赖穷举式数据灌喂（数亿英里训练），且无法保障未见场景下的安全性。
---
### [00:32](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=32s) 从 Chatbot 的进化汲取灵感：多策略决策系统（MPDM）
为了寻找更优解，Jacob 回顾了早期聊天机器人（Chatbots）的发展路径。早期大语言模型存在两个主要缺陷：一是无法处理需要深层推理和多步骤思维链的复杂问题，二是无法获取最新的现实信息。后来开发者通过引入检索增强（RAG）以及在推理阶段使用的思维链（Chain of Thought）推理，让系统能够在运行时推演多种可能并选择最优解。
May Mobility 将类似的思想应用到了自动驾驶领域，提出了**多策略决策系统（Multi-Policy Decision-Making, MPDM）**。该架构依然使用 Transformer 模型进行感知（Perception）、策略（Policy）和预测（Prediction），但在此基础上融合了实时在线推理机制。系统不仅依赖数据驱动，还能在车辆行驶过程中进行“即时思考”（Thinking on the fly），从而无需数亿英里的训练数据即可应对复杂路况。
```
+-------------------------------------------------------------------+
| Sensors (Camera / LiDAR / Radar) |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Transformer-based Perception |
| (Outputs Scene & Object Tracks) |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Policy Options Generation |
| (Learned ML Models / Mathematical Rules / Classical Models) |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Real-Time On-Vehicle Sim |
| (Evaluates 100s-1000s of Options / Second) |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Select Safe & Effective Action |
| (Executes 5 Times / Second, 5Hz) |
+-------------------------------------------------------------------+
```
- **MPDM 核心逻辑**：在车辆端实时评估多种候选策略，结合机器学习与传统数学/规则模型的优点。
- **超高仿真频率**：车辆在行驶中每秒运行数百到数千次仿真，每秒更新并执行决策 5 次（5Hz）。
---
### [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) 传统模型与机器学习的交织：边缘场景与仅 2 万英里训练数据
Jacob 深入剖析了这一混合架构在处理边缘场景时的独特优势。机器学习模型的短板在于未见过的场景，而传统规则与数学模型的短板在于极度复杂的环境。在 MPDM 架构下：
1. 当机器学习模型遇到未训练过的罕见路况时，传统模型会即时接管车辆（可能仅接管几分之几秒或两秒），引导车辆安全驶过。
2. 当传统模型达到逻辑极限时，机器学习策略则会无缝接替并平滑处理复杂互动。
这种互补机制带来三大突破：首先，数据需求量大幅降低，May Mobility 展示的示范模型仅使用了**约 2 万英里**的人类驾驶数据（同时巧用“从周围其他车辆的行驶轨迹中学习”这一技巧）；其次，具备极强的零样本（Zero-shot）泛化能力，能首考通过全新场景；最后，**推理过程完全透明**，事后可以精准追溯车辆作出某一决策的具体原因。
以避让停靠快递车并超越的实际案例说明：
- **常规机器学习假设**：预测对向来车（紫色轿车）与骑行者会各自靠边减速，并基于模仿学习直接尝试绕行。
- **May Mobility 的安全防护**：系统推演了对向紫色轿车可能采取激进驾驶（如向道路中间借道）的风险。由于模仿学习策略在此情况下可能存在碰撞风险，系统果断拒绝该策略，选择传统的减速等待策略；待风险消除后，再快速切回机器学习策略平滑超越。
- **训练数据对比**：相比同行数百万/数亿英里，仅用约 2 万英里人类驾驶数据即可达到同等甚至更优的安全性能。
- **决策透明性**：打破黑盒限制，所有选择均可解构与追溯，便于系统迭代与责任归因。
---
### [09:02](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=542s) 可视化实测展示与典型驾驶场景解析
现场播放了一段今年早期的实测驾驶视频，右侧展示了系统运行的简化可视化画面：
- **预测线段**：绿色与橙色线条代表对周围其他交通参与者（Agents）行为的预测，在道路平稳时线条收拢，在不确定性增加时展开。
- **模型贡献比例**：主车下方的状态条显示机器学习策略（蓝色部分）与传统策略的实时贡献比例。蓝色占比越大，说明机器学习策略主导程度越高。
在遇到多方交互、没有明确通行规则的复杂节点时，蓝色区域显著扩大，机器学习策略主导平滑驾驭；而在停下等 Stop 标志等规则明确的简单场景中，传统规则策略占主导。 Jacob 进一步列举了三种典型实测场景：
1. **狭窄空间会车**：需要机器学习准确预测对向来车的让行意图，实现像人类司机一样舒适平滑的贴边通过。
2. **规避与越线决策**：驾驶中最难的是判断何时打破标准作业程序（SOP）——例如何时该踩压双黄线绕过障碍物、何时该果断塞入狭窄的汇入车道，机器学习策略在这些类人决策上表现卓越。
3. **突发紧急情况与黄灯决策**：面对最后一刻变黄的信号灯或突发状况，系统无需复杂策略，而是依靠简单推理与极速响应，提供可重复、安全且稳妥的避险操作。
- **系统运行逻辑**：在不确定性极高时发挥 ML 的灵活性，在规则明确时发挥传统算法的稳定性。
- **规模化部署**：基于该架构，May Mobility 已将同一套通用自动驾驶模型部署至全美数十座城市，无需在每个新城市进行大规模重新训练或定制。
---
### [13:23](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=803s) 现场问答环节：跨气候迁移、传感器配置与地图依赖
演讲结束后进入提问环节，现场观众针对不同实际应用细节提出了问题：
- **关于寒冷气候（雪天）数据向温暖地区的迁移能力**：
观众询问在密歇根州安阿伯市雪地训练的车是否能直接在加州等地区运行。Jacob 回答称，绝大多数驾驶决策在不同地区是高度一致的。在密歇根州和明尼苏达州等北部站点积累的数据与传统数学模型均可无缝迁移。虽然雪地感知有其特殊性，但并不影响核心决策逻辑的通用性。
- **关于传感器硬件配置（Sensor Fusion）**：
现场有观众提问除了摄像头外还使用了哪些硬件。Jacob 透露 May Mobility 采用的是 **激光雷达（LiDAR）、摄像头（Camera）与毫米波雷达（Radar）** 的三元融合方案：
- **摄像头**：擅长语义理解与场景感知。
- **激光雷达**：提供精准的距离感知，作为安全兜底。
- **毫米波雷达**：擅长测速，对远距离及横向交汇车辆的速比感知至关重要。
- **关于改装旧车与地图依赖**：
针对是否能将该技术后装（Retrofit）到现存传统汽车上的问题，Jacob 表示目前公司暂未开展后装业务，但由于该技术本质上是纯软件架构，未来完全有可能适配更多第三方车型。在地图依赖方面，Jacob 确认该系统属于 **L4 级自动驾驶**，运行依赖“中等精细度地图”（Medium-definition Map）——不需要包含激光雷达点云级别的超高精数据，但需要具备基础的路网拓扑信息。
- **L4 级别地图策略**：依赖中精度路网地图（无需超高精点云），兼顾部署速度与安全性。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=4s) 黑盒模型必须穷尽车辆可能遇到的所有场景，一旦面对未曾见过的边缘场景（Edge Cases），其行为便不可预测。 — Jacob Crossman
> [00:04](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=4s) 人类在密歇根州考取驾照只需开数百英里，成为熟练驾驶员也仅需一万到两万英里，为何自动驾驶却要耗费如此巨大的计算资源与数据量去模拟“人类智力”？ — Jacob Crossman
> [00:32](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=32s) 大语言模型存在两个主要缺陷：一是无法处理需要深层推理和多步骤思维链的复杂问题，二是无法获取最新的现实信息。 — Jacob Crossman
> [00:32](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=32s) 系统不仅依赖数据驱动，还能在车辆行驶过程中进行“即时思考”（Thinking on the fly），从而无需数亿英里的训练数据即可应对复杂路况。 — Jacob Crossman
> [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) 当机器学习模型遇到未训练过的罕见路况时，传统模型会即时接管车辆（可能仅接管几分之几秒或两秒），引导车辆安全驶过。 — Jacob Crossman
> [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) 当传统模型达到逻辑极限时，机器学习策略则会无缝接替并平滑处理复杂互动。 — Jacob Crossman
> [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) 相比同行数百万/数亿英里，仅用约 2 万英里人类驾驶数据即可达到同等甚至更优的安全性能。 — Jacob Crossman
> [06:10](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=370s) 打破黑盒限制，所有选择均可解构与追溯，便于系统迭代与责任归因。 — Jacob Crossman
> [09:02](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=542s) 在不确定性极高时发挥 ML 的灵活性，在规则明确时发挥传统算法的稳定性。 — Jacob Crossman
> [09:02](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=542s) 驾驶中最难的是判断何时打破标准作业程序（SOP）——例如何时该踩压双黄线绕过障碍物、何时该果断塞入狭窄的汇入车道，机器学习策略在这些类人决策上表现卓越。 — Jacob Crossman
> [09:02](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=542s) 面对最后一刻变黄的信号灯或突发状况，系统无需复杂策略，而是依靠简单推理与极速响应，提供可重复、安全且稳妥的避险操作。 — Jacob Crossman
> [13:23](https://www.youtube.com/watch?v=Bh1e86jX6Bg&t=803s) 在密歇根州和明尼苏达州等北部站点积累的数据与传统数学模型均可无缝迁移。虽然雪地感知有其特殊性，但并不影响核心决策逻辑的通用性。 — Jacob Crossman
## Mentioned
Jacob Crossman, May Mobility, xAI, Colossus 2, Ai4, Ann Arbor, Michigan, Minnesota, California, RAG, Chain of Thought, MPDM, Transformer, LiDAR, Camera, Radar, SOP, L4
## Video description
Jacob Crossman - Sr. VP of Autonomy, May Mobility
Humans don’t need millions of miles of experience to drive safely, so why do we keep building AVs that do? As autonomous vehicles expand into more cities and edge cases multiply, chasing data is not enough to make our roads safer. I’ll explain how traditional rule-based and end-to-end AV models share a critical flaw: they can’t reliably handle situations they’ve never seen before. I’ll also show why pairing deep learning with reasoning models is the architecture that handles any situation and why that matters at scale. If you want to see how AVs will truly create a safer world, this presentation will take you through the how and why of building AVs responsibly.
[Source](https://www.youtube.com/watch?v=Bh1e86jX6Bg)
Source quotes
[0:04] Jacob Crossman: 黑盒模型必须穷尽车辆可能遇到的所有场景，一旦面对未曾见过的边缘场景（Edge Cases），其行为便不可预测。
[0:04] Jacob Crossman: 人类在密歇根州考取驾照只需开数百英里，成为熟练驾驶员也仅需一万到两万英里，为何自动驾驶却要耗费如此巨大的计算资源与数据量去模拟“人类智力”？
[0:32] Jacob Crossman: 大语言模型存在两个主要缺陷：一是无法处理需要深层推理和多步骤思维链的复杂问题，二是无法获取最新的现实信息。
[0:32] Jacob Crossman: 系统不仅依赖数据驱动，还能在车辆行驶过程中进行“即时思考”（Thinking on the fly），从而无需数亿英里的训练数据即可应对复杂路况。
[6:10] Jacob Crossman: 当机器学习模型遇到未训练过的罕见路况时，传统模型会即时接管车辆（可能仅接管几分之几秒或两秒），引导车辆安全驶过。
[6:10] Jacob Crossman: 当传统模型达到逻辑极限时，机器学习策略则会无缝接替并平滑处理复杂互动。
[6:10] Jacob Crossman: 相比同行数百万/数亿英里，仅用约 2 万英里人类驾驶数据即可达到同等甚至更优的安全性能。
[6:10] Jacob Crossman: 打破黑盒限制，所有选择均可解构与追溯，便于系统迭代与责任归因。
[9:02] Jacob Crossman: 在不确定性极高时发挥 ML 的灵活性，在规则明确时发挥传统算法的稳定性。
[9:02] Jacob Crossman: 驾驶中最难的是判断何时打破标准作业程序（SOP）——例如何时该踩压双黄线绕过障碍物、何时该果断塞入狭窄的汇入车道，机器学习策略在这些类人决策上表现卓越。
[9:02] Jacob Crossman: 面对最后一刻变黄的信号灯或突发状况，系统无需复杂策略，而是依靠简单推理与极速响应，提供可重复、安全且稳妥的避险操作。
[13:23] Jacob Crossman: 在密歇根州和明尼苏达州等北部站点积累的数据与传统数学模型均可无缝迁移。虽然雪地感知有其特殊性，但并不影响核心决策逻辑的通用性。
Source quotes
[0:14] Jacob Crossman: In Michigan... you have to drive for a few hundred miles before you can get a driver's license. Now, how much longer do you need to drive in order to become a good driver? Say maybe 10,000 miles, maybe 20,000 miles.
Original transcript
[0:04] Again,
[0:04] my name is Jacob Crossman from May Mobility.
I have spent about 30 years uh in different AI
[0:10] fields. About 15 of those years I spent uh dealing
with auto autonomous vehicles, first off-road and
[0:17] then on-road vehicles. And I've been with May
for about five years where I lead the autonomy
[0:22] efforts. And today I'm going to talk to you
about what it takes to make AV technology scale.
[0:30] So, um, as you all probably have heard, XAI has
launched, uh, their Colossus 2 earlier this year
[0:35] to accelerate the development of their XAI product
and their full self-driving. Um, the cost of this
[0:41] facility is estimated to be over $30 billion and
that just keeps going up. And this is just one of
[0:46] many. Uh, as the last talk talked about, there's
quite a few uh, of these data centers going out.
[0:51] Um, this type of facility uses more power than
the city where May is uh located in Ann Arbor,
[0:58] Michigan. And this growth in computation and the
data that it supports is feeding the AI boom.
[1:06] The race is now for bigger models, more data, and
more compute. Backed by theoretically the scaling
[1:13] laws which say that your your system should
get better as you add more data and compute
[1:17] and for the most part by the results
we're seeing in the real world. By 2030,
[1:21] it is projected that about 5% of all power
used in the US will go to just AI data servers,
[1:28] another 5% for other data
servers. So, it looks like
[1:32] um the matrix is going to turn out to be
true. We're just power plants for the AI.
[1:40] So, let me now focus in a little bit on uh
self-driving technology. One of the trendiest
[1:44] ways to build AVs today is to take the pixels or
the sensor data, put them into a big black box
[1:50] neural network and output gas, brake, steer. This
is called end-to-end learning. And there's various
[1:54] forms of this, some of which are more endto-end
than others. And the attraction of this approach
[2:00] is is that if you can get a big enough fleet
or collect enough data, you can let those big
[2:05] data centers do the hard work for you and output
what the car should do in different situations.
[2:10] Now the catch is is that that this approach
is very expensive. Why is it expensive? Well,
[2:15] you have to make sure that you train in all the
scenarios the car could uh come come up against.
[2:21] Any scenario that the car has not seen before
or something very similar um there's a risk
[2:25] that the car could do something that you can't
anticipate. And because of that uh people are
[2:30] always looking for adding more miles, hundreds
of millions of miles uh and more scenarios.
[2:38] And I asked this question right now. Well,
actually, just give me a second here.
[2:44] So, it works kind of, but there's this
question that I want to ask you all.
[2:48] How many miles did it take you to
learn to drive? Well, in Michigan,
[2:54] where my children just learned to drive not that
long ago, you have to drive for a few hundred
[2:57] miles before you can get a driver's license. Now,
how much longer do you need to drive in order to
[3:04] uh become a good driver? say maybe 10,000 miles,
maybe 20,000 miles. And so the question becomes,
[3:12] is this really a good thing that we're
building these huge data centers to
[3:15] try to train our cars to drive? Is
this really humanlike intelligence?
[3:19] Can or should we keep scaling like
this? Or is there a better way?
[3:27] The answer to these questions, to answer them,
I want to go back to chatbots for a minute and
[3:31] I want to think about the very beginnings of chat
bots. At those times there were two problems when
[3:35] when first chat GPTs and those types of tools came
out. The first problem was that those tools um
[3:42] they they could not answer questions that required
deep reasoning. Anything that required uh problem
[3:47] solving or chain of reasoning. They also couldn't
keep up with recent information. They could only
[3:52] tell you older things. So developers began
working on techniques to solve these problems.
[3:58] The first of those was they added the ability
to retrieve uh current information and add that
[4:02] to the context that was being reasoned about.
The second thing and maybe more importantly
[4:06] they added chain of thought reasoning which
allowed the systems at inference time to think
[4:11] through different options uh that that could be
relevant to the answer and pick the best one. And
[4:16] at May we bring a similar approach to AVs. Um we
call our approach uh multi-policy decision-making.
[4:24] It's kind of an academic word that means uh we're
thinking on the fly. We're thinking as the car is
[4:30] moving. We do like data. We train our models
using transformers for perception, for policy,
[4:37] and uh for prediction, but we augment that with
this multi-policy decision-m and what that's
[4:43] allows us to do is build a system that can reason
through complex situations without requiring
[4:49] millions of miles of driving. So, how does this
work? So it starts with an informationririch model
[4:57] and a trained system that can produce outputs
like tracks and objects in the world. This
[5:03] we we get a scene out of this and the scene is
nothing special. It just tells us what's going on
[5:07] in the world. But unlike end toend systems that
operate like black boxes, what we do is we take
[5:13] and we generate options for the car to execute
at that time. We call each of these options
[5:19] policies and we explicitly propose them and they
can come from anywhere. A lot of them do come from
[5:24] learned models. They come from learned uh policies
and uh other techniques but some of them come
[5:30] from more mathematical models or more rules-based
models. They come from many different places
[5:34] wherever we think we can get good driving
behavior. Now the critical part is that the system
[5:39] in real time plays these out in the computer
on the car. So it simulates them in real time,
[5:45] hundreds of them per fraction of a second,
thousands of them per second. And from that
[5:50] simulation, it's able to determine what might
happen if it takes those actions with the way the
[5:56] environment is set up right now. And then it picks
the options that uh are safe and make progress.
[6:03] This happens all the time on the car five
times a second. So why is this important?
[6:10] The current limitation of machine
learning for driving is the edge cases.
[6:16] Um, we need an enormous amount of training data
to catch all of the edge cases. But our blend
[6:22] with our blend of learned and classic models,
whenever a learned model reaches a situation it
[6:27] hasn't been trained on, the classic models tend to
take over and drive the car. And that might only
[6:31] be for a fraction of a second or two seconds, but
it's able to to find its way through. Similarly,
[6:37] we all know that there's limitations
to the mathematical and classic models.
[6:40] Whenever those models reach their limits, maybe in
a very complex situation, the learning takes over
[6:45] and it drives the car through that situation.
And this has many important advantages. So,
[6:49] we first is we can train our models on a fraction
of the data that our competitors uh uh use.
[6:55] In fact, the models I'm going to show you videos
of here in a couple seconds uh were trained on
[6:59] just about 20,000 miles of human data. Although,
we do apply a trick. We actually don't just learn
[7:04] from our car, we also learn from all the other
cars that we see around us. Um, we can also drive
[7:09] through many situations that we're seeing for the
first time. We don't have to train on everything
[7:13] first. We're able to get through them through a
combination of interleing of our different models.
[7:17] And finally, our reasoning is transparent. So
when our system does takes an action, we can
[7:22] go back and understand why it took that action.
And that is very useful whenever the car does
[7:27] something that we don't expect, whether it's good
or bad. And we can understand why it did that.
[7:33] To help you understand the approach, I'm going
to show an example here. I'm going to focus on
[7:37] the left to start. The situation on the left, we
have a May vehicle, that little white minivan,
[7:42] and uh it's approaching a delivery truck with a
delivery person who's going off. You've all seen
[7:46] this kind of thing before. Well, a bicyclist
and another car come in on the oncoming lane.
[7:51] So, a train system might output an action that
it thinks is most likely. Most likely, that
[7:56] cyclist is going to hug the side of the road or
maybe turn off. and then the car that's the purple
[8:00] car that's coming down is likely going to veer
in lane or maybe slow down a little bit and pass
[8:05] inside of a machine learned system. It's likely
that that's the kind of outcome that would be
[8:08] expected and the system would then take from
its imitation learning for itself and say, "Hey,
[8:13] I should just smoothly go around that delivery
truck." However, the May vehicle plays out
[8:18] all the plausible actions that could happen,
including aggressive uh the the purple driver
[8:23] being aggressive. What if they swing out wide and
fill in some of that shared space in the middle?
[8:28] In this case, the policies that we learned from
imitation of humans would potentially collide
[8:33] with that purple vehicle or at least cause a
conflict. So, the system rejects it and selects
[8:38] a more traditional policy that just slows
the vehicle and stops and waits. However,
[8:42] as the scenario gets more uh clear as time goes
on, it will quickly switch back to the learn
[8:48] policy and move past the delivery truck. This
happens over and over again as the car drives.
[8:59] Just want to make sure the video plays.
[9:02] There we go. So, um, this video shows
a drive from earlier in the year.
[9:08] On the right is a simplified visualization of
how our technology works. So, you're going to
[9:11] see some green and orange lines and sometimes
they cluster together and sometimes they
[9:15] fan out. Those are our predictions we're
making for the other agents in the world.
[9:19] And this is kind of how the world works. when
you're driving there's mo there's long stretches
[9:24] of fairly confident uh actions and then every
so often there'll be this period of uncertainty
[9:29] um it's actually when the uncertainty is highest
that you mo most need the learning components
[9:35] so the bar under the main vehicle shows
the contribution of learned policies
[9:39] to the uh classical policy so the more blue it is
the bigger that blue bar the more ML policies are
[9:45] taking over and driving the car less blue
means that it's more traditional policies
[9:51] There's no surprise that we see the learn policies
playing a bigger role when the complex situations
[9:56] arise. We focus a lot of our training examples on
these cases where multiple entities interact and
[10:00] the correct driving behavior is not obvious
and so they take over in those cases and we
[10:04] want the vehicle to behave smoothly in those. But
notice the classic policies rarely ever completely
[10:09] go away. They're used to drive the car in simpler
situations such as when rule following is required
[10:14] like when you want to stop at a stop sign. Do I
need to learn that? Not really. and to help bridge
[10:18] short gaps, maybe even fractions of a second
where the learn policies don't do the right thing,
[10:22] so we ignore them and do these other policies.
This is all pretty new stuff, and we're learning
[10:27] every day um how these two can combine to
uh form very interesting driving behaviors.
[10:35] This approach can handle all sorts of situations
that show up in driving. Here are some situations
[10:39] we've encountered. So, narrow space passing
can be very tricky. Learn policies are needed
[10:46] to accurately predict what oncoming vehicles are
going to do and the yield behavior that they may
[10:51] uh execute. Machine learn policies help to
navigate the space comfortably and smoothly
[10:56] in ways that a human would find comfortable.
Um but you can't just do this at any time.
[11:03] The vehicle has to consider vehicles behind and
adjacent even when simpler passing policies are
[11:09] sufficient. In fact, a lot of the difficulty
in driving is figuring out when to stick with
[11:13] your standard operating procedures and when
to break those standard operating procedures.
[11:17] When to squeeze into a tight merge window or
when to wait for a bigger one, when should
[11:21] I cross over that double yellow line because
there's no other way I'm going to make progress.
[11:24] When should I stay where I am? And this is where
machine learn policies really excel. They help
[11:29] us understand how to drive like a human and do
these things uh the way a human would do them.
[11:37] So, um, but it's not always obvious complexity.
Sometimes simple lane changes can be complicated
[11:45] by adjacent vehicles and that yellow light that
appears at the last moment and the car has to
[11:52] make decision. What should I do? Should I stop
or should I go? And then there's the unexpected.
[11:59] In many of these cases, you don't need fancy
policies. You just need simple reasoning
[12:03] and quick responses. You want responses to be
proven. You want responses that are repeatable
[12:10] and responses that are safe. So you see that our
technology which combines both the best of both
[12:16] learning from humans and from observing the
world along with online reasoning in the car
[12:22] leads to safe and comfortable driving in
a wide range of real world situations.
[12:30] Now, all of this that you're seeing was developed
again in about 20,000 miles of driving data at a
[12:34] fraction of the overall cost of anyone else in
the industry. We have deployed deployed these
[12:40] same models to dozens of cities and uh and we
always have from the very beginning of May.
[12:48] The ability to uh deploy these general purpose
AV capabilities to different cities without
[12:53] extensive retraining or customization
for each site is one of our strengths.
[12:59] And um by learning from both past
drives and reasoning in real time,
[13:04] this makes our approach and technology uniquely
suited to scale to move from place to place
[13:09] and to drive safely and provide uh uh services
to each community. We are really excited at the
[13:15] opportunities this technology is enabling in the
coming months and years. Thank you. [applause]
[13:23] So my question would be like for example a car
trained on an arbor with uh winter conditions
[13:30] will it react the same way if it is
driving in California or somewhere else?
[13:35] Is that considered during this training? Um so
it is important to train in different types of
[13:41] conditions. However, most of the driving decisions
are going to be very very similar between the two.
[13:46] Obviously, if there's snow on the ground, that's
different. But I know there's a lot of people
[13:49] have a misperception that Michigan is constantly
snow. It's not. There's a few days from Michigan.
[13:54] Excellent. Uh, welcome. But, um, but yeah, I there
there are some things that are different, but
[14:00] think about it this way. If you have some training
data from northern sites, it should transfer to
[14:04] other northern sites. We, for example, operate in
M Minnesota and in Michigan, and a lot of the data
[14:09] that you get from each transfers, as do a lot of
the classical mathematical models that we run. You
[14:16] have a lot criticiz where you upload about a
million videos a day into our cloud from the
[14:24] from the cab. So most of the things you showed
required the camera. Um I'm just wondering
[14:31] what other sensors technology required on
those vehicles to make your models work.
[14:37] So we use uh LAR, camera and radar and each
provides kind of a unique capability. Um,
[14:45] so cameras are especially good um, especially for
semantic interpretation of scenes. You can do a
[14:50] lot with cameras. Liar gives you that peace of
mind that you're going to know how far things are
[14:55] away from you so you can avoid them for safety.
And radar, of course, really helps you to get
[14:59] speeds that's especially useful for cross traffic
and further things. So, we use all three of those.
[15:05] As a follow-up questions, would you see this
technology as a retrofit for all existing cars?
[15:10] Um, we don't really right right now that's not
what we're doing, but the technology itself
[15:15] is softwarebased and could and we've already
deployed it in multiple vehicles. So, we think
[15:19] in the future there is the possibility that it
could be deployed on other vehicles uh than we
[15:24] have them on right now. Um, does the vehicle use
purely sensor based observations uh or perception
[15:31] or does it have a map as a fallback? The uh the
system that we're running is L4. So, it does
[15:37] use a map. I would call it a medium definition
map. It doesn't have a lot of detailed data like
[15:42] LAR details and things like that, but it does have
the basic route network of of where the roads are.
[15:48] All right, ladies and gentlemen,
let's give our presenter a round
[15:51] of applause. Thank you so much for being here.
