Zoox Deploys Distilled VLA Models to Expand Commercial Robotaxis

Navigating Scaling and Long Tail: VLAs, Data Generation, and Mapping Approaches in Autonomy — Ai4

Ai4

Physical fleet logging fails to capture rare edge cases at statistical scale, requiring autonomous operators to replace manual mapping and human triage with distilled multimodal foundation models and synthetic world generation.

AI-assisted summary of Navigating Scaling and Long Tail: VLAs, Data Generation, and Mapping Approaches in Autonomy — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published by Ai4 on YouTube on August 19, 2026, this presentation features Doron Portnoy, Senior Staff ML Engineer at Zoox, discussing autonomous vehicle scaling. The session explores how Zoox tackles rare long-tail perception and mapping challenges using Vision-Language-Action (VLA) models, generative synthetic data pipelines, and evolving Level 4 spatial mapping architectures.

## TL;DR Doron Portnoy outlines Zoox's four-pillar strategy for scaling Level 4 autonomous robotaxis across expanding metropolitan operational design domains. The talk details an onboard Vision-Language-Action (VLA) pipeline that combines visual foundation encoders with continuous flow-matching action heads to deliver low-latency trajectory control and adaptive reasoning. To solve data scarcity for safety-critical edge cases, Zoox leverages generative world models and neural scene editing to synthesize adverse weather and hazardous scenarios without manual re-labeling. Finally, the presentation breaks down a hybrid online/offline mapping strategy for navigating live road changes alongside autoencoder- and LLM-driven pipelines that automate re-simulation log sampling and fleet-wide issue triage.

## Atmosphere & style The presentation carries a polished, highly technical, and pragmatic tone delivered with steady engineering authority. Portnoy methodically guides the audience through sophisticated architectural diagrams, seamlessly connecting foundational machine learning theory to real-world robotaxi deployments and fleet metrics. The pacing feels rigorous yet accessible to autonomy practitioners, culminating in a focused, collaborative audience Q&A on real-world sensor artifacts and simulation fidelity.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=4s) Zoox has advanced its purpose-built, bidirectional robotaxi platform to commercial operations in Las Vegas and San Francisco, accumulating over 500,000 riders and 3 million autonomous miles. - [03:28](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=208s) Scaling Level 4 autonomy against long-tail edge cases is structured across four technical pillars: onboard driving models, generative data engines, scalable mapping, and automated validation and triage. - [04:47](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=287s) An onboard Vision-Language-Action (VLA) architecture processes multimodal sensor streams through an LLM decoder, using teacher-student distillation to deploy high-capacity reasoning within vehicle compute constraints. - [08:57](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=537s) Zoox trains in-house visual foundation encoders using a three-stage pipeline—unsupervised distillation from models like DINO and SAM, supervised multi-task fine-tuning, and on-vehicle student compression. - [10:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=617s) To prevent token explosion and context window bloating in multimodal LLMs, a hybrid fusion strategy uses mid-fusion compression for primary modalities alongside selective late-fusion pathways for safety-critical inputs. - [13:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=797s) Flow-matching action experts replace autoregressive discrete action tokens, generating continuous, non-quantized trajectory horizons in parallel with low latency. - [15:20](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=920s) The VLA reasoning head supports variable-length traces, executing deep chain-of-thought deliberation for complex scene negotiations while enabling zero-trace reflexive bypasses for emergency reactions. - [18:16](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1096s) Because real-world fleet logging cannot naturally capture high-risk long-tail events at scale, generative world models and simulation pipelines are necessary to synthesize targeted edge cases for training. - [24:36](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1476s) Generative weather transformation applies spatial conditioning to nominal driving logs, synthesizing adverse conditions like snow or flooding while preserving 3D bounding boxes to eliminate re-annotation costs. - [26:12](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1572s) A hybrid mapping paradigm combines high-precision offline fleet map priors with real-time online perception to navigate active construction zones and unmapped road changes safely. - [32:30](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1950s) Autoencoder-based log embedding and clustering allow Zoox to intelligently sample rare long-tail edge cases for re-simulation rather than replaying millions of redundant nominal miles. - [34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) Multimodal LLMs automate fleet issue triaging by parsing camera, LiDAR, and tracking logs into natural-language diagnostic summaries, removing human engineering bottlenecks.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=4s) Zoox's Commercial Milestones and Autonomous Vision

Doron Portnoy, a Senior Staff Machine Learning Engineer on the Foundation Model team at Zoox, introduces the company's core mission: engineering a "safer, cleaner and more enjoyable future." Reflecting on Zoox's unique vehicle architecture—a purpose-built, bidirectional robotaxi designed without steering wheels or pedals—he highlights the milestone of securing regulatory approval for commercial robotaxi operations in the United States.

Zoox has transitioned from localized testing to commercial deployments in major metropolitan areas, including Las Vegas and San Francisco, alongside ongoing testing programs across ten different cities nationwide. Over the course of these operations, the fleet has carried more than 500,000 riders and logged over 3 million autonomous driving miles, providing an extensive data foundation for scaling machine learning models.

- **Operating Locations:** Commercial service in Las Vegas and San Francisco, with active fleet testing across 10 US cities. - **Ridership & Fleet Mileage:** Over 500,000 passenger rides completed and more than 3,000,000 autonomous miles driven. - **Vehicle Architecture:** Purpose-built, fully bidirectional L4 robotaxi with no manual controls (no steering wheel or pedals).

---

[03:28](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=208s) The Four Technical Pillars for Scaling Autonomy

Deploying robotaxis at scale introduces severe operational challenges, primarily driven by the long-tail distribution of rare, complex edge cases. To structure the technical roadmap required for robust autonomous operations across expanding geofences, Zoox organizes its scaling strategy around four interconnected pillars:

```mermaid flowchart TD accTitle: Four Technical Pillars of Scaling Autonomy accDescr: High-level overview of the four core technical pillars at Zoox.

subgraph Pillars["Scaling Pillars"] P1["1. Onboard Driving Model<br/>(Foundation Encoders & VLAs)"] P2["2. Data Engine<br/>(World Models & Synthetic Generation)"] P3["3. Scalable Mapping<br/>(Hybrid Onboard/Offboard & World Changes)"] P4["4. Validation & Triage<br/>(Automated Testing & Fleet Triaging)"] end

P1 --> P2 P2 --> P3 P3 --> P4 ```

1. **Onboard Driving Model:** Multimodal Vision-Language-Action (VLA) architectures and visual foundation encoders built to reason through rare, safety-critical edge cases in real time. 2. **Data Engine:** Generative systems and world models designed to synthesize hard-to-capture driving scenarios for model training and counterfactual evaluation. 3. **Mapping Infrastructure:** Scalable hybrid onboard/offboard architectures capable of adapting dynamically to physical world changes and construction zones. 4. **Validation and Triage:** Automated infrastructure to validate model updates, benchmark rare-event performance, and triage fleet-wide disengagements or anomalies.

---

[04:47](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=287s) Onboard Driving Architecture: Teacher-Student VLA Pipeline

As autonomous fleets expand into new operational design domains (ODDs), long-tail scenarios appear at higher absolute frequencies. Traditional modular pipelines struggle to anticipate these rare combinations of agent behaviors and environmental clutter. Zoox addresses this by deploying an onboard Vision-Language-Action (VLA) framework.

```mermaid flowchart LR accTitle: Vision-Language-Action Architecture accDescr: End-to-end dataflow from raw multimodal inputs through encoders, fusion, LLM decoder, and task heads.

subgraph Inputs["Multimodal Sensor Inputs"] Sensors["Cameras, LiDAR, Radar,<br/>Audio, Ego State"] end

subgraph Processing["Representation & Reasoning"] Enc["Foundation Encoders"] Fuse["Multimodal Fusion Layer"] LLM["LLM Decoder"] end

subgraph Heads["Task-Specific Outputs"] Reason["Reasoning Head<br/>(High & Low Latency)"] Action["Action Head<br/>(Trajectories & Controls)"] end

Sensors --> Enc Enc --> Fuse Fuse --> LLM LLM --> Reason LLM --> Action ```

The system ingests raw data across diverse onboard sensor streams—including cameras, LiDAR, radar, audio, and vehicle ego-states. Each modality passes through a dedicated foundation encoder before being integrated via a unified fusion layer. The fused representations feed into a central large language model (LLM) decoder that powers two task-specific heads: - **Reasoning Head:** Designed to execute both high-latency deliberative reasoning (e.g., semantic scene interpretation) and low-latency safety-critical queries. - **Action Head:** Generates continuous output trajectories (waypoints) along with low-level actuation commands (steering, acceleration, and braking).

Because the full-capacity teacher model is too computationally heavy to run directly on vehicle compute hardware with real-time latency constraints, Zoox employs a teacher-student distillation strategy to compress the large VLA into an efficient, on-vehicle student model.

---

[06:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=393s) Visual Foundation Encoders and 3D Spatial Semantics

The visual foundation encoder serves as the perceptual backbone for the driving stack, requiring dense representations that balance 3D geometric reasoning with high-level semantic scene understanding. To validate representation quality across conditions, frozen encoder backbones are evaluated using lightweight linear probing heads across tasks such as object detection, semantic segmentation, classification, and depth estimation.

In benchmark evaluations across both nominal weather and severe environmental conditions (such as heavy rain, fog, and snow), the in-house visual foundation encoder outperforms standard open-source vision backbones like DINO.

When evaluating feature self-similarity by placing a query probe on a double-yellow lane divider: - **Open-Source Baselines (e.g., DINOv2/DINOv3):** Produce diffuse, noisy similarity activations across unrelated portions of the asphalt. - **Zoox Foundation Encoder:** Produces sharp, concentrated activations that trace the road geometry along the lane line precisely, despite not being trained explicitly on road geometry labels.

---

[08:57](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=537s) Three-Stage Recipe for Foundation Encoder Training

To develop an encoder with high representational capacity that can still operate under automotive-grade compute constraints, Zoox uses a structured three-stage training pipeline:

```mermaid flowchart TD accTitle: Three-Stage Foundation Encoder Pipeline accDescr: Training progression from label-free distillation to supervised fine-tuning and student compression.

S1["Stage 1: Label-Free Knowledge Distillation<br/>• Transfer knowledge from models like DINOv2 / SAM<br/>• Unsupervised training on unlabelled internal fleet data"] S2["Stage 2: Supervised Multi-Task Fine-Tuning<br/>• High-resolution multi-task heads<br/>• SOTA pseudo-labels + high-precision human annotations"] S3["Stage 3: On-Vehicle Student Distillation<br/>• Feature distillation + Logit distillation<br/>• Deployed directly to onboard vehicle compute"]

S1 --> S2 --> S3 ```

- **Stage 1 (Label-Free Distillation):** A single backbone absorbs visual representations from multiple state-of-the-art open-source foundation models (e.g., DINO, SAM) on unlabelled, fleet-collected driving data without requiring manual ground truth annotations. - **Stage 2 (Supervised Multi-Task Fine-Tuning):** The unified backbone is fine-tuned at higher input resolutions using multi-task perception heads (3D bounding boxes, dense segmentation, classification, and monocular depth) using a combination of pseudo-labels and human annotations. - **Stage 3 (On-Vehicle Compression):** The high-capacity model is distilled into a lightweight student model via joint feature-level and logit-level distillation for low-latency onboard inference.

---

[10:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=617s) Sensor Fusion Strategies: Mitigating Token Explosion

While standard vision-language models (VLMs) prioritize vision and text pairings, Level 4 autonomous driving strictly requires non-visual safety modalities, notably LiDAR and radar. Naively tokenizing raw, high-resolution sensor streams directly into an LLM context window causes severe scaling bottlenecks:

| Fusion Strategy | Architecture Mechanism | Advantages | Disadvantages / Risks | |:--- |:--- |:--- |:--- | | **Late Fusion** | Tokenizes all cameras, LiDAR, and radar streams directly into the core transformer context. | Maximum cross-attention across raw modality tokens. | Triggers "token explosion and context window bloating," causing unsustainable memory usage and high inference latency. | | **Mid-Fusion** | Feeds tokenized modality streams through an intermediate fusion network before routing to the LLM. | Dramatically reduces context length and computational burden. | Potential loss of fine-grained spatial and geometric cues during intermediate pooling/compression. | | **Hybrid Fusion (Zoox Approach)** | Combines mid-fusion compression for primary modalities with selective late-fusion pathways for safety-critical inputs. | Balances real-time compute efficiency with uncompromised safety margins. | Requires careful sensor-rate tuning and selective routing architectures. |

Zoox leverages its repository of over 3 million autonomous driving miles to learn joint cross-modal alignments between cameras, LiDAR, and radar, bridging modalities that off-the-shelf multimodal LLMs cannot natively interpret.

---

[13:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=797s) Action Head Paradigms: Discrete Tokens vs. Flow Matching

Translating autoregressive transformer activations into physical driving trajectories requires choosing an effective action decoding paradigm:

```mermaid flowchart TD accTitle: Action Decoding Comparison accDescr: Contrasting discrete auto-regressive token generation with continuous flow matching action heads.

subgraph Discrete["Paradigm 1: Discrete Action Tokens"] D1["Continuous Action Space"] --> D2["Grid Discretization / Bins"] D2 --> D3["Autoregressive LLM Generation"] D3 --> D4["Quantization Artifacts & High Latency"] end

subgraph FlowMatching["Paradigm 2: Flow Matching Action Expert"] F1["LLM Context / Latent Embeddings"] --> F2["Flow Matching Conditioning"] F2 --> F3["Continuous Trajectory Vector Field"] F3 --> F4["Parallel Horizon Decoding (Low Latency)"] end ```

- **Discrete Action Tokens:** The continuous control space is mapped into discrete spatial-temporal bins and represented as language tokens. While this enables unified cross-entropy training alongside language reasoning tasks, it introduces quantization artifacts on fine motor control and suffers from high latency due to sequential autoregressive decoding. - **Flow Matching Action Expert:** The LLM serves as a semantic conditioning context for a continuous flow-matching policy head. This generates smooth, non-quantized trajectories and control outputs across the entire planning horizon via parallel decoding, avoiding sequential token latency while preserving continuous precision.

[15:20](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=920s) Variable-Length Reasoning Traces and Low-Latency Trajectory Control

Rather than decoding vehicle actions token by token, the Vision-Language-Action (VLA) architecture outputs complete, full-horizon trajectories in just a few steps. This non-autoregressive trajectory generation ensures that downstream vehicle control remains smooth and continuous while eliminating the high computational latency typical of standard language model generation loops.

```mermaid flowchart TD accTitle: VLA Reasoning Head and Trajectory Output Paths subgraph S_IN["Input Perception"] IN["Multi-Modal Sensor Stream"] end

subgraph S_REASON["Adaptive Reasoning Head"] direction TB LONG["Long Reasoning Trace<br/>(Scene Understanding &rarr; Agent Prediction &rarr; Planning)"] SHORT["Short Trace / Zero-Trace Bypass<br/>(Emergency / Reflex Path)"] end

subgraph S_OUT["Control Execution"] WP["Smooth Trajectory Waypoints & Controls"] end

IN --> LONG IN --> SHORT LONG --> WP SHORT --> WP ```

To balance deep deliberation with real-time safety, the VLA incorporates an engineered reasoning head capable of producing dynamic reasoning traces of variable lengths. For intricate edge cases—such as multi-vehicle negotiation at an intersection—the model generates an extensive chain of thought that first interprets scene semantics, predicts specific agent behaviors, formulates a maneuver plan, and outputs refined waypoints. Conversely, when immediate action is mandatory (such as encountering a green utility tractor abruptly obstructing the lane), the reasoning head adapts on the fly: it either outputs an ultra-concise descriptive trace or completely bypasses the reasoning trace to pass raw waypoints straight to the vehicle controllers.

- Trajectories are predicted across the full planning horizon in only a few computational steps to maintain smooth, continuous motion control. - Deep reasoning workflows systematically progress through scene understanding, actor trajectory prediction, high-level planning, and waypoint generation. - Reflexive paths eliminate intermediate chain-of-thought generation entirely during low-latency, critical reaction windows.

---

[16:56](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1016s) Scaling Fleets, Geofence Expansion, and VLA Waypoint Execution

As autonomous fleets scale in size and expand into new operational geofences across diverse cities, long-tail edge cases transition from statistical anomalies to everyday occurrences. Rare encounters—ranging from road incident zones and active construction flaggers to flooded streets and horse-mounted police units—require the onboard VLA to act as a resilient safety layer capable of handling novel scenes that fall outside conventional rule-based heuristics.

Visualizations of the onboard VLA demonstrate real-time waypoint outputs overlaid in yellow onto live driving feeds. The system detects anomalous, safety-critical environments and dynamically adapts its driving style based on urgency. In fast-unfolding dynamic collisions, the model produces sharp, evasive waypoints to navigate around the hazard; in static aftermath scenarios where an accident has already occurred, the VLA chooses a measured, decelerated trajectory to smoothly maneuver past the obstruction.

- Geofence expansion brings edge cases into frequent nominal operations, necessitating generalized semantic reasoning. - Handled long-tail scenarios include crash incident zones, active traffic flaggers, horse-mounted police units, and localized road flooding. - Real-time VLA outputs translate directly into validated trajectory waypoints that govern physical braking, lane deviations, and passing maneuvers.

---

[18:16](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1096s) The Data Flywheel and the Long-Tail Simulation Imperative

To drive continuous performance gains across the VLA stack, Zoox operates a rigorous data flywheel. Models are initialized on existing driving datasets and pushed into large-scale simulation environments for stress testing. Simulation logs are automatically triaged to pinpoint critical performance gaps, prompting targeted queries into petabyte-scale historical fleet databases. When matching real-world edge cases are retrieved, they are labeled, packaged, and fed back into the training corpus.

```mermaid flowchart TD accTitle: Autonomous Driving Data Flywheel subgraph S_CYCLE["Continuous Learning Flywheel"] direction TB TRAIN["Train VLA Model"] --> SIM["Large-Scale Simulation & Validation"] SIM --> TRIAGE["Performance Gap Triage"] TRIAGE --> MINE["Database Mining (Real Fleet Logs)"] MINE --> LABEL["Data Annotation & Ingestion"] LABEL --> TRAIN end

subgraph S_SYNTH["Synthetic Generation Branch"] SYNTH["Generative World Models & Simulation"] -.->|"Fills Non-Existent Edge Cases"| TRAIN end ```

However, genuine long-tail events expose a structural flaw in mining-dependent flywheels: because these incidents are exceedingly rare, historical databases rarely contain enough samples to support statistical learning. Even when isolated edge cases are surfaced, their sample volume is insufficient for model convergence and can only serve as validation checks. Overcoming this data deficit requires high-fidelity generative simulation pipelines that manufacture targeted, controllable edge-case data for both training and validation.

- The standard data flywheel relies on closed-loop triage between simulation failures and historical log mining. - Real-world database mining breaks down on true edge cases due to the near-zero natural frequency of high-risk scenarios. - Synthetic generation is required to synthesize novel edge cases at scale for model training, rather than relying solely on passive data collection.

---

[19:52](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1192s) Multi-Modal Synthetic Data Generation and World Model Architecture

Generating high-fidelity autonomous driving data requires a blended pipeline of three distinct simulation modalities, used standalone or in hybrid workflows:

| Simulation Modality | Core Mechanism | Key Strengths & Use Cases | |:--- |:--- |:--- | | **Generative World Models** | Latent diffusion / spatiotemporal generative models with prompt and spatial conditioning | Generates end-to-end scenes from scratch or edits existing real-world drives while preserving underlying geometry. | | **City Simulation** | Classical 3D rendering engines with synthetic mesh/texture assets | Provides precise, millimeter-level 3D spatial control over object placement, rendered across multi-camera and LiDAR sensor rigs. | | **Physically Based Simulation** | Physics-based atmospheric and sensor degradation rendering | Simulates environmental scatter and optical occlusion, delivering fine-grained parameter control over weather like dense fog. |

```mermaid flowchart LR accTitle: High-Level Generative World Model Architecture subgraph S_INPUTS["Model Conditioning & Inputs"] direction TB NOISE["Latent Noise / Sensor State"] CTRL["Controls: Text Prompts, 3D Boxes, HD Maps, Routes"] end

subgraph S_LATENT["Core Model Engine"] ST_LATENT["Spatiotemporal Latent Space<br/>(Learns Physics, Motion Dynamics, and Scene Evolution)"] end

subgraph S_OUTPUTS["Joint Multi-Modal Generation"] direction TB CAM["Synchronized 360 Camera Feeds"] LIDAR["Synchronized 360 LiDAR Point Clouds"] end

NOISE --> ST_LATENT CTRL --> ST_LATENT ST_LATENT --> CAM ST_LATENT --> LIDAR ```

At the center of this generative framework lies a spatiotemporal latent space engineered to capture real-world physics and complex scene dynamics over time. During generation, the world model takes in latent sensor inputs alongside rich conditioning signals—including natural language prompts, bounding boxes, HD road maps, and planned vehicle routes. The model simultaneously synthesizes temporally coherent 360-degree camera imagery and matching LiDAR point clouds, ensuring unified spatial truth across the entire virtual sensor platform.

---

[22:28](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1348s) High-Risk Object Insertion, Scene Editing, and Multi-Sensor Consistency

Collecting real-world data for extreme road hazards—such as pedestrians lying in traffic lanes or structural debris scattered across a highway—is virtually impossible through standard fleet logging. Staging such events in physical test tracks presents unacceptable safety hazards and logistical constraints. Consequently, multi-modal simulation is the only safe, scalable pathway for exposing perception networks to severe road anomalies before public deployment.

To maintain sensor consistency, insertion workflows employ both classical rendering and neural scene editing. Using 3D asset pipelines, synthetic meshes of debris or fallen pedestrians are rendered jointly into camera images and LiDAR point clouds, aligning visual edges with laser depth returns. Alternatively, world model scene-editing replaces existing vehicles in real logs with rare agents, such as mounted police officers common in specific urban geofences. This neural replacement alters only the target object while leaving the ambient background, lighting, and peripheral traffic intact.

- Rare and dangerous assets (e.g., fallen pedestrians, lane debris) cannot be safely staged or mined at scale in the physical world. - Sensor consistency is strictly preserved across multi-view camera feeds and LiDAR returns during asset insertion. - Generative world models support selective object replacement, seamlessly swapping nominal road vehicles with rare entities like horse-mounted police units.

---

[24:36](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1476s) Extreme Weather Synthesis: Conditioned World Models and Physical Fog

Relying on fleet vehicles to physically chase severe storms is financially prohibitive and operationally inefficient. More critically, the probability of capturing compound edge cases in real life—such as an unmapped obstruction occurring simultaneously during a torrential storm—approaches zero. Synthetic weather augmentation resolves this data scarcity by taking nominal, sunny driving logs and converting them into adverse weather conditions through prompt-guided world models.

```mermaid flowchart TD accTitle: Conditioned Weather Transformation Pipeline subgraph S_BASE["Nominal Base Log"] LOG["Sunny / Nominal Fleet Drive<br/>(Pre-Annotated Dynamic Objects & HD Maps)"] end

subgraph S_GEN["Conditioned Generation Engine"] PROMPT["Adverse Weather Prompting:<br/>Snow, Sandstorm, Flooding, Heavy Rain"] COND["Spatial Bounding Box Conditioning<br/>(Locks Actor Positions & Trajectories)"] end

subgraph S_SYNTH["Augmented Synthetic Data"] OUT["Augmented Multi-Modal Scene<br/>(Camera Water Scatter + LiDAR Backscatter)"] REUSE["Zero Re-Labeling Cost:<br/>Original Annotations Remain 100% Valid"] end

LOG --> S_GEN PROMPT --> OUT COND --> OUT OUT --> REUSE ```

A crucial aspect of this weather generation pipeline is spatial conditioning: the 3D bounding boxes and coordinates of vehicles and pedestrians remain fixed in place while the environment around them is modified to show snow-covered asphalt, flooded lanes, or sandstorms. Preserving spatial positions eliminates the need for downstream re-annotation, circumventing an expensive and time-consuming labeling bottleneck. For fog simulation, generative AI is paired with physical scattering simulations to realistically degrade both 360-degree optical cameras and LiDAR point returns with accurate atmospheric backscatter.

- Base logs recorded in clear conditions are modified into snow, localized floods, or sandstorms via language-guided world models. - Spatial conditioning preserves the exact coordinates of scene objects, saving substantial time and labeling expense. - Fog generation pairs physical optical simulation with generative neural models to replicate realistic sensor degradation across both LiDAR and camera modalities.

---

[26:12](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1572s) The Mapping Bottleneck and the Hybrid Online-Offline Strategy

Expanding autonomous operations into new metropolitan areas exposes clear limitations in traditional, purely manual High-Definition (HD) mapping workflows. Manual annotation across sprawling road networks is slow and cost-prohibitive. Furthermore, modern urban centers undergo constant infrastructure churn—frequent lane restriping, emergency construction closures, and daily utility work mean that static offline maps inevitably become outdated shortly after release.

```mermaid flowchart TD accTitle: Hybrid Online and Offline Mapping Architecture subgraph S_OFF["Offline Mapping (High Accuracy Prior)"] LOGS["Multi-Pass Fleet Runs"] --> OPT["Unconstrained Compute & Multi-Run Optimization"] OPT --> FLEET_MAP["Fleet HD Map (OTA Push)"] end

subgraph S_ON["Online Perception (Real-Time Resilience)"] SENSORS["Onboard Cameras & LiDARs"] --> DETECT["Real-Time Map Feature Detector"] end

subgraph S_FUSE["Onboard Mapping Fusion"] FLEET_MAP --> FUSION["Real-Time Map Discrepancy Engine"] DETECT --> FUSION FUSION --> SAFE_DRIVE["Safe Trajectory Execution Through Active Road Changes"] end ```

To balance geometric accuracy with dynamic adaptability, Zoox utilizes a hybrid mapping strategy that unifies offline and online mapping pipelines:

- **Offline Mapping Strengths:** Aggregates sensor data across multiple fleet passes without real-time compute limits, running deep optimization passes with strong structural priors to build high-precision base maps even when individual drives suffer from poor weather or visual occlusion. - **Online Mapping Strengths:** Runs lightweight map-element perception networks in real time on the vehicle, identifying active construction zones, lane shifts, and dynamic closures that deviate from the prior map. - **Over-the-Air (OTA) Synchronization:** When real-time systems detect persistent world changes, map updates are pushed over the air to keep the broader fleet aligned. - **Onboard Fusion:** Merges the offline fleet map with real-time perception detections, allowing the autonomous vehicle to navigate active road changes safely without stalling.

[29:19](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1759s) Dynamic Map Updates and Driving Through In-Flight Road Changes

The onboard perception stack allows autonomous vehicles to generate local spatial representations on the fly, enabling the vehicle to drive reliably even when encountering discrepancies with the prior map. In parallel to onboard driving, the system takes detected map discrepancies and pushes them back to the central infrastructure. Aggregating these local map detections across a vast fleet allows the backend engineering pipeline to optimize, rebuild, and verify specific localized map patches where physical road conditions have shifted. Once rebuilt, the updated map patches are pushed back out over-the-air to the entire fleet in an ongoing, continuous update loop.

Doron illustrates this capability with two real-world autonomous driving examples. In the first instance, the vehicle encounters a newly repaved road cutting directly through an active construction zone, safely maneuvering through the area without prior geometry. In the second instance, road crews completely repainted lane markings, but the onboard perception system reconstructed a local map in real time and navigated the updated lane layout without hesitation.

```mermaid flowchart TD accTitle: Fleet Map Update Cycle accDescr: Diagram illustrating onboard map discrepancy detection, back-office aggregation and rebuilding, and fleet-wide map deployment.

subgraph Onboard ["Onboard Vehicle"] A["Online Map Detection\n& Real-Time Driving"] end

subgraph Backend ["Back-Office Infrastructure"] B["Fleet-Wide Ingestion\n& Data Aggregation"] C["Patch Optimization\n& Map Rebuilding"] end

A -->|"Upload Map Discrepancies"| B B --> C C -->|"Push Updated Map Patches"| A ```

---

[30:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1833s) Scaling Challenges in Validation and Continuous Triaging

As autonomous driving deployments scale up in operational design domains and cumulative mileage, two primary bottlenecks emerge: large-scale validation and continuous issue triaging. Large-scale validation relies heavily on re-simulation. However, with massive repositories spanning millions of logged driving miles, re-simulating every single mile is both computationally prohibitive and redundant, given that the vast majority of day-to-day driving contains uneventful, uninteresting miles. To address this, the system requires smart sampling strategies to select the most informative segments for re-simulation.

In parallel, scaling the fleet causes a surge in logged edge-case events that require continuous triaging. Manually diagnosing why a vehicle exhibited an unexpected behavior demands specialized domain expertise and cannot scale linearly with human engineers alone. Automating this triage workflow with AI models becomes essential to sustaining fleet expansion without exploding engineering overhead.

- **Re-simulation Volume:** Evaluating full 3-million-mile datasets is cost-prohibitive without intelligent filtering. - **Redundancy Reduction:** Smart sampling filters out nominal, repetitive driving to prioritize safety-critical and long-tail events. - **Triage Scalability:** Automated AI pipelines replace human bottlenecking as fleet-level issue reports expand.

---

[32:30](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1950s) Autoencoder-Based Log Sampling and Edge-Case Discovery

To systematically uncover and prioritize rare scenarios for re-simulation, Zoox employs an autoencoder-driven log sampling pipeline. The framework trains an autoencoder on driving logs containing fused sensor tracks, HD maps, road context, and vehicle state data to reconstruct the log. Once trained, the decoder is discarded, leaving the encoder to transform every logged driving snippet into a rich, compact embedding vector.

By clustering these log embeddings across the database, the system produces a distribution consisting of large nominal clusters alongside numerous tiny, long-tail clusters. If logs were sampled uniformly or randomly, rare edge cases represented in the smallest clusters would be overlooked. By deliberately sampling heavily from the smallest clusters alongside a representative sample of nominal driving, engineers ensure high-fidelity validation of critical scenarios, such as maneuvering tightly around double-parked vehicles.

```mermaid flowchart LR accTitle: Log Autoencoding and Clustering Pipeline accDescr: Pipeline showing how raw vehicle logs are encoded into embeddings, clustered, and sampled for simulation.

subgraph FeatureEncoding ["Log Encoding"] A["Raw Driving Log\n(Tracks, Maps, States)"] --> B["Trained Encoder"] B --> C["Log Embedding Vector"] end

subgraph ClusteringSampling ["Discovery & Sampling"] C --> D["Latent Clustering"] D --> E["Rare Clusters\n(Edge Cases)"] D --> F["Dominant Clusters\n(Nominal Driving)"] E & F --> G["Smart Re-simulation Set"] end ```

---

[34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) Automated Event Triaging with Multimodal Large Language Models

Manual issue triaging requires complex contextual reasoning rather than simple object labeling. When an autonomous vehicle exhibits an unexpected behavior—such as over-yielding to a stationary pedestrian waiting near an intersection line—a human triager must inspect the raw sensor data, reconstruct the vehicle's intent, and synthesize a diagnostic summary describing the root cause.

To automate this cognitive workflow, Zoox leverages multimodal language models (MLMs / LLMs). The model ingests camera imagery, text metadata, LiDAR streams, tracking histories, and map data from the event log. Instead of outputting trajectory waypoints or motion controls, the model decodes the multimodal context into natural-language diagnostic text outlining the exact scenario and undesirable behavior. These natural language diagnostic outputs are then embedded and clustered, providing fleet managers with an aggregate, categorized dashboard detailing edge-case issue frequencies across the entire vehicle fleet.

- **Multimodal Inputs:** Incorporates synchronized camera feeds, LiDAR point clouds, onboard object tracks, and HD map context. - **Reasoning Output:** Produces human-interpretable text diagnostic summaries (e.g., "ego vehicle over-yielded to a stationary pedestrian"). - **Dashboard Clustering:** Groups generated triage descriptions into semantic clusters to measure systemic fleet behavior trends over time.

---

[36:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2193s) Audience Q&A: Weather Simulation, Control Waypoints, and Sensor Artifacts

During the closing question-and-answer session, attendees query several core technical details regarding simulation fidelity, model control outputs, and real-world sensor variances:

When asked how Zoox validates synthetic weather models to prevent training on inaccurate physical artifacts, Portnoy explains that they use a blend of generative and physics-based models. Because the fleet does encounter real-world precipitation in limited quantities, they benchmark synthetic weather against real captured rain and fog scenes. By running perception stacks and end-to-end models across both synthetic and real counterparts, they verify whether downstream vehicle behaviors match.

Another audience member asks whether the trajectory waypoints predicted by the vision-language-action (VLA) models include explicit timing and speed targets for vehicle braking. Portnoy clarifies that every predicted waypoint is paired with an associated timestamp and corresponding control commands, allowing the planning stack to translate paths directly into smooth acceleration and deceleration actuators.

Finally, an attendee raises the challenge of unit-to-unit hardware variances, environmental noise, and ghost detections in LiDAR sensors. Portnoy notes that across millions of autonomously driven miles, the perception pipelines have observed diverse hardware quirks, environmental reflections, and sensor noise profiles directly in real-world logs, enabling the training and validation pipelines to account for sensor artifacts robustly.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=4s) safer, cleaner and more enjoyable future — Doron Portnoy

> [04:47](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=287s) Because the full-capacity teacher model is too computationally heavy to run directly on vehicle compute hardware with real-time latency constraints, Zoox employs a teacher-student distillation strategy to compress the large VLA into an efficient, on-vehicle student model. — Doron Portnoy

> [10:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=617s) While standard vision-language models (VLMs) prioritize vision and text pairings, Level 4 autonomous driving strictly requires non-visual safety modalities, notably LiDAR and radar. — Doron Portnoy

> [10:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=617s) token explosion and context window bloating — Doron Portnoy

> [15:20](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=920s) Rather than decoding vehicle actions token by token, the Vision-Language-Action (VLA) architecture outputs complete, full-horizon trajectories in just a few steps. — Doron Portnoy

> [18:16](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1096s) However, genuine long-tail events expose a structural flaw in mining-dependent flywheels: because these incidents are exceedingly rare, historical databases rarely contain enough samples to support statistical learning. — Doron Portnoy

> [22:28](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1348s) Collecting real-world data for extreme road hazards—such as pedestrians lying in traffic lanes or structural debris scattered across a highway—is virtually impossible through standard fleet logging. — Doron Portnoy

> [24:36](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1476s) Preserving spatial positions eliminates the need for downstream re-annotation, circumventing an expensive and time-consuming labeling bottleneck. — Doron Portnoy

> [30:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1833s) Large-scale validation relies heavily on re-simulation. However, with massive repositories spanning millions of logged driving miles, re-simulating every single mile is both computationally prohibitive and redundant — Doron Portnoy

> [32:30](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1950s) If logs were sampled uniformly or randomly, rare edge cases represented in the smallest clusters would be overlooked. — Doron Portnoy

> [34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) ego vehicle over-yielded to a stationary pedestrian — Doron Portnoy

## Mentioned Doron Portnoy, Zoox, Ai4, Las Vegas, San Francisco, DINO, DINOv2, DINOv3, SAM, Vision-Language-Action (VLA)

## Video description Doron Portnoy - Senior Staff ML Engineer, Zoox

Deploying autonomous vehicles on public roads at scale demands high robustness of perception and mapping stacks. This includes handling rare, unpredictable "long-tail" scenarios - the ultimate stretch of perception systems. Overcoming these edge cases requires moving beyond traditional methods of perception toward a new generation of learned, scalable approaches. In this presentation, Doron Portnoy, senior staff ML engineer, Perception at Zoox, will take a technical deep-dive into how Vision-Language-Action (VLA) models are being architected and fine-tuned to interpret complex, unprecedented environments. Because training these intelligent models demands elusive, hard-to-capture data, he will also dive into how the industry is bridging the gap through innovative synthetic data generation and scene editing techniques. Last, Doron will anchor how spatial mapping approaches for L4 have evolved to unlock autonomous driving at scale.

[Source](https://www.youtube.com/watch?v=SmDB2LdX15Q)

中文简报

Zoox部署蒸馏VLA模型推进商业化Robotaxi规模化落地 实体车队的日常采集无法在统计学规模上覆盖罕见边缘工况，自动驾驶运营商亟须以蒸馏多模态基础模型和合成世界生成技术替代人工建图与人工问题归因排查。 Doron Portnoy概述了Zoox在日益扩张的都会区运行设计域（ODD）中规模化落地L4级自动驾驶Robotaxi的四大支柱战略。演讲详细拆解了车载视觉-语言-动作（VLA）管线，该管线将视觉基础编码器与连续流匹配动作头相结合，以实现低延迟轨迹控制与自适应推理。为解决安全关键长尾工况的数据稀缺挑战，Zoox利用生成式世界模型和神经场景编辑技术，在无需人工重新标注的情况下合成恶劣天气与突发危险场景。最后，演讲分析了用于应对实时道路变化的线上/线下混合建图策略，以及基于自编码器和大语言模型（LLM）构建的日志重仿真自动采样与全车队级问题排查分诊管线。 Zoox已将其专有设计的双向Robotaxi平台推进至拉斯维加斯和旧金山的商业化运营阶段，累计承载乘客超50万人次，自主行驶里程达300万英里。 (0:04) 攻克长尾边缘工况以实现L4级自动驾驶规模化，主要依托四大技术支柱：车载驾驶模型、生成式数据引擎、可扩展建图，以及自动化验证与排查分诊。 (3:28) 车载视觉-语言-动作（VLA）架构通过LLM解码器处理多模态传感器数据流，并采用教师-学生模型蒸馏技术，在严苛的车载算力约束下部署高容量推理能力。 (4:47) Zoox通过三阶段管线训练自研视觉基础编码器：基于DINO和SAM等模型开展无监督蒸馏、进行监督多任务微调，以及面向车载端的学生模型压缩。 (8:57) 为抑制多模态LLM中的token爆炸和上下文窗口过度膨胀，系统采用混合融合策略，对主要模态进行中融合压缩，并为安全关键型输入保留选择性后融合通路。 (10:17)

Reading summary

According to Zoox machine learning engineer Doron Portnoy, scaling commercial robotaxis across expanding urban operating domains requires replacing brittle, rule-based systems and manual testing with foundation model architectures. To resolve the long-tail distribution of rare driving hazards, Zoox structures its platform around vision-language-action driving models, generative simulation engines, hybrid mapping, and automated validation pipelines. Multimodal VLAs and Efficient Onboard Reasoning (4:47) Zoox deploys a teacher-student Vision-Language-Action (VLA) architecture to navigate complex urban interactions. The perceptual core relies on an in-house visual foundation encoder trained via a three-stage pipeline: label-free distillation from models like DINO and SAM on fleet data, supervised high-resolution multi-task fine-tuning, and compression into an on-vehicle student model. In linear probing evaluations, this encoder precisely traces lane-divider geometry where open-source backbones yield diffuse activations. To prevent the context-window token explosion caused by raw camera, LiDAR, and radar feeds, Zoox adopts a hybrid fusion strategy combining intermediate compression with selective late-fusion pathways. Downstream trajectory planning avoids discrete language tokenization, which Portnoy notes introduces motor quantization artifacts and high sequential decoding latency. Instead, Zoox uses a continuous flow-matching policy conditioned on multimodal embeddings to generate smooth, full-horizon trajectory waypoints in parallel. The architecture also incorporates an adaptive reasoning head: complex negotiations trigger multi-step scene deliberation and actor trajectory prediction, whereas immediate safety hazards bypass intermediate reasoning traces entirely for reflexive execution. Generative World Models for Long-Tail Scenarios (18:16) While Zoox's fleet has logged over 3 million autonomous miles and carried 500,000 riders across commercial services in Las Vegas and San Francisco, Portnoy emphasizes that real-world log mining cannot resolve extreme edge cases due to their near-zero natural frequency. To train and evaluate safety-critical behaviors without hazardous physical staging, Zoox supplements its closed-loop data flywheel with generative world models, city simulation engines, and physics-based atmospheric rendering. The generative pipeline unifies multi-view camera feeds and LiDAR returns within a spatiotemporal latent space conditioned on text prompts, HD maps, and 3D bounding boxes. This framework enables the insertion of synthetic hazard meshes—such as fallen pedestrians or roadway debris—and neural replacement of standard traffic with rare entities like mounted police units. For adverse weather, the system converts nominal sunny fleet drives into snow, torrential rain, or heavy fog. By locking 3D bounding box coordinates during neural restyling, spatial annotations remain intact without expensive re-labeling. Hybrid Mapping for Dynamic Infrastructure (26:12) Traditional reliance on static, manually annotated high-definition (HD) maps creates severe scaling bottlenecks as fleets expand into new metropolitan areas subject to continuous infrastructure churn. To balance geometric accuracy with dynamic adaptability, Zoox implements a hybrid mapping framework combining offline structural priors with onboard online perception. Unconstrained offline optimization merges multi-pass fleet sensor data to construct base HD maps, while lightweight onboard networks detect real-time deviations such as active construction barriers, shifted lanes, or freshly repaved asphalt. When vehicles encounter unmapped infrastructure shifts, real-time perception overrides the stale map prior to safely navigate through the zone. Simultaneously, onboard discrepancy detections are uploaded to back-office systems, which aggregate observations across the fleet, rebuild local map patches, and redistribute them via over-the-air updates. Automated Re-simulation Sampling and Event Triaging (30:33) Validating model updates across millions of recorded driving miles creates computational and labor bottlenecks, as re-simulating entire datasets is computationally prohibitive and dominated by nominal, uneventful driving. Zoox addresses this through an autoencoder pipeline trained on fused sensor tracks, road maps, and vehicle states. Discarding the decoder leaves a latent encoder that converts driving snippets into compact embeddings; clustering these vectors isolates small, long-tail clusters—such as tight maneuvers around double-parked vehicles—enabling targeted, non-uniform re-simulation sampling. To eliminate human bottlenecks in fleet diagnostics, Zoox applies multimodal large language models to automate triage. Rather than outputting vehicle controls, these models ingest event sensor streams, map context, and tracking histories to generate natural-language diagnostic summaries of anomalous behaviors, such as over-yielding to a stationary pedestrian. Semantic clustering of these text reports populates fleet dashboards to track systemic behavioral trends over time. Zoox's end-to-end framework aims to decouple autonomous fleet expansion from linear engineering labor by deploying foundation models across onboard driving, synthetic simulation, mapping, and triage. However, Portnoy notes that synthetic training pipelines remain bounded by the need to continuously benchmark generative models against real-world sensor quirks, environmental noise, and physical precipitation.

精炼阅读摘要

Zoox基础模型团队资深员工Doron Portnoy阐述了公司规模化推进L4级完全自动驾驶的核心技术路线，指出突破长尾场景必须依托车载视觉-语言-动作（VLA）模型、多模态合成数据引擎、混合制图架构以及自动化验证分流四大支柱。 商业化里程碑与四项技术支撑 (0:04) Portnoy指出，Zoox针对无方向盘和踏板的专用双向L4级Robotaxi架构，已在美国获得商业化运营许可，并在拉斯维加斯与旧金山提供商业服务，同时在全美10个城市开展车队测试。截至演讲时，其车队已累计服务超过50万名乘客，自主行驶里程突破300万英里。随着运营设计域（ODD）从局部测试走向城市级扩张，罕见复杂的长尾场景在绝对数量上显著增加，传统的模块化规则系统难以应对不可预测的环境与行为组合。为此，Zoox将技术路线系统化为四大支柱：负责实时处理安全关键边缘案例的车载驾驶模型、用于训练与反事实评估的数据引擎、适应现实道路变动的混合制图架构，以及处理车队级异常的自动化验证与分流基础设施。 车载VLA架构与特征蒸馏压缩 (4:47) 为兼顾高阶推理能力与车载计算芯片的严苛延迟约束，Zoox构建了教师-学生架构的车载VLA系统。输入端汇集相机、LiDAR、毫米波雷达、音频和自车状态，各模态经由专属基础编码器提取特征后在融合层整合，随后输入核心大语言模型（LLM）解码器。为了避免海量高分辨率传感器令牌导致上下文窗口爆炸，Zoox弃用将所有模态直接分词的后期融合（Late Fusion），转而采用对常规模态做中间融合压缩、对关键安全模态做选择性后期融合的混合融合策略。在输出端，系统摒弃了存在离散量化误差与串行自回归延迟的动作分词（Action Tokens），采用以LLM上下文为条件的流匹配（Flow Matching）策略头，在数步内并行预测完整时域的连续轨迹航向点及油门、转向、制动控制量。其推理头支持动态长度思维链：面对交叉路口博弈等复杂场景时执行深层因果推导，而在遭遇道路突然受阻等突发事件时，则直接短路或绕过推理链以极低延迟输出轨迹。此外，Zoox通过无标签自监督蒸馏、高分辨率多任务微调以及特征与Logit联合蒸馏的三阶段流水线，将庞大教师模型压缩为高效的学生模型部署上车。 多模态世界模型与极端长尾合成 (18:16) 针对现实车队日志难以采集极罕见长尾样本（如行人横卧车道、严重道路杂物或狂风暴雨中的突发障碍）的数据飞轮瓶颈，Zoox通过多模态世界模型构建反事实与边缘数据合成流水线。该框架整合了端到端时空隐空间世界模型、具备毫米级控制力的城市三维渲染仿真，以及用于模拟大气散射与传感器光学退化的物理仿真。研发人员利用文本提示词与空间三维边界框作为条件约束，能够在锁定动态参与者空间坐标的同时，将晴天采集的常规日志无损转换为暴雪、局部积水或沙尘暴环境，且原始三维标注完全复用，避免了二次人工标注的巨大开销。在物体层面，该系统不仅能通过三维管线将合成杂物同时渲染进全景相机与LiDAR点云以确保几何一致性，还能利用神经场景编辑技术将真实车流中的常规车辆无痕替换为骑警等罕见实体，为感知与规划网络提供物理不可复现的极端安全训练用例。 混合制图与动态道路重构闭环 (26:12) 针对传统完全离线高精地图人工标注成本高且无法应对城市基础设施高频变更（如突发施工改道、临时车道线重涂）的缺陷，Zoox确立了离线先验与在线感知相结合的混合制图架构。离线端利用不设算力上限的多次车队遍历与深度优化算法，建立高几何精度的底图；车载端则运行轻量化实时地图元素检测网络，动态捕捉物理道路相较于先验地图的偏差。两套信号在车端融合引擎中实时校验，使车辆即使在遭遇刚铺设路面的施工区或全新标线时，也能依靠局部即时地图生成无停滞通行。当车端检测到持续性的物理世界变动时，异常数据会被回传至云端后台进行车队级聚合，经重构与验证后的局部地图补丁通过OTA空中下载实时推送到全车队，形成闭环更新机制。 自编码聚类抽样与大模型自动分流 (30:33) 面对数百万英里测试日志带来的仿真验证算力瓶颈与海量异常工单，Zoox引入了基于无监督自编码器的数据聚类抽样与多模态大语言模型（MLM）智能分流体系。系统使用自编码器将包含传感器轨迹、地图与车态的行驶片段压缩为稠密隐向量，并在潜在空间中执行聚类。为避免随机或均匀抽样被占绝大多数的常规里程稀释，工程师重点对分布极其稀疏的长尾簇进行高权重定向重采样，确保双排违停绕行等关键博弈场景获得充分验证。对于偶发的车辆非预期行为（如对路口静止行人过度让行），Zoox部署多模态大模型综合摄入图像、LiDAR、轨迹历史与地图先验，自动输出自然语言根因诊断报告替代人工排查。这些诊断文本被进一步向量化归类并在看板集中呈现，帮助工程团队系统量化全车队行为缺陷演进。 Zoox通过VLA模型、可控合成世界模型、混合制图与自动化验证体系，为破解L4级自动驾驶的长尾安全困境构建了闭环工程路径。不过，演讲问答也表明，合成极端天气与真实物理传感器退化之间的保真度验证、各批次LiDAR硬件噪声与鬼影差异的实车适应，依然是模型部署落地过程中需要持续攻关并严密校验的现实挑战。

Detailed notes — zh

## Background 本内容由频道 Ai4 于 2026 年 8 月 19 日在 YouTube 平台发布，记录了 Zoox 基础模型团队资深主任机器学习工程师 Doron Portnoy 的技术演讲。视频主要探讨了大规模部署自动驾驶车辆时，如何通过视觉-语言-动作（VLA）模型、前沿合成数据生成以及端云混合建图技术来攻克感知与建图栈中的长尾极端场景挑战。 ## TL;DR 演讲围绕 Zoox 应对自动驾驶规模化与长尾挑战的四大核心支柱展开：首先介绍了车载端基于多模态基础编码器与流匹配动作头的 VLA 架构，以及兼顾深思与即时避障的长短思维链设计；其次阐述了结合生成式世界模型、3D 渲染与物理仿真的三位一体数据引擎，用于无损合成高危目标与极端天气；接着展示了在线检测与离线车队地图协同自愈的混合建图方案；最后分享了基于自编码器聚类的智能重采策略与利用多模态大模型实现全自动故障归因的验证体系。 ## Atmosphere & style 整场演讲呈现出严谨扎实、务实求真的硬核工程与学术交流氛围。演讲者条理清晰、自信从容，从宏观商业化里程碑自然过渡到底层算法与系统架构细节，既有对技术权衡（Trade-offs）的坦诚剖析，也有对前沿落地的笃定。现场问答环节互动紧凑且直击技术痛点，让听众深刻感受到顶级自动驾驶团队在面对极其复杂的真实物理世界时所展现的系统工程底蕴与技术洞察力。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=4s) Zoox 作为全美首家获得商用许可的专用无方向盘、无脚踏板 Robotaxi 公司，已在拉斯维加斯和旧金山落地运营，完成超 300 万英里自主行驶并接待逾 50 万乘客。 - [03:28](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=208s) 应对自动驾驶规模化与长尾边缘场景的核心技术体系建立在四大支柱之上：车载驾驶模型、数据引擎、扩展建图与系统验证分类。 - [04:47](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=287s) 车载端引入多模态视觉-语言-动作（VLA）模型，并采用教师-学生模型蒸馏架构，将参数量庞大的离线云端大模型压缩后部署上车，兼顾复杂推理与计算效率。 - [06:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=393s) 自研视觉基础编码器具备兼顾高密度 3D 几何与高层语义的表征能力，在全天候多维基准测试及车道线无监督几何捕捉上表现显著优于 DINO 等开源方案。 - [08:57](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=537s) 基础编码器采用三阶段训练策略：先通过无标签自监督从 DINOv3、SAM 等蒸馏出通用主干，再做高分辨率多任务 SFT，最后以特征和 Logit 双重蒸馏生成车载轻量化学生模型。 - [10:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=617s) 针对多模态长上下文引发的 Token 爆炸与延迟飙升，Zoox 摒弃暴力的晚期融合，采用以中度融合为主、关键安全传感器受控挂载的混合折中融合架构。 - [13:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=797s) 在动作输出端，相较于自回归解码易产生量化伪影与高延迟的离散动作 Token，Zoox 选用以 LLM 为条件的流匹配动作专家头，实现全时域连续平滑轨迹的并行解码。 - [15:20](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=920s) VLA 的推理头设计支持动态长短思维链：复杂博弈场景展开多步意图预测与行为推理，突发紧急险情则直接触发短链甚至零延迟直通输出避障路径点。 - [19:52](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1192s) 针对真实路测难以采集的高危罕见长尾工况，构建集生成式世界模型、3D 资产城市仿真与物理基仿真于一体的数据生成引擎，实现极端场景与恶劣天气的可控合成。 - [23:56](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1436s) 在世界模型恶劣天气合成中引入严格几何条件约束，在不破坏既有标注坐标的前提下实现零成本数据复用，大幅削减高昂的重标注时间与费用。 - [26:12](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1572s) 规模化建图采用车云协同的混合建图架构：云端高精车队地图提供全局先验，车端在线建图实时应对施工封闭和车道重绘，并回传世界变更闭环更新全局图层。 - [32:30](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1950s) 为解决数百万英里路测全量重跑仿真算力不可承受的问题，基于自编码器日志嵌入向量进行聚类抽样，集中算力重点回放稀疏小簇中的高危长尾边缘场景。 - [34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) 引入多模态大模型（MLLM）全流程模拟资深排查专家认知，自动将异常行车事件解析并输出结构化自然语言归因报告，突破传统人工分诊的规模瓶颈。 - [36:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2193s) 物理生成天气场景的保真度通过真实黄金样本进行感知输出一致性与端到端决策行为一致性双重校验，输出轨迹点均严格绑定时间戳与动力学控制量以保证闭环执行。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=4s) 开场介绍：Zoox 的愿景与无人驾驶商业化里程碑 来自 Zoox 基础模型（Foundation Model）团队的资深主任工程师（Senior Staff ML Engineer）Doron Portnoy 向现场观众致意。他表示非常荣幸能在大会最后的环节与大家分享 Zoox 如何应对自动驾驶技术在大规模扩展（Scale）过程中所面临的长尾挑战（Long-tail）。Doron 强调，Zoox 的愿景是构建一个“更安全、更清洁、更有趣的未来世界”（a safer, cleaner and more enjoyable future），并在进入深度技术分享前播放了一段两分钟的品牌介绍短片。 随后，Doron 自豪地宣布了 Zoox 近期达成的重大里程碑：Zoox 成为全美首家获准开展商业化运营、且车辆专为无人驾驶定制（Purpose-built Robotaxi，车内完全没有方向盘和脚踏板）的自动驾驶公司。去年，Zoox 已在拉斯维加斯和旧金山正式落地商业化运营服务，并正在全美 10 个不同城市开展测试。截至目前，Zoox 已接待了超过 50 万名乘客，累计完成了超过 300 万英里的自主行驶里程。 - **商业化运营车型**：无方向盘、无脚踏板的专用 Robotaxi（Purpose-built）。 - **已落地运营城市**：拉斯维加斯、旧金山；另在全美 10 座城市展开测试。 - **里程碑数据**：接待乘客超 50 万人次，自动驾驶行驶里程超 300 万英里。 --- ### [03:28](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=208s) 技术架构总览：应对自动驾驶扩展挑战的四大支柱 随着运营规模的扩张、车队规模的扩大以及进入新的地理围栏（Geo-fence）与城市，自动驾驶系统遭遇长尾极端情况（Edge Cases）的频率显著上升。为了系统化解决扩展过程中的技术瓶颈，Doron 将 Zoox 的技术攻坚方向划分为四大核心支柱： ```mermaid flowchart LR accTitle: Zoox 扩展与长尾挑战四大支柱 subgraph P1["支柱一：车载模型"] A["车载驾驶模型<br/>(基础编码器 & VLA)"] end subgraph P2["支柱二：数据飞轮"] B["数据引擎<br/>(世界模型 & 合成数据生成)"] end subgraph P3["支柱三：高精地图"] C["扩展建图<br/>(端云协同 & 环境动态变化)"] end subgraph P4["支柱四：系统验证"] D["验证与分类分析<br/>(Validation & Triage)"] end P1 --> P2 --> P3 --> P4 ``` 1. **车载驾驶模型（Onboard Driving Model）**：基于基础编码器（Foundation Encoder）与多模态语言-动作模型（Multimodal Language-Action Model, VLA），帮助车辆在长尾极端场景中安全导航。 2. **数据引擎（Data Engine）**：利用世界模型（World Models）等前沿技术生成高质量数据，同时服务于驾驶模型的训练与离线评估。 3. **扩展建图（Mapping at Scale）**：探索如何结合离线建图与车载端上计算的混合架构（Offboard-Onboard Hybrid），并实时应对现实世界的动态环境变化。 4. **验证与分类分析（Validation and Triage）**：建立可规模化扩展的安全验证与故障诊断排查机制。 --- ### [04:47](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=287s) 支柱一：车载端 VLA 架构与教师-学生模型蒸馏 针对日益复杂的长尾事件，Zoox 的核心方案是在车载端集成视觉-语言-动作模型（VLA）。系统的输入端涵盖丰富的多模态传感器套件，包括环视相机（Camera）、激光雷达（LiDAR）、毫米波雷达（Radar）、音频信号（Audio）以及自车状态（Ego State）等。每种模态首先经过各自独立的基础编码器提取特征，随后输入联合融合层（Fusion Layer）进行特征交织，再送入大语言模型解码器（LLM Decoder）进行深度情境理解。 解码后的表征会被分发到两个专用任务头：一个是**推理头（Reasoning Head）**，专门设计用于同时处理高延迟的深度推断与低延迟的紧急反射事件；另一个是**动作头（Action Head）**，负责直接输出规划轨迹路径点（Waypoints）及制动、转向等具体控制指令。由于该全量 VLA 模型的容量过大、计算开销极高，无法直接部署在车载计算平台上，Zoox 采取了“教师-学生蒸馏”（Teacher-Student Distillation）架构：将庞大的云端/离线 VLA 作为教师模型，蒸馏压缩为一个紧凑高效的小型 VLA 模型后再部署上车。 ```mermaid flowchart TD accTitle: Zoox 车载 VLA 模型架构图 subgraph Sensors["多模态传感器输入"] S1["相机 / 激光雷达 / 毫米波雷达"] S2["音频信号 / 自车状态数据"] end subgraph Encoders["特征提取与融合"] E1["多模态基础编码器"] E2["跨模态融合层 (Fusion Layer)"] end subgraph Core["语言模型核心"] LLM["LLM 解码器 (Decoder)"] end subgraph Heads["下游任务头"] H1["推理头 (Reasoning Head)<br/>支持高/低延迟事件响应"] H2["动作头 (Action Head)<br/>输出 Waypoints 轨迹与底盘控制"] end Sensors --> Encoders E1 --> E2 --> LLM LLM --> H1 LLM --> H2 ``` --- ### [06:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=393s) 自研视觉基础编码器：几何与语义兼顾的表征能力 车载视觉基础编码器必须具备高强度的鲁棒特征，不仅要能稳定应对常规场景，更要在极端长尾场景下保持可靠。Zoox 自研的视觉基础编码器能够输出高密度的特征表征，同时赋能 3D 空间几何重构与高层语义理解。在评估该编码器的性能时，团队将其主干网络参数冻结（Freeze），仅在顶层接入轻量级探测头（Lightweight Probes）来执行 3D 目标检测、语义分割、图像分类及深度估计等下游任务。 测试雷达图显示，在涵盖晴天、大雾、暴雨、降雪等各种严苛天气的多维度基准测试中，Zoox 自研编码器在各项指标上全面超越了以 DINO 为代表的开源业界顶尖基础模型。此外，Doron 展示了一组相似度特征图对比：当在图像中的双黄虚线中心选择一个红点作为查询锚点时，DINOv3 计算出的全图相似度响应图模糊且充满噪声；而 Zoox 编码器生成的相似度图边界极其清晰锐利，精准勾勒出车道线的物理走势。Doron 兴奋地指出，这一结果非常令人惊喜，因为模型在预训练时并未针对道路几何特征进行显式监督，这充分证明了该编码器真正具备通用的基础泛化能力。 --- ### [08:57](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=537s) 基础编码器的三阶段训练策略 为了训练出兼具超大容量与极高精度的视觉基础模型，并最终成功压缩部署到车载芯片上，Zoox 建立了一套严谨的三阶段训练流程： ```mermaid flowchart LR accTitle: 基础编码器三阶段训练流程 subgraph S1["阶段 1：无标签知识蒸馏"] M1["开源基础模型<br/>(DINOv3, SAM 等)"] -->|无监督蒸馏| B1["单一通用主干网络<br/>(吸收多源知识)"] end subgraph S2["阶段 2：高分辨率多任务 SFT"] B1 -->|多任务头微调| SFT["有监督微调 (SFT)<br/>检测/分割/深度/分类"] end subgraph S3["阶段 3：车载轻量化蒸馏"] SFT -->|特征蒸馏 + Logit 蒸馏| STU["车载端学生编码器<br/>(部署上线)"] end ``` - **第一阶段（无标签知识蒸馏）**：从 DINOv3、SAM 等开源前沿视觉大模型中，将多源视觉先验知识蒸馏并汇聚到 Zoox 统一的单一大模型主干网络中。此过程完全基于 Zoox 内部采集的海量真实行车数据，以无标注（Label-free）自监督方式完成。 - **第二阶段（高分辨率有监督微调 SFT）**：在更高分辨率的图像输入下，接入检测、分割、分类、深度估计等多任务头（Multitask Heads）。训练数据结合了业界最先进的伪标签（SOTA Pseudo Labels）与高精全标注真实数据（Fully Labelled Data）。 - **第三阶段（端上学生模型蒸馏）**：将训练完毕的庞大基础模型作为教师，通过**特征级蒸馏（Feature Distillation）**与 **Logit 级蒸馏（Logit Distillation）**的双重约束，将表征能力迁移给轻量化车载学生模型，实现车载端的低延迟稳定运行。 --- ### [10:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=617s) 多模态融合难题：长上下文膨胀与融合路径取舍 尽管强大的视觉编码器是攻克长尾的关键，但对于 L4 级 Robotaxi 而言，激光雷达（LiDAR）与毫米波雷达（Radar）是绝对不可妥协的安全冗余基石。然而，现有的开源多模态大模型（MLM）基本都以“视觉-文本优先”（Vision-Text First）为主，缺乏对雷达点云等空间时序模态的原生支持。如果简单粗暴地将车身周围高分辨率相机与高清激光雷达的所有数据切片直接输入 LLM，系统会瞬间遭遇**Token 爆炸（Token Explosion）**与**上下文窗口膨胀（Context Window Bloating）**，导致车载计算平台的显存占用飙升、推理延迟失控，这在实时驾驶系统中是完全不可接受的。 为了解决多模态对齐与计算开销的矛盾，Zoox 依托累计超 300 万英里的自主驾驶多模态数据，对 LiDAR 和 Radar 进行模态对齐。在融合架构的设计上，Doron 深入分析了三种不同的演进路径： | 融合策略 | 实现机制 | 优势 | 面临的挑战 | |:--- |:--- |:--- |:--- | | **晚期融合（Late Fusion）** | 所有传感器独立编码为 Token 后，直接全部打平输入 VLA | 保留全量原始传感器信息，注意力机制全局可见 | 导致严重的 Token 爆炸与上下文膨胀，显存及计算延迟极高 | | **中度融合（Mid Fusion）** | 传感器数据初步 Token 化后，先经过中间融合层压缩提炼，再送入 VLA | 大幅精简输入 Token 数量，显著降低 LLM 计算负载 | 特征压缩过程中容易丢失细粒度的关键安全细节 | | **混合折中方案（Hybrid Method）** | 以中度融合为主干基底，挑选关键安全传感器进行受控的晚期融合挂载 | 兼顾计算效率与关键安全模态的高保真表征，实现两全其美 | 需要精心调整关键传感器的采样分辨率、帧率及通道优先级 | --- ### [13:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=797s) 动作头设计抉择：离散动作 Token vs. 流匹配连续轨迹专家 在完成多模态特征融合与 LLM 解码推理后，核心挑战转化为：**如何将 LLM 输出的抽象 Token 转换为车辆底盘能够执行的精确物理驾驶指令？** 业界目前主要存在两大主流设计范式： 1. **离散动作 Token 化（Discrete Action Tokens）**： - **机制**：将连续的车辆动作空间（如加速度、曲率等）进行离散化网格量化，将每个动作区间编码为特定的 Token（类似于自然语言词汇）。在推理阶段，LLM 以自回归（Autoregressive）逐词生成的方式逐步解码动作序列。 - **痛点**：离散网格量化会不可避免地引入**量化伪影（Quantization Artifacts）**，导致控制平滑度受损；同时，面对长预测时域（Full Horizon），自回归串行解码的高延迟特性极大地限制了在线响应速度。 2. **流匹配动作专家头（Action Expert with Flow Matching Head）**： - **机制**：由 LLM 负责提取高层环境上下文与语义意图，并将其作为条件约束（Conditioning），引导底层的流匹配（Flow Matching）连续生成头。 - **优势**：该架构摆脱了离散化量化的精度损失，能够针对未来长时域轨迹实现**全视界并行解码（Decode in parallel for full horizon）**，在大幅降低推理延迟的同时，输出平滑、高保真的连续物理控制轨迹。 ### [15:20](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=920s) VLA 模型的推理头与长短思维链设计 在自动驾驶中，动作解码的效率与平滑度至关重要。与传统语言模型按词逐个解码（token by token）不同，Zoox 的视觉-语言-动作（VLA）模型仅需极少数几步即可直接输出覆盖完整时域（full horizon）的连续轨迹，从而消除自回归解码带来的延迟，并为自动驾驶车辆提供极为平滑、连续的横纵向控制。 为了让系统在复杂多变的交通环境下兼具深度思考与即时反应能力，工程团队专门为 VLA 设计了一个自适应的推理头（Reasoning Head），使其能够根据场景危急程度动态生成不同长度的推理轨迹（reasoning traces）： - **长思维链（Long Reasoning Traces）：** 针对需要博弈与深层理解的复杂长尾边缘场景（例如两车在狭窄路口交互协商）。模型会首先对场景全局进行综合理解，随后对具体交通参与者（agents）的行为进行意图预测，接着执行运动规划，最终输出路径点（waypoints）与底层控制指令。 - **短思维链与零延迟直通（Short Reasoning Traces / Direct Output）：** 针对需要紧急避险的突发场景。例如前方突然出现一台绿色多功能拖拉机（utility tractor）死死堵住车道，系统必须在瞬间做出决断。此时模型只会输出极短的推理链（例如简明识别出“前方存在黑色/绿色拖拉机阻挡”），或者在极度追求超低延迟的极端危机下直接绕过推理文本生成，毫秒级直接输出规避路径点。 ```mermaid flowchart TD accTitle: VLA 模型自适应推理架构 accDescription: 展示 VLA 根据场景紧急程度在深思熟虑长链推理与低延迟快速决策之间的动态切换机制 subgraph InputStage ["环境感知输入"] A["多模态传感器输入<br/>(相机图像 / 激光雷达点云)"] end subgraph DecisionEngine ["VLA 自适应推理引擎"] B{"场景紧迫度评估"} C["长思维链推理 (Long Trace)<br/>场景理解 ➔ 意图预测 ➔ 行为规划"] D["短思维链 / 零推理直通 (Short/Zero Trace)<br/>极速生成避障意图或跳过文本"] end subgraph OutputStage ["轨迹与控制输出"] E["连续轨迹与路径点 (Waypoints)<br/>(无需逐 Token 解码，平滑连续)"] F["底层动力学控制执行"] end A --> B B -->|"复杂博弈 / 慢速交互"| C B -->|"突发阻挡 / 紧急避障"| D C --> E D --> E E --> F ``` 随着车队规模的扩大以及地理围栏（geofence）向全新城市的不断拓展，原本罕见的长尾极端案例正变得越来越普遍。Zoox 选择引入 VLA 来应对这些复杂边缘工况，包括突发车祸事故区（crash zones）、道路施工手旗员（flaggers）、积水道路以及骑警（horse-mounted police）巡逻等。在真实演示中，VLA 会在判定进入关键险情后高亮显示黄色路径点并迅速转化为具体动作：面对突发的动态碰撞，系统能瞬间大幅打舵变道避让；而面对已经发生、现场已被封锁的静态车祸残留，系统则从容提前减速慢行并平稳绕行。 --- ### [18:16](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1096s) 数据飞轮与长尾场景的生成式破局 要在车端成功部署高可靠性的 VLA 模型，核心在于必须提供源源不断的高质量训练与评估数据。Zoox 采用数据飞轮（Data Flywheel）工程方法推动模型持续演进：首先利用既有数据训练基础模型，随后将其置于大规模端到端仿真系统（large-scale simulation）中运行验证，并通过自动化分流与分类（triage）精准定位模型在特定场景下的性能断层（performance gaps）。 ```mermaid flowchart LR accTitle: 自动驾驶长尾数据闭环与仿真增强飞轮 accDescription: 说明在真实日志难以捕捉长尾工况时，通过混合生成仿真引擎补充训练数据的完整闭环 subgraph FleetLoop ["常规数据飞轮"] A["模型训练 (VLA Training)"] --> B["大规模仿真验证 (Large Scale Sim)"] B --> C["性能缺陷分类归因 (Triage Gaps)"] C --> D{"真实数据底座挖掘"} D -->|"检索到真实样本"| E["真值重标注并回流"] E --> A end subgraph LongTailEngine ["长尾合成与仿真引擎"] D -.->|"真实世界极罕见/缺失"| F["三位一体仿真生成引擎<br/>(世界模型 + 3D渲染 + 物理仿真)"] F -->|"合成长尾极端工况"| A F -->|"高难度评测集"| B end ``` 然而，这套传统飞轮在面临长尾场景时遇到了严峻瓶颈： - **底库匮乏与样本稀疏：** 极罕见的长尾事件在海量行车数据库中往往根本不存在；即便偶然录入了一两起片段，其数据量也过小，充其量只能作为单次验证集，完全无法支撑起模型监督微调（SFT）所需的训练规模。 - **采集与实兵演练不可行：** 许多高风险或反常场景根本无法在公共道路上人工偶遇，更无法人为在道路上排演真实危险。 - **双向仿真需求：** 自动驾驶团队必须拥有独立的数据生成与仿真引擎，在数据生成环节创造用于训练的合成样本，在模拟验证环节提供高保真的闭环测试环境。 --- ### [19:52](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1192s) 三位一体的数据生成范式：世界模型、3D 资产与物理仿真 针对长尾数据的生成难题，Zoox 建立了结合三种技术路径的数据引擎架构，既可单独运行，亦可混搭协同（mix and match）： 1. **生成式世界模型（World Model）：** 基于文本提示词（text prompting）与条件约束（conditioning）的端到端生成框架。世界模型既支持从零构建全新街景，也能够针对现有实车图像进行局部元素编辑与替换。其核心依赖于时空隐空间（spatiotemporal latent space）来捕捉环境底层的物理法则与时空动力学，确保生成的交互动态真实可信。 2. **三维城市仿真（City Simulation）：** 基于 3D 资产（assets）的经典图形学渲染管线。将高精度三维模型跨模态投影渲染到全车多个相机画面与激光雷达（LiDAR）点云中。该方式能够对障碍物的三维空间坐标、姿态实现极其细腻的毫米级精准控制。 3. **物理基仿真（Physically-based Simulation）：** 专用于精确还原特定恶劣天气与介质物理现象。例如针对特定光衰减和散射模型的浓雾生成，对自动驾驶视觉感知边界测试至关重要。 这套体系最强大之处在于支持跨范式融合：系统可以先在图像中经典渲染一个 3D 物体以确定几何绝对坐标，随后交由世界模型进行纹理重塑与光影自然融合；同时控制端输入文本提示、地图拓扑、目标路径规划与传感器噪点，协同联合生成 360 度环视全景相机图像与时空对齐的 3D LiDAR 点云。 --- ### [22:28](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1348s) 插入高危罕见目标与场景对象编辑 数据引擎的首要任务，是在传感器流中无缝注入现实中“不可寻、不可控、不可演”的超高危极端目标： - **横卧路面行人与散落杂物（Lying Pedestrians & Road Debris）：** 倒在地上的醉汉或伤员、散落在高速车道中的货物残骸极具致命危险，在现实中难以刻意收集，更不可能聘请演员去公路上躺卧实测。通过仿真管线，3D 资产被同步投射进相机画面与激光雷达点云，不仅纹理逼真，而且跨模态在几何几何尺度与空间反射率上完全保持时间与空间一致性。 - **局部对象无损置换（World Model Scene Editing）：** 依托世界模型局部编辑能力，系统可将原本平平无奇行驶在前方的一辆普通轿车，替换为特定运营地（例如旧金山特定辖区）真实存在的骑警编队。替换过程中，周围建筑、道路反射、光照条件以及整体背景环境丝毫不受破坏，只有目标区域的几何与交互动力学发生平滑改变。 --- ### [23:56](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1436s) 极端天气生成与零重标注的条件控制 出于严苛的安全性考量，Zoox 的车载自动驾驶方案采用了激光雷达与多机位环视相机互补的多模态冗余架构。然而在暴雨、暴雪等恶劣工况下，水汽喷溅、水滴附着以及激光束的水雾散射会引发各类传感器独特的物理失真（quirks），必须借助海量极端天气样本进行感知训练与基准验证。 现实中派车队四处“追逐风暴”（chasing storms）不仅耗资极其昂贵，而且当恶劣风暴与长尾突发事件（如暴风雨夜中突然出现横卧行人）发生重叠复合时，在物理现实中采集几乎成为不可能完成的任务。 针对这一困境，团队从数据库中现成的晴天名义日志（nominal logs）出发，通过提示词引导世界模型凭空生成大雪纷飞、沙尘暴或是道路严重积水汪洋的复杂场面： - **严格条件约束（Conditioning）避免重复标注：** 生成恶劣天气时，世界模型被施加了强条件约束，强制锁定画面中所有既有车辆、行人和车道线几何坐标保持不动。"This is extra important for us because if we don't then we need to relabel this data and relabeling is both costly and it's taking a long time."（这对我们极为关键，否则我们必须对生成数据重新做全套标注，重标注不仅代价高昂，而且极其耗时。）保持标注真值零成本复用，极大提升了数据流转速度。 - **物理机理与 GenAI 混合雾效：** 在浓雾模拟中，团队将物理散射模型计算与生成式 AI 渲染深度结合，使原本晴朗的实景采集日志能够在 360 度全景环视图像与激光雷达扫描点云中，呈现出物理上完全自洽、符合光波/激光衰减特性的逼真浓雾。 --- ### [26:12](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1572s) 规模化建图的挑战与混合建图架构（Hybrid Mapping） 在探讨完车载模型与仿真数据引擎后，演讲切入支撑规模化 L4 自动驾驶落地的核心基石——地图构建（Mapping）。 随着车队规模扩张至越来越多的城市与地理围栏，传统依赖人工精调离线高清地图（HD Map）的模式正逼近极限。不仅人力与时间成本高昂，现实世界道路的瞬息万变更让纯离线方案捉襟见肘。以旧金山为例，几乎每天街头都会出现道路管制（closure）、施工禁行区（construction zone）或重新粉刷的车道线（repainted lanes）。面对频繁的物理变更，如果仅依靠漫长的离线数据回传、人工清洗建图再下发，车队根本无法正常运营。 为此，Zoox 确立了**混合建图策略（Hybrid Mapping Strategy）**： | 维度 | 离线建图（Offboard / Offline） | 在线建图（Onboard / Online） | |:--- |:--- |:--- | | **计算资源** | 算力不受限制，可跨多趟行驶日志（multiple runs）进行海量先验融合与全局重优化 | 车端算力受限，需毫秒级实时计算 | | **环境鲁棒性** | 多次采集对冲恶劣天气偶发遮挡，精度极高 | 易受浓雾、眩光等局部亚优感知条件干扰 | | **时效性与应对变化** | 流程较长，无法应对正在发生的突发物理变更 | 实时检测道路最新变动（如突发施工、改道） | | **核心职责** | 提供全局精准的车队底图（Fleet Map） | 动态修正地图偏差，确保在活动施工区内安全行驶 | 纯在线建图在恶劣环境下（例如浓雾弥漫遮挡地标）难以单帧生成绝对精准的地图拓扑；而纯离线建图又缺乏动态应变能力。混合建图则打通了两者的闭环：车载在线地图检测器（Map Detector）生成的实时图层与云端下发的高精车队地图（Fleet Map）在车端进行在线融合比对，一旦实时算法甄别出二者存在不一致（如突发封闭），系统不仅能确保车辆直接依据实时融合结果在施工区域内安全穿梭通行，还能通过 OTA 无线网络将世界变更（world change）标记回传，更新整个车队的全局地图系统。 ### [29:19](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1759s) 离线地图更新闭环与动态道路环境应对 在车端实时感知生成可供驾驶的局部表征之后，即使先验地图发生变更，车辆也能安全行驶，但这只是整个体系的一半。与车端行驶并行发生的，是一套紧密协作的云端（Back Office）闭环流程：车辆在行驶中捕获到的局部几何与要素变化检测，会被实时或批量回传至后台系统。随着车队规模的扩大，后台会持续汇集来自大量车辆在不同时间、不同角度的感知检测结果；随后，离线系统投入算力进行数据聚合、联合优化，并在发生改变的具体道路图块（Patches）上重新构建局部高精地图。新生成的地图补丁经过验证后，会被重新分发下发给车队。这一闭环流程伴随着车队运营区域的拓展和现实物理世界路网的变迁，永不停歇地持续循环运转。 演讲现场展示了两段端到端自动驾驶系统应对地图突变场景的实际行驶记录。在左侧视频中，路面刚刚经历重新铺装（Repaved Road），周围布满施工区域标志、临时围挡与粗糙地面，自动驾驶系统毫无阻碍地穿行其中；在右侧视频中，地面车道线已被彻底重新涂刷（Repainted），与先验图层存在显著差异，但车辆依旧凭借现场感知的局部建图能力，顺利构建出可行驶表征并平稳驶过。 ```mermaid flowchart LR accTitle: 闭环地图更新与车队自愈架构 subgraph Car["车端感知与执行"] A["车载传感器实时感知"] --> B["生成局部可行驶表征"] B --> C["完成道路行驶/应对变化"] end subgraph Cloud["云端聚合与优化"] B -.->|"回传地图变化检测"| D["车队数据汇聚平台"] D --> E["跨车辆数据聚合与图块优化"] E --> F["重新构建变更区域地图"] end F -->|"分发增量地图补丁"| Car ``` - **车端自主性**：遇到道路重新铺装、标线重画等突发情况时，依赖实时感知构建局部地图，无需强制依赖静态先验。 - **群体优化机制**：多车冗余观测抑制单车误检，针对发生变化的物理空间局部重新优化，避免全图重构带来的海量算力消耗。 --- ### [30:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1833s) 规模式验证与海量事件排查的工程瓶颈 在大规模自动驾驶研发闭环中，“规模化”（Scaling）不仅意味着车队里程的指数级扩张，更带来了验证（Validation）与问题分类排查（Triaging）两大基础设施维度的严峻工程挑战。在软件迭代过程中，每次代码或模型更新都需要经过大规模仿真测试验证。然而，随着累计采集的真实路测里程跨过数百万英里门槛（例如 300 万英里），全部重新仿真一遍不仅算力成本高到令人难以承受，工程效率也极低。日常行驶中充斥着海量平淡无奇、缺乏信息量的“垃圾数据”（Uninteresting Data），对所有数据做盲目的重跑仿真没有任何实质价值，核心矛盾在于如何精准挑选高价值场景。 与此同时，随着部署车辆的大幅增加，来自车队的异常事件与长尾问题报告量同样水涨船高。每一个触发报警或表现不佳的事件都需要由专业工程师深入排查，厘清根本原因，并判断是否需要研发介入修复。这种传统的人工分诊模式存在不可逾越的扩展性瓶颈——排查团队的人力扩张无法跟上车队规模的爆发式增长，而且分类排查是一项对专业领域认知要求极高的复杂脑力工作。因此，Zoox 的工程解法直指两端：前端构建“智能数据采样策略”（Smart Sampler Strategies），后端利用 AI 工具全面推行自动分诊流水线。 - **全量仿真的局限**：对数百万英里真实数据全量进行重新仿真（Re-simulation），算力开销巨大且包含海量冗余日常场景。 - **人工排查的扩展瓶颈**：异常事件排查不仅是简单的画框标注，需要理解行为意图与时空交互演进，传统人工分类随着规模扩大难以维系。 --- ### [32:30](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1950s) 基于自编码器与嵌入聚类的智能数据重采策略 为了破解数百万英里数据无法全量仿真的计算难题，Zoox 构建了一套基于自编码器（Autoencoder）的两阶段数据采样流水线。在训练阶段，算法提取数据库中包含目标轨迹（Tracks）、地图结构要素、文本上下文以及各类传感器原始信息的多模态完整日志（Log），输入编码器映射为潜在表征，再经由解码器重建原始日志。一旦自编码器成功训练并获得高保真重建能力，解码器就会被直接丢弃，因为当前任务的核心是表征提取。流水线仅保留训练成熟的编码器，专门用来处理海量历史日志并生成高维嵌入向量（Embedding）。 ```mermaid flowchart TD accTitle: 基于自编码器嵌入聚类的数据智能采样流程 subgraph Stage1["第一阶段：自编码器训练"] A["原始行车日志 (轨迹/地图/上下文)"] --> B["Encoder (编码器)"] B --> C["潜在表征向量"] C --> D["Decoder (解码器)"] D --> E["日志无损重建任务"] end subgraph Stage2["第二阶段：嵌入聚类与定向采样"] F["海量历史数据日志"] --> B B --> G["提取场景高维 Embedding"] G --> H["聚类划分 (Clustering)"] H --> I["长尾稀疏小簇 (优先高频采样)"] H --> J["常见名义大簇 (适度稀疏抽样)"] end ``` 通过对提取的嵌入向量进行无监督聚类，场景库被拆分成大小不一的聚类簇（Clusters）。直方图分布显示，海量数据自然形成了少数样本量极其庞大的“常见名义簇”（Nominal Cases），以及大量分布在边缘的长尾“小簇”（Small Clusters）。如果采用传统的全库随机抽样，算法有极大概率漏掉这些样本稀缺的小簇，而这些小簇恰恰对应着系统最脆弱的长尾边缘工况（Edge Cases）。通过聚类后进行分层定向采样，系统优先且集中抽取稀疏小簇中的极端长尾样本，同时在大型簇中抽取少量常规名义工况，使每次重新仿真都能将昂贵的算力精准倾斜在关键安全场景上，例如“绕行静止停放车辆”（Maneuver around parked vehicle）等高风险且非罕见的关键交互工况。 --- ### [34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) 借助多模态大模型（MLLM）实现自动化分类归因 传统的故障排查流程本质上是一套复杂的认知推演。以右侧屏幕展示的典型长尾视频为例，自动驾驶车辆在路口行驶时，未能在停车线前及时制动，反而对路边一名处于静止状态的行人产生了“过度礼让”（Over-yielding）。人工排查人员在面对这段无先验提示的视频时，必须在大脑中经历完整的多阶段思考链：首先解析场景静态元素（“自动驾驶主车正处于前方十字路口”）；接着研判动态时空交互（“主车正尝试直行，但未能准时减速让行”）；最后综合上下文给出不期望行为（Undesirable Behavior）的具体定性结论（“主车过度礼让了一名静止不动的行人”）。这种工作对专业领域知识和场景意图理解要求极高，绝非低阶标注人员能轻易替代。 为了实现端到端的排查自动化，Zoox 引入了多模态大模型（MLLM）来模拟资深排查专家的认知逻辑。系统将相机图像、文本信息、激光雷达点云与底层结构化日志（感知物体目标、高清地图特征等）全部作为输入进行多模态深度融合。与此前用于输出驾驶动作和行为推理的模型不同，该模型被调优用于直接解码并生成详细的**自然语言诊断文本**，完全对齐人类排查员的标准诊断报告。随后，自动化系统对生成的诊断文本进行词嵌入编码并再次聚类；在监控看板上，海量突发问题被自动收束为一个个带有语义标签的聚类簇（例如明确标为“簇 1：过度礼让”），团队得以实时一目了然地掌握车队在特定工况下的失败频次与行为画像，全面摆脱了人工作坊式的分类排查瓶颈。 --- ### [36:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2193s) 问答互动：天气保真度验证、时序控制与雷达环境噪声 在演讲结束后的问答环节，现场多位参会者就前沿工程实现细节与 Doron 展开了交流探讨： ```mermaid flowchart TD accTitle: 复杂物理环境模型验证与时空轨迹输出架构 subgraph Q1["问题一：合成天气模型的保真度校验"] W1["物理仿真生成恶劣天气场景"] --> W2["对照真实罕见极端天气样本"] W2 --> W3["感知算法双向输出一致性比对"] W2 --> W4["端到端驾驶行为表现一致性评测"] end subgraph Q2["问题二：轨迹点时序与控制指令"] T1["模型预测 Waypoints (路径点)"] --> T2["携带严格时间戳 (Associated Time)"] T2 --> T3["联合输出底层动力学控制指令"] end subgraph Q3["问题三：激光雷达传感器一致性与鬼影"] L1["亿级自主行驶里程"] --> L2["涵盖硬件单元差异与鬼影噪声"] L2 --> L3["训练集泛化学习 + 严密闭环校验"] end ``` 针对首位提问者关于恶劣天气生成模型真实性的疑问（"Did you develop those?... What happens if the weather model that you're training your vehicle on is not as accurate as you would want it to be?"），Doron 强调团队并不仅押注于单一模型，而是采用混合建模架构。引入物理模型（Physical Model）的目的正是为了严格约束真实感。尽管极端天气数据在现实中非常罕见，但工程团队依然保有一定体量的真实样本作为黄金标准（Ground Truth）。验证时，团队会利用物理模型合成与真实极端天气相同的场景，然后并跑两类评测：一是分别在真实数据与合成数据上运行感知系统，检查目标检测与分割的一致性；二是将整套端到端自动驾驶软件栈接入两套环境，验证车辆在面对两类输入时是否做出了完全相同的控制与规控决策，从而严密标定合成环境的保真度。 随后的提问者连续提出了两个关于规划与硬件的硬核问题。首先，提问者指出模型输出路径点（Waypoints）时，如果没有显式时间信息，下游控制器就无法得知期望车速和刹车时机；Doron 证实，模型输出的每一个路径点都绑定有明确的时间戳（Associated Time），并且轨迹点与底层具体控制量是联合输出的，下游控制模块可直接解析并执行制动与转向动作。针对提问者提出的第二个问题——激光雷达（LiDAR）在雨雾等特定物理环境中会因高反射率产生“鬼影”（Ghost detections），且同一型号不同硬件个体单元之间也存在响应差异；Doron 回应道，Zoox 车队已积累了海量的自主行驶里程，这些由不同硬件个体、不同天气折射所引发的感知扰动和数据漂移（Data Drift）早已被系统性地捕获并沉淀至训练集与离线验证集中，模型在训练期已学会抵抗此类传感器硬件变异，确保了感知栈在物理层面的高容错性。 - **天气仿真校验标准**：对比真实极端天气与物理合成天气的感知输出一致性，以及端到端规控反应的一致性。 - **路径点动力学绑定**：轨迹路径点非纯空间几何坐标，每个坐标点均附带严密的时间戳与动力学控制耦合属性。 - **传感器长尾硬件噪声治理**：依托大规模常态化真实道路里程，覆盖激光雷达不同个体的制造公差与水汽高反射鬼影。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=4s) a safer, cleaner and more enjoyable future — Doron Portnoy > [13:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=797s) Decode in parallel for full horizon — Doron Portnoy > [23:56](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1436s) This is extra important for us because if we don't then we need to relabel this data and relabeling is both costly and it's taking a long time. — Doron Portnoy > [36:33](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2193s) Did you develop those?... What happens if the weather model that you're training your vehicle on is not as accurate as you would want it to be? — 现场参会者 > [32:30](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=1950s) Maneuver around parked vehicle — Doron Portnoy > [34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) 簇 1：过度礼让 — Doron Portnoy > [34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) 主车正处于前方十字路口 — Doron Portnoy > [34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) 主车正尝试直行，但未能准时减速让行 — Doron Portnoy > [34:17](https://www.youtube.com/watch?v=SmDB2LdX15Q&t=2057s) 主车过度礼让了一名静止不动的行人 — Doron Portnoy ## Mentioned Zoox, Doron Portnoy, Ai4, DINO, DINOv3, SAM, Robotaxi ## Video description Doron Portnoy - Senior Staff ML Engineer, Zoox Deploying autonomous vehicles on public roads at scale demands high robustness of perception and mapping stacks. This includes handling rare, unpredictable "long-tail" scenarios - the ultimate stretch of perception systems. Overcoming these edge cases requires moving beyond traditional methods of perception toward a new generation of learned, scalable approaches. In this presentation, Doron Portnoy, senior staff ML engineer, Perception at Zoox, will take a technical deep-dive into how Vision-Language-Action (VLA) models are being architected and fine-tuned to interpret complex, unprecedented environments. Because training these intelligent models demands elusive, hard-to-capture data, he will also dive into how the industry is bridging the gap through innovative synthetic data generation and scene editing techniques. Last, Doron will anchor how spatial mapping approaches for L4 have evolved to unlock autonomous driving at scale. [Source](https://www.youtube.com/watch?v=SmDB2LdX15Q)

Source quotes

[0:04] Doron Portnoy: a safer, cleaner and more enjoyable future [13:17] Doron Portnoy: Decode in parallel for full horizon [23:56] Doron Portnoy: This is extra important for us because if we don't then we need to relabel this data and relabeling is both costly and it's taking a long time. [36:33] 现场参会者: Did you develop those?... What happens if the weather model that you're training your vehicle on is not as accurate as you would want it to be? [32:30] Doron Portnoy: Maneuver around parked vehicle [34:17] Doron Portnoy: 簇 1：过度礼让 [34:17] Doron Portnoy: 主车正处于前方十字路口 [34:17] Doron Portnoy: 主车正尝试直行，但未能准时减速让行 [34:17] Doron Portnoy: 主车过度礼让了一名静止不动的行人

Source quotes

[0:04] Doron Portnoy: safer, cleaner and more enjoyable future [4:47] Doron Portnoy: Because the full-capacity teacher model is too computationally heavy to run directly on vehicle compute hardware with real-time latency constraints, Zoox employs a teacher-student distillation strategy to compress the large VLA into an efficient, on-vehicle student model. [10:17] Doron Portnoy: While standard vision-language models (VLMs) prioritize vision and text pairings, Level 4 autonomous driving strictly requires non-visual safety modalities, notably LiDAR and radar. [10:17] Doron Portnoy: token explosion and context window bloating [15:20] Doron Portnoy: Rather than decoding vehicle actions token by token, the Vision-Language-Action (VLA) architecture outputs complete, full-horizon trajectories in just a few steps. [18:16] Doron Portnoy: However, genuine long-tail events expose a structural flaw in mining-dependent flywheels: because these incidents are exceedingly rare, historical databases rarely contain enough samples to support statistical learning. [22:28] Doron Portnoy: Collecting real-world data for extreme road hazards—such as pedestrians lying in traffic lanes or structural debris scattered across a highway—is virtually impossible through standard fleet logging. [24:36] Doron Portnoy: Preserving spatial positions eliminates the need for downstream re-annotation, circumventing an expensive and time-consuming labeling bottleneck. [30:33] Doron Portnoy: Large-scale validation relies heavily on re-simulation. However, with massive repositories spanning millions of logged driving miles, re-simulating every single mile is both computationally prohibitive and redundant [32:30] Doron Portnoy: If logs were sampled uniformly or randomly, rare edge cases represented in the smallest clusters would be overlooked. [34:17] Doron Portnoy: ego vehicle over-yielded to a stationary pedestrian

Original transcript

[0:04] Uh hello um good afternoon everyone.  I appreciate you joining for the last [0:08] session. Um I'm Don Potnoi. I'm a senior  staff engineer at the foundation model [0:15] team at Zuks. It's a pleasure to  be here and give this talk to you. [0:18] So I'm very excited to share with you a small  glimpse of what ZUX is doing to handle the [0:25] large scale and also the longtail that comes  with it. So let's start the presentation. [0:34] Okay. So our vision at Zuks is to build a  safer, cleaner and more enjoyable future [0:40] in the world. And if you got to drive  a Zuks I hope you know what I mean. [0:46] So before we deep dive into the technical session, [0:48] I wanted to give a quick  twominut get to know Zox video. [1:10] Up [1:16] [music] [1:21] B up [music] [1:30] B up up [1:34] B up up [1:38] me up up down up down up down up down  up down up down up down up down up down [1:40] up down up down hey n hey n hey n hey  n hey n hey n hey n hey n hey n hey n [1:46] up. [music] [1:57] Up. [2:05] [music] [2:10] [music] [2:14] [music] [2:19] Heat. Heat. [music] [2:32] [music] [2:38] [music] [2:44] [snorts] [2:51] Okay. Um so I'm very proud to say that very  recently we hit a major milestone in Zuks. We [2:57] are the first US company with purpose-built  robot taxi no pedals that got approved for [3:05] commercial operations and we last year  we launched our services here in Vegas [3:11] and also in San Francisco and we are testing  our services on 10 different CI across the US. [3:17] We welcome more than half a million riders already  and we driven over three million autonomous miles. [3:28] Okay. So with that short intro we can  deep dive into the technical session. [3:33] So I'm going to talk about the scaling  challenges of autonomous driving in Zuks. [3:38] And in this talk I'm going to split the  technical challenges into four pillars. [3:43] The first pillar would be the onboard driving  model and I'm going to talk about our foundation [3:48] encoder and multimodel language action models. We  use this to self safely navigate through the edge [3:56] cases. We'll then talk about the data engine  which we use to generate data for our driving [4:03] models both for training purposes and evaluation  purposes with technologies such as world models. [4:10] After that I'm going to talk about mapping.  We'll talk about how we scale mapping, [4:15] what the challenges in scaling the mapping and how  we combine offboard onboard highbe architecture. [4:21] We also talk about world changes and how they  affect the mapping operation. And finally for [4:28] the last pillar I'm going to talk about what  validation and triage is and how we scale it. [4:37] Okay. So let's deep dive into the first part of [4:40] the presentation which is  the onboard driving model. [4:47] Okay. So generally as we scale our operation we  get a larger fleet we go to a new ge offenses [4:55] between a new city we will encounter a longtail  events much more frequently and our solution was [5:01] to take a VA and integrate it on board. So in this  slide you can see a highle diagram of such VA. So [5:11] starting from the left we have our multimodel  input sensors which includes for example camera [5:18] liars radar audio ego state and additional  modalities. Each modality is going through [5:26] its own foundational encoder and then they are  jointly processed by a fusion layer. The fusion [5:33] layer passes it information to the LLM decoder.  And finally we have two task specific heads. The [5:40] first head is the reasoning. The second head  is the action. We engineer the reasoning head [5:48] to provide a support for both high latency and  low latency events as they are both required. [5:58] And as for the action part, the action part is  outputting way points. So essentially a trajectory [6:06] going forward as well as control like braking,  steering etc. What you are not seeing here is [6:14] that this is high model capacity which prevent us  from deploying it on vehicle. So essentially we [6:21] use this model as a teacher and distill model to  a smaller VA model which we then deploy on board. [6:33] So let's start talking about the encoders. So  encoders in general and specifically in here [6:38] our vision potential encoder should have a robust  feature that supports both the nominal cases as [6:45] well as the lontel edge cases. And our visual  foundation encoder can output dense feature [6:53] representation that supports both 3D geometric  task as well as semantic understanding tasks. [7:01] And an example for such tasks would be 3D special  semantics in understanding which is crucial for [7:09] operating on the long day. On the right upper side  you can see a radar chart in which you can see our [7:18] in-house visual foundation encoder compared to  other open-source foundation decoder like dino. [7:26] The way we achieve this diagram is we are taking  our encoder, we freezing it and then we attach a [7:33] lightweight probe on top of it. Essentially,  you can output detection or segmentation, [7:38] classification and other type of tasks. And as you  can see on the radar chart, each one of these is [7:44] tested at multiple environments. It can be simple  environment and challenging environment like fog, [7:49] rain, snow, etc. And overall you can see that we  outperform most of the visual foundation encoder [7:58] across this chart on most of the most of the  activities that are interesting to us. At the [8:06] right side you can see a similarity map. The red  dot is sitting right on the center on the double [8:15] lane line. And we compute the similarity between  this red dot and other pixel in the image. And [8:21] what you can see on the left side is Dino V3  when you can see the similarity map is blurry [8:27] and noisy. Whereas in the right side you can see  our own visual foundation encoder and you can [8:34] see it's much more well defined and sharp and  extentuate the lengths and essentially this is [8:42] kind of surprising and I'm happy about it because  we haven't trained our model specifically on [8:48] row geometry which is very nice sign for the  encoder being a real foundational encoder. [8:57] Okay. So how do we get this foundation coder  with high capacity and deploy it on vehicle? So [9:04] essentially we follow this three steps recipe.  In the first stage we perform knowledge isolation [9:12] from open-source models such as Dino V3 or SM3  into a single backbone that can absorb all this [9:21] knowledge. We do this in a label-free manner on  our own internal data. But we don't need labels [9:26] for that. The second stage is the supervised  stage. We use a super supervised fine tuning [9:32] stage coupled with multitask heads on various  task that are interesting perception task like [9:38] object detection, segmentation, classification,  depth estimation etc. This stage is done in higher [9:46] resolution on both soda label data and fully  labelled data. And finally after this stage [9:54] we have our foundation model which we need  to distill to on vehicle student and we do [10:00] this with two types of method of both feature  distillation and logic distillation. And after [10:07] this stage we have our visual foundation  encoder that we can deploy on vehicle [10:17] while robust visual foundation encoder  is really in the core of bringing new [10:22] features. So we can deploy in the longtail.  The L4 robot taxi still needs other modality [10:32] which is our critical safety for us like LAR  and radar. And if you look at the left side, [10:38] you can see that out oftheshelf MLMs are  usually focused to optimize vision text [10:44] first and are not using lighter modalities or  radar modalities for various reasons. Also, [10:53] if we try to naively take and extend our sensor  modalities and just plug them in into our LLM, [11:01] we might hit some engineering challenges.  As you can see on the right, for example, [11:05] taking high resolution LAR, very high resolution  image and just plugging it in, you'll get token [11:11] explosion and context window bloating, which by  itself will give you high memory consumption and [11:18] very high latency, which is very undesirable  for online driving vehicle. And in addition, [11:26] even if you want to take a new LAR, for example,  and plug it into an existing LM, we need to able [11:32] to train it, which requires a lot of data and a  lot of multimodel data, which is not easy to get. [11:42] So, how do we overcome these challenges? So, first  let's talk about the alignment. So fortunately for [11:48] us as I said we have around 3 million miles that  we already driven in autonomous way. So we can [11:55] use this data to align other modalities that are  not well aligned today with off-the-shelf MLMs [12:02] such as LAR and radar. But this is not enough.  We need to do it in an efficient manner. So take [12:10] a look at the late fusion chart on the right.  The late fusion chart says that we are taking [12:15] all of the modalities, encoding them into  tokens, and then pass them directly into the [12:20] VA. This by itself will cause a token explosion  because we have a lot of high resolution sensors [12:26] around the vehicle and we pass them all and  we expect the VA to attend to all of this. [12:32] What we can do is use the midfusion method  in which we take sensors tokenize them but [12:39] then pass them through a fusion layer before we  finally pass them to the VA. This will reduce [12:44] the amount of tokens but we can lose information  while doing that. So a compromised third method [12:50] could be to combine them. Take the midfusion  and reduce the load out of the VA but still [12:56] plug in some late fusion sensors. For example,  we don't have to plug them all. we can plug [13:01] only specific critical safety sensors or or  we can reduce the some of the resolution or [13:08] some of the frequency and still plug them in.  And that way we can benefit from both worlds. [13:17] So now that we've passed the modalities at the  inputs and talked about the encoder and a little [13:25] bit about the fusion layer, let's talk about the  specific heads starting with the action head. [13:30] So the design choice we need to make here is  we have a LLM. The LLM outputs tokens and we [13:37] need to figure out how we take these tokens and  convert them to some kind of actionable driving [13:42] commands at the end. So at the left you see two  mainstream paradigms to do that starting from the [13:50] the street action token one. So in this method  we essentially treat the tokens as language. [13:57] We take the continuous actions action space  and we discretize it. Each discretiz quanta [14:04] is pushed into tokens and then essentially we  have a words or a sentence which we can train [14:09] with LLMs and then we can call train them  with VLM task for example and we'll talk [14:15] about reasoning in a while after we train them  we can do inference and inference with the LLM [14:22] is a usual auto reggressive inference when we  get token by token action and then we take this [14:28] action and really turn them into controls on the  vehicle. So what's the drawbacks? The drawbacks [14:35] is because we doing discrete quantization as you  can see on the right we take the continuous space [14:41] and we snap it into discrete grid which causes  quantization artifacts unfortunately and also [14:48] the auto reggressive as I said before can be  very expensive in terms of latency if we need [14:52] to decode a lot of tokens. So what we can do is  go to the second method which is action expert [15:03] with flow machine head. In this method we use the  LLM to constrain or condition the flow machine [15:12] head. And the good thing with flow machine head is  that it can decode in parallel for full horizon. [15:20] So we don't need to decode token by token. And  also it gives us smooth and continuous control. [15:25] So you can see on the left side of the image when  we get a full trajectory with full horizon only [15:31] with couple of steps and this way we can  we don't need to uh decode token by token [15:40] and finally for our VA let's talk about  a reasoning head. So as I said like a [15:46] a reasoning head was engineered to  generate various amount of trains of [15:53] uh of traces that varied in length. So  we can either generate long reasoning [15:59] traces for complex edge cases say negotiation  between two vehicles. In this specific example, [16:05] you can see our reasoning model first reason about  the scene understanding what's happening in the [16:12] scene and then it can do prediction of specific  agents and then the planning part and finally you [16:20] can output way points and controls where on the  left side you can see a more immediate response [16:26] is required which is why we output a short  reasoning trace. You can see a green utility [16:32] tractor which blocking our way and we need to  make abrupt decision. So we can either output [16:38] a very short reasoning trace saying there's  a black utility tractor something like that [16:44] or we can avoid the trace altogether really low  latency event and go directly to the waypoints. [16:56] So just as a summary as we scaling up both  in terms of size of the fleet and in terms [17:06] of geoences to new cities we will encounter  long tail more frequently. So kind of the [17:12] longtail is becoming something more common  and we get a new longtails and we've chosen [17:18] to take the VA to help us safely navigate  through various complex use cases. And at [17:23] the bottom you can see some of these use cases  for example incident zones where we see a crash [17:28] there or flaggers or flooded roads or host  mounted police or another interesting cases. [17:39] So here you can see a video of our  VA in action. In yellow you can see [17:44] the waypoints. The VA recognizes the scene  as a critical scene and output way points [17:50] which we then take processing to actions and  then drive upon. So on the left side you can [17:57] see an immediate dynamic accident that we  need to maneuver around it abruptly and on [18:02] the right side you can see something that  is more slow when we have time to respond [18:07] because the accident already happened and we  need to just slow down and get around them. [18:16] Okay. So now that we talk about the onboard  VA, we need to talk about how to provide data [18:23] to train this VA. um we need data for  both training and evaluation purposes. [18:29] So in this section I'm going to talk about what  data engine do we use to generate such data. [18:40] To constantly improve our VLA we use data  flywheel methodology. You can see in this [18:46] diagram we start from training our model with  existing data and then we need to validate [18:54] it. We validate it using large scale simulation.  After feeding our VA into large scale simulation, [19:02] we then need to validate it and triage it to  try and find critical performance gaps. When [19:09] we do find such a gaps, we need to mine our own  databases in order to surface this real world [19:16] data that we have. If we find it, we can then  label it and pass it back to our VA for training. [19:27] The problem with longtail event is that they  usually are not in our databases and even if they [19:31] are they are very small and we can't really train  with them. We can only use them for validation. [19:36] So this is why we need engine to simulate our  data both for training and validation both for [19:42] the generation step and for the simulation  step. So let's look at couple of examples. [19:52] So in this slide you can see three technique  we use individually and jointly together to [19:58] generate data. Starting with the world  model. The world model is generative [20:04] frameworks that enable us to use both prompting  and conditioning in order to generate scenes. [20:11] It's enable us to generate scene from scratch or  we can adapt things inside of the image change [20:18] some things and we have the ability to keep  a lot of things and I'll show some examples [20:22] afterwards for the city simulation it's  more classical method in which if we have [20:28] 3D assets we can use them render them through  our different modalities so render the assets [20:34] on the different cameras render the assets on  the different lightars and we get a real fine [20:41] grain control on the positioning of the objects  which important on the last part we'll get the [20:47] physical base simulation which we use to generate  specific weather condition for example we can get [20:55] a fine grain control on fog which is really  important for us and as I said we can combine [21:01] all of this method together for example we can  render an object into an image and then we can [21:06] use world model to start from this point or we  can have is mix and match between these methods. [21:16] As you can see in this slide, this is our  high level architecture of the generative [21:21] war model we use. So in the core, we have special  temporal latent space which is used to capture the [21:29] underlying physics and dynamic of the scene which  is important to be able to generate diverse set of [21:36] scenario with the lante. At the left side you can  see the inputs which are noise sensors depending [21:45] on if it is the training stage or inferior stage  like cameras, lighters etc. On the upper side [21:52] you can see the controls like text prompting  objects maps future route etc. We use this [22:01] to control the scene. And on the right side you  can see an example of generated data. So this is [22:08] specific an example of front camera generated data  coupled with the LAR data. So they are jointly [22:15] generated and we can essentially create 360 data  together with all of our sensor with this method. [22:28] Let's look at the first application of this  and we talk about various generation methods. [22:34] So first one is the insert in rare and  high risk objects. The problem with rare [22:42] and high-risk object is that we can't  really find them. And if you find them, [22:46] they are very rare. I'm talking about things like  line pedestrians or debris on the road. This is [22:50] something you can't really hunt and you can't  really stage because it's like really something [22:56] that's it's high risk. So we can't implement them.  So the way we do it is we do it with simulation. [23:03] So on the upper right side you can see a  classical 3D simulation in which we take [23:08] 3D assets like pedestrian or the breeze and we  render them across the modalities. And if you [23:15] notice you'll see the debris and the line  pedestrians appear both on the camera side [23:21] and also on the lighter side. And the consistency  between the sensor is important here. And on the [23:27] right lower side you can see another example where  we edit the scene. In this example, we use world [23:34] model technology to edit an object or replace an  object. So we replace the vehicle in front of us [23:40] with horsemounted police that exist in some of  our geoences which is important for us. And you [23:46] can generally see that the scene overall stays  the same and only the implanted object change. [23:56] So in Zuks we are very safety aware which is  why we decided to get multimodel sensors for [24:04] our onboard stack. So you can see an example  of some of the sensors we use lighter modality [24:09] and camera modality. And in this specific  example you can see us driving to the rain. [24:15] And if you'll notice, you'll see that  each modality come with its own querif. [24:21] Which is why we need data to train extreme weather  condition and validate extreme weather condition. [24:36] So weather condition chasing storm can be like  okay but but it's it's very costly. And if [24:44] you'll look at the compounding factor of storms  plus rare events like uh line pedestrian. So it [24:52] becomes unfeasible to collect all of these edge  cases which is why we simulate it. And in the [24:58] right side you can see world model simulation. We  are starting from the base log on the right which [25:05] is a sunny day that exists in our databases. And  then we'll use word model with prompting and say [25:13] get snow on the road or sandstorm or flooded road  in this case. And what's more important is that we [25:20] use conditioning for the object to stay in place.  This is extra important for us because if we don't [25:27] then we need to re label this data and reabeling  is both costly and it's taking a long time. [25:37] Another example of control weather generation is  fog generation. We do this with a combination of [25:45] physical simulation as well as genai technique.  We combine them together. And as you can see the [25:51] upper side is the nominal log in which we just  drive in sunny day and on the lower side you can [25:59] see the rendering of the physical fog both in  360 camera view as well as in the lighter view. [26:12] After covering the online driving  models and data engine let's talk [26:17] about mapping. what we do to scale mapping  and what are some of the challenges there. [26:26] As we expand to you geoences, it's becoming  increasingly hard to do manual map creation. [26:33] Imagine that we are doing more and more  cities and it's becoming just impossible [26:38] because it costs a lot and it's very  manual process and it can take time. [26:44] So the other problem is that there are  very frequent map changes. So if you look [26:49] at San Francisco for example, every day you see  something else, a closure, construction zone, [26:56] lanes repainted, etc. So we can't do only offline  processes that takes a long time. And this is why [27:03] we prompted from the hybrid mapping strategy  in which we combine both offline mapping and [27:10] online mapping together. When we see a world  change, we can use overtheair method to push [27:17] this map changes into our fleet. And another  important aspect is if we are at a active map [27:25] change like construction closer, we need  to be able to drive safely inside of it. [27:34] Why hybrid method? So you can ask yourself  if you already have online mapping method, [27:39] why do we even need offline? we can do everything  online. So the answer is partially given in the [27:45] right side of the screen. So you can see the  image is foggy and unclear and sometimes we [27:51] got suboptimal conditions online and it's it  could be hard to generate a very accurate map [27:58] if you are at suboptimal conditions whether  at offline condition we can use more than a [28:05] single run we can aggregate multiple runs for  example and we also not constrained by compute [28:11] so we can do heavy optimization with various  priors and we can get more accurate mapping [28:17] because of it. And as of the online mapping,  we've already discussed about it in the last [28:22] slide. We need it because there are a lot  of road changes happening on the time. So we [28:28] can't rely only on offboard. We need something to  detect when there is a map change and we need the [28:33] fleet to be able to drive even in a map change.  So this is the reason why we need them both. [28:42] Talking about the cycle. So the cycle of how we  combine both online and offline mapping is in the [28:51] diagram on the right. Now let's start with the  lower part of the diagram. So this is an onboard [28:57] diagram. You can see it gets two mapping inputs.  One is generate out of his own map detector [29:05] and the other one is a flip map. This is the map  that's generated offline. So the online algorithm [29:11] is taking these two maps and fuse them and trying  to find real-time map changes. We then take this [29:19] and generate representation we can drive on on  a vehicle. So this means that we can drive even [29:26] if there is a map change which is really really  important for the offline part. What happens in [29:32] parallel is that we'll take this detection and  push them to the back office. Then we can get [29:39] a various map detection from a large fleet and we  can then spend time to aggregating it, optimizing [29:46] it and rebuilding the map at the specific patches  where something changed. We can then push it back [29:52] to the fleet. And this is a cycle that's going  on and on as we expand, as map changes, etc. [30:02] In here you can see two examples of end to end  driving on how we drive at such math situation. [30:09] So on the left you can see a repave road as we  drive through it. You see a lot of construction [30:15] area going on and we successfully drive there  with no specific issue. And on the right side [30:21] the lanes has been totally repainted. Still  we can build a local map and drive through it. [30:33] For the last part of the talk, I'm  going to talk about what scaling [30:37] means at validation and triagine and  what we do to overcome scaling issues. [30:46] So let's review first what is large scale  validation for us and what's continuous triaging [30:52] and then we can talk about the solution. So first  large scale validation I talked about before the [31:00] cycle that we do to improve our performance and in  there I showed you a blog that says we are doing [31:06] validation. So this large scale validation mean  that we take the VA and we somehow validate it. [31:11] The way we do it is large scale simulation. The  problem is as we expand our miles we have huge [31:18] amount of data in our databases and it's become a  question of what data do we need to reimulate. Do [31:24] we need to reimulate all of the 3 million miles  that can be for one very expensive and also there [31:30] is a lot of redundant information in there. If  you think on day-to-day driving there are a lot [31:34] of uninteresting stuff. So just reimulating it  doesn't give you anything. So the solution for [31:41] us is to produce smart sampler strategies so we  can sample our databases and decide which one to [31:48] reimulate and then we can both reduce the cost  and do much more some smart sampling. The lower [31:57] part continuous triaging. So as we expand the  fleet we'll see a lot more issues coming from [32:04] the fleet and each issue we need to triage in. We  need to understand what's the problem and if there [32:09] is a real problem we need to fix it. The problem  is that as you scale more and more and more you [32:14] need more people to do this triaging and this is  not a simple job. So the solution is to automate [32:21] this pipeline and I'll talk about in couple of  slides on how we automated it using AI tools. [32:30] Let's start with the data sample data sampling.  So as you can see there is two-stage pipeline [32:36] in here. The upper one we use an autoenccoder  to take our log in the database and we pass [32:44] it through the encoder. We decode it and  then we try to reconstruct the log. The log [32:49] essentially is a combination of tracks and maps  and words and everything we have inside the log. [32:56] Now that we managed to reconstruct  it and train such an autoenccoder, [32:59] we can essentially throw the decoder because for  this task, it doesn't interest us. So what does [33:04] interest us is the encoder. We take the logs  from a data set and we encode them. And now [33:10] we have each of the each of the sets in the logs  as an embedding. And when we have an embedding, [33:17] we can cluster them. So now as you can see  in the histogram, we have a lot of small to [33:23] large clusters. So imagine that if we just  sample randomly out of all of our databases, [33:29] there is a good chance that we'll miss this small  cluster. And this small cluster is important [33:33] because they are covering the edge cases. So after  we cluster and we'll get the small cluster, we can [33:41] mostly sample them and probably sample some of the  larger one because we also want nominal cases, but [33:46] we do want the edge cases. So this helps us focus  on the right place when we doing re simulation. [33:55] This is a single example for such a cluster  just to put it more concrete. For example, [34:00] maneuver around parked vehicle.  So you can see a three images [34:04] when we successfully maneuver around the  vehicle which is probably a scenario we [34:07] want to test on because it's critical and  probably have it's not very rare. Okay. [34:17] Triaging the manual triaging is very complex. You  can see at the right side a video that triagger [34:27] sees in which we drive and over yield to a  pedestrian. We didn't stop before the line [34:36] and so a trier will get that and we'll try to  understand what's happened within the scene. [34:40] No one tells him what's the problem. So it will  try to describe the scene and says okay there is [34:45] an ego vehicle and a forward intersection and  then it will try to understand what's the key [34:51] situation here. So the vehicle is attempting  to go straight but maybe it's not yielding on [34:57] time and finally it'll need to come back to  a conclusion with an undesirable behavior in [35:03] this case saying the ego vehicle over yielded to a  stationary pedestrian. So you can see it involves [35:10] a thought process. It's not only annotate object  which is very simple. It's someone that needs to [35:15] understand the domain. It's someone that needs  to understand the stages of what's happening [35:19] etc. So how do we resolve it? We need  some way to imitate the triage work. [35:29] The way we achieve it is harness the powers of  MLM. Again we have multiple modalities at input [35:35] coming from our databases. For example camera,  text and lighter as we talked about before. [35:40] But here in addition we can input all of our  logs. So the objects map everything and then [35:48] we can fuse it pass it through an LLM and at  this time instead of decoding it to action [35:54] and reasoning we can actually decode it to  text which is similar to what a triagger is [36:00] doing with this triaging problem. And when  we have such a text and assuming we did the [36:05] training well we can take it embed it and  cluster it again. Now you can imagine that [36:11] we have multiple clusters each cluster  with a description and this gives you a [36:16] very good view of what's happening with  your vehicle at any moment. We will see [36:21] a large map says cluster one over for example  and we know how many scenarios we overed with. [36:33] That's it. I hope you enjoyed this scaling  journey with me. Um, and please visit our [36:38] career page and learn more about open  positions. Thank you. Thank you so much, [36:44] Doran. Does anyone have any questions? Since he's  already up here, if anyone's got any questions, [36:50] we'll come at you cuz I know it's was  a very, very exciting presentation. [36:55] I've got I'm coming. I'm coming coming.  Just doing my my walk over. Yes. Hello. [37:07] Um, hi, great presentation first of all. Thank  you. Um, I wanted to ask about the weather [37:13] uh sort of models. Did you develop those? Where  did you get them? And how did you make sure [37:19] um that they are accurate? Because I mean  I see you're taking just a regular street [37:25] putting weather on it to train the model. But what  happens if the weather model that you're training [37:31] your vehicle on is not as accurate as you would  want it to be? Yeah, that's a great question. So [37:37] as I said before, we are not actually using only  one model. We have a combination of model. So one [37:43] of the reason that we are using the physical model  to generate some weather condition is because of [37:48] that exactly and you can imagine that some of  these weather condition we do have them we just [37:54] don't have them like in large quantities. So what  we can do for example is generate a scenario that [38:00] we already have and then compare it somehow  to the scenario that is real scenario. We can [38:06] run perception on both and check um how well they  behave. We can run end to end stack and see if we [38:13] are behaving the same in the same situation and  this way we can validate ourselves. Fantastic. [38:19] Thank you. Anyone else last bitters for question?  I think I see a gentleman here at the front. [38:30] Hi. Um, I have a couple of questions, but I'll be  quick. Um, your model sort of gives way points, [38:37] but is there an associated time with it that  it outputs? Because just seeing the way points, [38:43] I don't know what the speed of the vehicle should  be. So, is there an associated time with that? [38:48] And that helps the model to know, okay, now I  actually need to break or something like that. Um, [38:54] yeah. So, so great question. Yeah, it is every way  points come with an associated time and also the [38:59] controls together. So, we can turn it to specific  control. So, okay. Um, the other question I have [39:06] is often when we work with LAR and stuff like  that. The behavior the LAR exhibits depends [39:11] on what kind of environment we have. It also  varies from unit to unit. like a same company LAR [39:18] might like different units might have different  responses to the same sort of environments, right? [39:24] Um is there some sort of way that you like tackle  these issues? For example, ghost detections from [39:32] reflectivity issues or like rain. So yeah. Yeah,  a great question. So as as I said like because [39:40] we driven around autonomous miles, so we've seen  a lot of this variation. So you can imagine that [39:46] we've seen them in training and that we also  have validation to capture all of this. That [39:51] way we can measure ourselves and see what happens  at these different instances that you mentioned.