Tesla Unveils Custom D1 Chip and Humanoid Robot to Scale Autonomy

Tesla

Tesla is leveraging its automotive vision networks and automated labeling pipeline to build a custom supercomputer and a humanoid robot that aims to make physical labor optional.

AI-assisted summary of Tesla AI Day 2021 — original source. Check the original for full context.

Source published: 2021-08-20

Key ideas

Tesla unveiled its custom D1 chip and Project Dojo architecture, packaging 25 D1 dies into a 9-petaflop training tile and scaling to an ExaPOD supercomputer capable of 1.1 exaflops of compute to accelerate neural network training. (111:39)
Tesla Vision shifted completely to multi-camera surround video networks operating directly in 3D vector space, integrating spatial RNNs and feature caching to eliminate millimeter-wave radar from production cars. (69:35)
Tesla announced the Tesla Bot (Optimus), a 5'8", 125-pound humanoid robot designed to automate repetitive, dangerous manual tasks by repurposing vehicle Autopilot hardware, vision models, and compute. (125:14)
Elon Musk announced Hardware 4 (FSD Computer 2) will debut with the Cybertruck in roughly a year, delivering approximately 4x greater compute to target a 1,000% safety improvement over human drivers. (171:15)

Detailed notes

## Background Published on YouTube by Tesla on August 20, 2021, this video captures the full broadcast of Tesla AI Day 2021. The event description invites engineers to join Tesla in building the future of AI, laying out a technical schedule spanning Tesla Vision, planning and control, auto-labeling, simulation, custom hardware, Project Dojo, the Tesla Bot, and an audience Q&A.

## TL;DR Tesla engineering leadership presents an in-depth technical breakdown of the company's vertically integrated artificial intelligence stack, framing the vehicle as an embodied synthetic animal. Key presentations detail the evolution of Tesla Vision from per-camera 2D space to multi-camera 3D vector space using spatial RNNs and cross-attention, which enabled the removal of radar, alongside hierarchical multi-agent trajectory planning. The team also demonstrates its massive automated labeling pipeline and photorealistic simulation engine before unveiling Project Dojo—a supercomputing architecture powered by the custom D1 chip and modular training tiles. Concluding the presentation, Elon Musk reveals the humanoid Tesla Bot (Optimus), targeting the automation of repetitive, dangerous, and boring physical labor.

## Atmosphere & style The presentation feels far more like an advanced, highly specialized graduate computer science symposium than a polished marketing keynote, dominated by dense architectural diagrams, code breakdowns, and deep technical transparency. Speakers like Andrej Karpathy, Ashok Elluswamy, and Ganesh Venkataramanan present with rapid-fire, candid precision, highlighting grueling engineering bottlenecks, memory constraints, and practical hardware hurdles. Elon Musk anchors the session with his characteristic mix of dry humor, grand macroeconomic predictions, and philosophical reflections on AI alignment, giving the marathon event an ambitious, intensely focused recruitment vibe.

## Bulletin - [01:51:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6699s) Tesla unveiled its custom D1 chip and Project Dojo architecture, packaging 25 D1 dies into a 9-petaflop training tile and scaling to an ExaPOD supercomputer capable of 1.1 exaflops of compute to accelerate neural network training. - [01:09:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4175s) Tesla Vision shifted completely to multi-camera surround video networks operating directly in 3D vector space, integrating spatial RNNs and feature caching to eliminate millimeter-wave radar from production cars. - [02:05:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7514s) Tesla announced the Tesla Bot (Optimus), a 5'8", 125-pound humanoid robot designed to automate repetitive, dangerous manual tasks by repurposing vehicle Autopilot hardware, vision models, and compute. - [02:51:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10275s) Elon Musk announced Hardware 4 (FSD Computer 2) will debut with the Cybertruck in roughly a year, delivering approximately 4x greater compute to target a 1,000% safety improvement over human drivers.

## Key takeaways - [48:44](https://www.youtube.com/watch?v=j0z4FweCy4M&t=2924s) Tesla frames autonomous vehicle development as building a synthetic animal, replacing biological nervous systems and visual cortices with cameras, FSD computers, and deep neural networks. - [51:31](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3091s) The HydraNet vision pipeline uses RegNet backbones and BiFPNs to share feature extraction across dozens of perception tasks, utilizing multi-scale feature caching to accelerate head iteration. - [58:10](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3490s) Direct projection from 2D image space to 3D world coordinates failed due to depth noise; Tesla solved this by mapping raw 8-camera inputs directly into 3D vector space (Bird's Eye View) using transformer cross-attention. - [01:06:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4014s) A spatial RNN maintaining a 2D ground-plane lattice acts as an implicit, feature-level HD map, preserving environmental memory during static idling and dynamic occlusions. - [01:09:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4175s) Temporal video networks matched and exceeded radar depth and velocity accuracy, enabling Tesla to completely remove radar sensors from production vehicles. - [01:13:11](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4391s) Trajectory planning is handled via a two-stage hierarchical system: discrete coarse search establishes a topological corridor to avoid non-convex local minima, followed by continuous numerical spline optimization. - [01:22:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4935s) Tesla is introducing neural network value functions guided by Monte Carlo Tree Search (MCTS) to dramatically cut planning search states in unstructured interactive driving environments. - [01:28:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5308s) Automated server-side labeling ingests multi-trip fleet video to reconstruct dense 3D point clouds and implicit NeRF-style road surfaces using temporal hindsight, distilling offline models into vehicle networks. - [01:34:52](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5692s) The custom photorealistic simulation engine generates mathematically ground-truthed edge cases, sensor physics distortions, and digital twin failure replays, having served over 300 million synthetic training images. - [01:51:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6699s) The D1 chip features 354 custom 64-bit training nodes on a 7nm process, delivering 22.6 TFLOPs and double the I/O bandwidth of standard networking switch chips. - [01:54:06](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6846s) Dojo training tiles utilize vertical power delivery and dual-sided liquid cooling to overcome extreme sub-one-volt electrical current resistance ($I^2R$) and thermal bottlenecks. - [02:00:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7228s) The Dojo compiler dynamically partitions clusters into DPUs, automatically extracting data, model, and graph parallelism from standard PyTorch code without handwritten kernels. - [02:09:00](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7740s) Tesla Bot leverages the vehicle autonomy stack, sensors, and actuators to address labor shortages, with Musk predicting general-purpose humanoid robots will eventually make physical labor optional. - [02:34:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9296s) Musk cautions against applying machine learning to factory floor automation and operational workflows, noting deterministic software and mechanical redesign (like single-piece gigacastings) are superior in 99.9% of cases. - [02:51:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10275s) Musk projects Hardware 3 is capable of achieving self-driving safety 200% to 300% better than human drivers, with future Hardware 4 pushing that margin toward 1,000%.

## Detailed walkthrough ### [46:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=2814s) AI Day Begins: Tesla's Mission in Real-World AI

Elon Musk opens the presentation by welcoming the audience and apologizing for the delay caused by minor technical difficulties, joking, “really need AI for this.” He immediately clarifies the true purpose of the event: demonstrating that Tesla is far more than just an electric car company. Beneath the surface, the company maintains extensive artificial intelligence efforts across custom hardware, inference, and training. Musk makes the case that Tesla has become a preeminent force in physical-world applications, stating: “I think we're, I think arguably the leaders in real-world AI as it applies to the real world.”

Musk notes that anyone following the progress of the Full Self-Driving (FSD) beta can directly observe the rapid velocity at which Tesla's neural networks are learning how to drive. While autonomous driving represents an immediate, high-impact demonstration of this technology, Musk hints that the underlying software and hardware foundations will unlock broader applications in the near future. He emphasizes that the primary goal of the event is recruiting elite talent: “We basically want to encourage anyone who is interested in solving real-world AI problems at either the hardware or the software level to join Tesla or consider joining Tesla.” With that invitation, Musk hands the presentation over to Andrej Karpathy to begin the deep dive into the Autopilot stack.

[48:44](https://www.youtube.com/watch?v=j0z4FweCy4M&t=2924s) Tesla Vision: The Synthetic Animal and Vector Space

Andrej Karpathy introduces himself as the lead of the Autopilot vision team, expressing his excitement to walk through the technical architecture that enables the vehicle to drive autonomously. The core objective of the vision system is converting raw sensor streams from eight surround cameras into a unified, real-time 3D representation termed the "vector space." This vector space provides everything required for downstream path planning and vehicle control, including the 3D geometry of lane lines, road edges, curbs, traffic lights, and traffic signs, alongside bounding volumes, dynamic orientations, depths, and velocities of other vehicles.

Karpathy draws an explicit analogy between autonomous vehicles and biological organisms: “What I find kind of fascinating about this is that we are effectively building a synthetic animal from the ground up.” The vehicle navigates the physical world, perceives dynamic stimuli through sensors, and exercises autonomous agency. Tesla develops every layer in-house: the mechanical structure serves as the body, the low-voltage electronics act as the nervous system, the Autopilot computer functions as the brain, and the vision neural networks constitute a synthetic visual cortex. Karpathy illustrates how biological vision routes photonic inputs from the retina through the lateral geniculate nucleus (LGN) into visual cortex areas V1, V2, V4, and IT across the ventral and dorsal streams, explaining that Tesla's engineering task is to mirror this structured flow of information inside an artificial neural network.

```mermaid flowchart LR accTitle: Camera to Vector Space Pipeline accDescr: Pipeline showing raw optical inputs passing through the synthetic visual cortex to output a unified vector space. subgraph Inputs["Sensor Layer"] C["8 Surround Cameras 1280x960 @ 36 Hz"] end subgraph Processing["Synthetic Visual Cortex"] NN["Deep Neural Network Stack Feature Extraction & Fusion"] end subgraph Outputs["Vector Space Representation"] VS["3D Driving World • Lane lines & curbs • Signs & lights • Vehicle 3D bounds & velocities"] end Inputs --> Processing Processing --> Outputs ```

[51:31](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3091s) Per-Image Neural Networks: RegNets, BiFPNs, and HydraNets

Karpathy reviews the chronological evolution of Tesla's vision networks, starting roughly four years prior when Autopilot's operational domain was predominantly single-lane highway cruising. In that era, the network was tasked primarily with basic lane-keeping and maintaining headway behind a lead vehicle, operating strictly on individual camera frames rather than temporal sequences. The raw sensor input for each camera consists of a 1280×960 resolution image with 12-bit integer depth streaming at approximately 36 Hz.

To process these frames, Tesla uses a residual neural network backbone, specifically adopting RegNets because their parameter design space allows an optimal balance between low compute latency and high accuracy. The RegNet backbone outputs an inverted feature hierarchy: lower layers retain fine spatial resolution with few channels (scrutinizing granular pixel details), while top layers yield coarse spatial resolution with high channel depth (capturing broader scene context). The multi-scale features are fused using a bidirectional feature pyramid network (BiFPN). Karpathy explains the value of cross-scale feedback with an example: if a lower-layer neuron observing a tiny visual patch cannot tell whether it is looking at an actual vehicle, context propagated from top layers indicating that the patch sits exactly on the highway vanishing point quickly resolves the ambiguity.

```mermaid flowchart TD accTitle: HydraNet Architecture accDescr: Diagram illustrating the RegNet backbone feeding into a BiFPN, which fans out to multiple modular task heads. subgraph Backbone["Shared Feature Extractor"] IMG["Raw Image 1280x960 12-bit @ 36Hz"] --> RN["RegNet Backbone Multi-scale Residual Blocks"] RN --> FPN["BiFPN Layer Cross-Scale Feature Fusion"] end subgraph Heads["HydraNet Task Heads"] FPN --> H1["Detection Head YOLO-like Object Raster"] FPN --> H2["Lane Head Road Geometry & Edges"] FPN --> H3["Traffic Light Head Detection & State"] FPN --> H4["Auxiliary Heads Attributes & Static Cues"] end ```

Object detection downstream from the BiFPN utilizes a one-stage, YOLO-like head that initializes a spatial raster, outputting a detection probability bit alongside geometric attributes (x, y offsets, bounding width, height, and vehicle class). Because the vehicle must concurrently handle dozens of perception tasks—including reading lane lines, road markings, and traffic lights—Tesla adopted a branched multi-task architecture called the "HydraNet." The HydraNet utilizes a single shared backbone that fans out into specialized task heads. Karpathy points out two critical advantages of this design: - It amortizes expensive backbone feature extraction over all tasks, keeping inference runtime low on the vehicle's onboard computer. - It decouples tasks during engineering cycles, enabling engineers to independently iterate on datasets or head architectures without unintended cross-task regressions.

[54:40](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3280s) HydraNet Workflows and Feature Caching

Andrej Karpathy explains how the Tesla vision team manages the computational complexity of training their multi-task HydraNets. Because of the architectural bottleneck at the trunk level, validating every single task after a minor modification would be prohibitively expensive. To streamline the workflow, the team frequently caches the intermediate multi-scale trunk features directly to disk. This allows researchers to freeze the heavy backbone and fine-tune individual task heads off the cached representations without repeatedly running full image backpropagation.

Every once in a while, the team executes a joint, end-to-end training run across the entire network to refresh the trunk representations, re-caches the resulting multi-scale feature maps, and resumes head-specific fine-tuning. Karpathy showcases sample outputs from the HydraNet architecture as deployed several years prior: from single image inputs, the network simultaneously predicted stop signs, stop lines, lane edges, vehicles, traffic lights, curbs, parking states, and static obstacles like cones and trash cans. While effective for standalone 2D perception, this setup treated each frame in isolation, which soon proved insufficient for autonomous navigation.

- Training iterations cycle between periodic end-to-end joint optimization runs and cached-feature head-only fine-tuning. - Feature caching isolates individual task heads during development, eliminating the cost of continuous regression testing across unaffected heads. - Legacy HydraNet models produced hundreds of simultaneous image-space predictions per frame, but lacked holistic spatial integration.

---

[55:50](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3350s) The Limits of Image-Space Predictions and Smart Summon

The single-image paradigm met its limits during the development of Smart Summon, where an unoccupied vehicle must navigate complex parking lots to reach a passenger. Driving safely through a lot requires continuous knowledge of curbs, drivable surfaces, and static obstacles across all camera views and over time. To bridge this gap, Tesla initially processed curb detection on individual cameras and piped the predictions into an external, hand-engineered C++ tracking module called the occupancy tracker.

```mermaid flowchart LR accTitle: Legacy Image Space Tracking Failure accDescr: 2D camera detections piped into a hand-engineered C++ occupancy tracker create severe vector space projection errors. subgraph LegacyPipeline["Legacy Image-Space Pipeline"] Cam["8 Separate Cameras"] --> Hydra["Single-Camera HydraNets"] Hydra --> Det["Image-Space 2D Predictions"] Det --> Cpp["C++ Occupancy Tracker (Stitching & Depth Projection)"] Cpp --> BadOut["Distorted Vector Space Output (Undrivable)"] end ```

The occupancy tracker attempted to stitch detections across camera boundaries and track them over time, but the approach quickly unraveled. Manually tuning its complex C++ heuristics and hyper-parameters proved nearly impossible. More fundamentally, Karpathy emphasizes that 2D image space is simply the wrong target output space. When 2D curb lines were cast into 3D world coordinates, the resulting vector-space geometries were severely distorted and erratic because projection requires extreme per-pixel depth accuracy that 2D networks cannot reliably provide. Furthermore, occluded surfaces could not be predicted in image space at all, and large objects—such as a nearby vehicle spanning five of the car's eight cameras—could not be holistically recognized by any single camera.

> “You don't want to do this explicitly by hand in C++, you want this to be inside the neural network and train that end to end.”

> “You see how the predictions are quite bad in vector space and the reason for this fundamentally is because you need to have an extremely accurate depth per pixel in order to actually do this projection and so you can imagine just how high of the bar it is to predict that depth so accurately in these tiny in every single pixel of the image and also if there's any occluded area where you'd like to make predictions you will not be able to because it's not an image space concept in that case.”

---

[58:10](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3490s) The Multi-Camera Vector-Space Vision and Transformer Architecture

Tesla concluded that all eight camera streams had to be fed simultaneously into a single neural network capable of outputting representations directly in vector space (Bird's Eye View, or BEV). Achieving this required solving two major challenges: developing a differentiable neural network layer to transform image-space features into vector space, and creating large-scale 3D vector-space ground-truth datasets. Addressing the architectural challenge, Karpathy details why a naive geometric projection fails: projecting a BEV ground-plane pixel back into camera frames depends heavily on variable road topography, slopes, crests, and dynamic occlusions like parked cars.

```mermaid flowchart TD accTitle: Multi-Camera Transformer BEV Architecture accDescr: Image features from 8 cameras attend to vector-space query vectors via a transformer to produce direct BEV features. subgraph ImageDomain["Image Feature Extraction"] Raw["8 Camera Streams"] --> Rect["Rectification Layer (Virtual Camera)"] Rect --> Backbone["Shared Trunk Backbones"] Backbone --> Feat["Multi-Scale Features (Keys & Values)"] end subgraph TransformerSpace["BEV Transformer Cross-Attention"] Grid["BEV Output Grid + Positional Encodings"] --> MLP["MLP Query Generator"] MLP --> Queries["BEV Queries ('Looking for curbs at position X,Y')"] Queries & Feat --> Attention["Multi-Headed Cross-Attention"] end subgraph VectorOutput["Vector Space Decoding"] Attention --> BEVFeat["Vector Space Feature Map"] BEVFeat --> TaskHeads["Decoders (Curbs, Road Surface, Objects, Kinematics)"] end ```

To handle these data-dependent projections, Tesla implemented an attention-based transformer architecture. The network initializes an output spatial raster representing the desired bird's-eye canvas, tiled with sine and cosine positional encodings, which an MLP maps into a lattice of query vectors. Simultaneously, the backbone features extracted from all cameras emit keys and values. Through multi-headed self- and cross-attention, the queries interact multiplicatively with the image keys—effectively asking what visual evidence exists for a road feature at a specific physical location—and pull the relevant feature values across cameras and occlusions.

> “And so effectively what's happening is that every single image piece is broadcasting in its key what it is, what is it a part of: so hey, I'm part of a pillar in roughly this location and I'm seeing this kind of stuff, and that's in the key. And then every query is something along the lines of: hey, I'm a pixel in the output space at this position and I'm looking for features of this type.”

- The BEV raster uses sine and cosine positional encodings mapped by an MLP into query vectors. - Cross-attention decouples feature lookup from rigid flat-ground projection assumptions, accommodating road slope and camera occlusions. - A single transformer block is capable of performing the spatial re-representation from perspective camera views to bird's-eye view.

---

[01:01:32](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3692s) Synthetic Virtual Camera Rectification and Vector Space Results

Deploying this multi-camera architecture across Tesla's commercial fleet introduced a critical hardware variance issue: physical camera alignments vary across vehicles due to manufacturing tolerances and mounting angles, leaving cameras slightly "cockeyed." Directly concatenating raw calibration matrices into neural network MLPs produced suboptimal results. Instead, Tesla introduced a synthetic virtual camera rectification layer placed directly after image capture.

This calibration-driven transformation warps every incoming frame into a unified, idealized virtual camera frame. Karpathy demonstrates the effect by showing time-averaged views from the rear-facing repeater cameras: unrectified images result in a blurry mess due to mechanical variation across cars, whereas rectified frames snap into alignment, making features like the rear edge of the car crisply defined. Passing these normalized frames into the multi-camera transformer yielded dramatic improvements. Karpathy presents side-by-side comparisons of the legacy C++ occupancy tracker against the new vector-space network: jagged, fragmented curb detections are replaced by smooth, continuous vector boundaries.

> “On the left we are seeing what we had before and on the right we're now seeing significantly improved predictions coming directly out of the neural net; this is a multi-camera network predicting directly in vector space and you can see that it's basically night and day, you can actually drive on this.”

- Raw mechanical misalignment across production cars is corrected by warping frames into a shared virtual camera coordinate space. - Time-averaged overlays of side repeater views reveal crisp alignment of vehicle body panels once the rectification transform is applied. - The end-to-end vector-space model resolves multi-camera stitching artifacts, providing smooth, stable road geometries and unified object detections across camera seams.

[01:03:05](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3785s) Transition from Single-Camera to Multi-Camera Video Networks

Andrej Karpathy illustrates the shortcomings of single-camera predictions compared to multi-camera fusion in vector space. Displaying a video comparison where single-camera detections appear in orange and multi-camera detections appear in blue, he highlights what happens when an object crosses camera fields of view: “Basically if you can't predict these cars if you are only seeing a tiny sliver of a car so your detections are not going to be very good and their positions are not going to be good but a multi-camera network does not have an issue.” In tight spaces, vehicles crossing camera boundaries create severe prediction instability, or "jank," which breaks down entirely when encountering large vehicles. Multi-camera architectures resolve these stitching boundaries by reasoning across sensors simultaneously.

However, resolving cross-camera boundaries at a single static instant is insufficient for driving. Operating purely per-frame leaves critical driving questions unanswered: whether a car is moving or parked, its precise velocity, whether it still exists after being temporarily occluded, and what road geometry lies ahead based on signs passed tens of meters earlier. To solve this, Tesla introduced temporal context into the neural network architecture by inserting a feature queue and a video fusion module between the spatial feature extractors and the output decoding heads, augmented with ego-car kinematics such as velocity and acceleration.

```mermaid flowchart LR accTitle: Video Neural Network Architecture accDescr: Pipeline showing feature extraction, queuing across time and space, temporal fusion, and task decoding heads. subgraph InputStage["Inputs"] CAM["Multi-Camera Images"] KIN["Kinematics (v, a)"] end subgraph FeatureStage["Feature Pipeline"] FEAT["Multi-Scale Features"] QUEUE["Feature Queue (Time & Space)"] VID["Video Module (Temporal Fusion)"] end subgraph OutputStage["Prediction Heads"] HEADS["Vector Space Heads (Road, Objects, Depth)"] end CAM --> FEAT FEAT --> QUEUE KIN --> QUEUE QUEUE --> VID VID --> HEADS ```

[01:04:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3894s) Dual Caching: Time-Based and Space-Based Feature Queuing

Building a temporal feature queue requires careful engineering around memory push and pop mechanics. The queue caches concatenated spatial features, ego-kinematics, and positional encodings over time. Karpathy presents an intersection scenario to demonstrate why simple time-based queues fail in autonomous driving: an ego car approaches an intersection, encounters cross-traffic that occludes oncoming vehicles, and then waits at a red light.

If the network relies strictly on a time-based queue (for instance, pushing features every 27 milliseconds), it handles temporary dynamic occlusions effectively by referencing past frames. However, if the car waits stationary at a red light for an extended period, a time-based buffer will overwrite and purge critical environmental context—such as road markings, painted turn arrows, or overhead lane-assignment signs observed 50 meters back. To counteract this limitation, the architecture incorporates both time-based and space-based feature queues:

- **Time-based queue**: Pushes incoming frames at fixed time intervals (e.g., every 27 milliseconds) to preserve high-frequency temporal motion, tracking dynamic obstacles and brief occlusions. - **Space-based queue**: Pushes cached feature representations whenever the vehicle travels a fixed incremental distance, preserving static road markers, lane assignments, and turning constraints indefinitely while the vehicle is idling at red lights or in heavy traffic.

[01:06:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4014s) Spatial Recurrent Neural Networks (RNN) and Feature-Level Mapping

When evaluating temporal fusion architectures, the team benchmarked 3D convolutions, standard transformers, axial transformers (evaluated for computational efficiency), and numerous RNN variants. The approach they favored was a spatial recurrent neural network (Spatial RNN). Because driving naturally takes place on a two-dimensional ground plane, the network arranges its hidden state into a 2D spatial lattice. As the ego car maneuvers, ego-kinematics continuously integrate vehicle motion into the grid coordinates, updating the RNN state selectively only where the car is physically located and has clear visibility.

```mermaid flowchart TD accTitle: Spatial RNN Hidden State Updating accDescr: Spatial recurrent network updating a 2D ground plane lattice selectively based on visibility and kinematics. KIN["Ego Kinematics Integration"] --> GRID["2D Hidden State Lattice"] FEAT["Multi-Camera Features"] --> GATE{"Selective Read/Write Gate"} GATE -- "Clear Visibility" --> UPDATE["Update Local Spatial Cells"] GATE -- "Occluded" --> RETAIN["Retain Prior Hidden Memory"] UPDATE --> GRID RETAIN --> GRID GRID --> MAP["Implicit Feature HD Map"] ```

During training optimization, individual channels within the hidden state specialize autonomously, tracking structural elements such as road surfaces, curbs, centerlines, and boundaries. By analyzing activations (such as the mean of the first 10 channels across intersection runs), Karpathy demonstrates how the network dynamically selectively reads and writes memory. If an adjacent vehicle blocks the view, the network skips writing over those spatial coordinates; as soon as the line of sight clears, it commits the newly observed road geometry into memory. Karpathy notes the broader architectural implications of this approach: “Number of cars a number of clips could be collaborating to build this map basically and effectively an HD map except it's not in a space of explicit items it's in a space of features of a recurrent neural network which is kind of cool I haven't seen that before.”

[01:09:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4175s) Dynamic Occlusion Handling and Radar Removal

The video architecture produces significant performance gains in both dynamic object persistence and continuous kinematic estimation. Demonstrating an occlusion event where passing traffic momentarily obscures two parked cars, Karpathy compares the output modes: - **Single-frame networks**: Experience total dropouts when objects are occluded, output duplicate bounding boxes from multiple cameras across boundary lines, and fabricate erratic, incorrect bounding box geometry when only tiny, partial slivers of a vehicle re-emerge. - **Video networks**: Retain object states through memory buffers during full occlusion, maintain bounding box stability across camera handoffs, and suppress false geometric jumps during partial reveals because the network contextualizes partial observations with prior visibility states.

These temporal networks proved crucial for Tesla's initiative to remove physical radar sensors from production vehicles. Karpathy showcases comparison telemetry where single-frame camera predictions (orange) struggle to achieve smooth velocity and depth estimation, while the video module (blue) achieves smooth, stable velocity and distance curves that match or exceed the accuracy of physical automotive radar signals (green).

[01:11:02](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4262s) End-to-End Vision Architecture and the Vector Space

Andrej Karpathy demonstrates how temporal video modules dramatically outperform single-frame models when estimating kinematic properties. With single-frame networks, computing velocity requires differentiating noisy depth outputs over consecutive frames, whereas the video module aligns directly with physical radar ground truth. Tying together every vision component developed over the previous years, Karpathy details Tesla's modern network pipeline. Raw images from all surrounding cameras enter a rectification layer to correct for hardware calibration differences and project sensor data into a standardized virtual camera space. These standardized views pass into RegNet residual backbones to extract multi-scale features, merge through a BiFPN (bidirectional feature pyramid network), and project into 3D vector space via cross-attention transformer modules. The vector features accumulate into a spatiotemporal queue processed by a temporal video module (such as a spatial RNN), which ultimately forks into a branching HydraNet structure with individual trunks and task-specific heads.

```mermaid flowchart TD accTitle: Tesla Vision End-to-End Neural Architecture accDescr: Pipeline from raw multi-camera image feeds to multi-task HydraNet predictions in 3D vector space.

subgraph Inputs["Sensor Processing"] A["Raw Camera Images"] --> B["Rectification Layer (Virtual Camera)"] B --> C["RegNet Backbones (Multi-Scale Features)"] C --> D["BiFPN (Feature Fusion)"] end

subgraph VectorFusion["Space & Time Fusion"] D --> E["Transformer Cross-Attention (3D Vector Space)"] E --> F["Spatiotemporal Feature Queue"] F --> G["Video Module (Spatial RNN)"] end

subgraph Outputs["Multi-Task Heads"] G --> H["HydraNet Trunks & Heads"] H --> I["Dense Rasters & Kinematics"] end ```

Karpathy notes how dramatically the visual processing system has expanded: “this architecture has definitely complexified from just a very simple image based single network about three or four years ago and continues to evolve”. Despite these breakthroughs, active engineering efforts target several bottlenecks. Currently, temporal and spatial fusions happen relatively late in the network graph; the team is exploring earlier fusions via cost volumes or optical flow-style lower layers. Additionally, outputting dense rasters creates a latency and computational tax because post-processing these representations directly on car hardware is expensive under real-time constraints. Consequently, Tesla is pursuing sparse geometry predictions—such as directly generating point-by-point road structures—to bypass dense raster post-processing entirely before handing off the 3D vector space to downstream planning.

- Single-frame velocity estimation relies on differentiating depth estimates over time, whereas temporal video modules replicate the accuracy of millimeter-wave radar directly from vision. - Multi-camera rectification normalizes physical camera misalignment into a uniform synthetic viewpoint. - Output rasters currently impose heavy embedded post-processing loads; the vision team is migrating toward direct sparse point-based road geometry prediction.

---

[01:13:11](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4391s) The Planning Problem: Non-Convexity and High Dimensionality

Ashok Elluswamy, director of Autopilot software, introduces the planning, control, auto-labeling, and simulation domains. The planner's core mission is to consume the 3D vector space synthesized by the vision network and maneuver the car to its destination while maximizing safety, passenger comfort, and travel efficiency. While Autopilot handled highway environments reliably as far back as 2019—staying centered in lanes, performing tactical lane changes, and navigating interchange ramps—city driving introduces an entirely different tier of difficulty. Urban environments rarely possess pristine lane markings; instead, they feature free-form trajectories, occluded sightlines, crossing traffic, roadside curbs, and unpredictable pedestrian behaviors.

Elluswamy breaks down the primary mathematical dilemma of real-time autonomous planning into two compounding characteristics: high dimensionality and an intensely non-convex action space. The high dimensionality stems from the necessity of planning a continuous trajectory over a 10-to-15-second rolling horizon, solving for precise positions, velocities, and accelerations across every discrete time increment. Simultaneously, non-convexity implies that driving scenarios contain multiple locally optimal solutions (e.g., passing on the left versus yielding on the right) separated by severe local minima:

> “Discrete search methods are really great at solving non-convex problems because they are discrete, they don't get stuck in local minima, whereas continuous function optimization can easily get stuck in local minima and produce poor solutions that are not great.”

To harness the advantages of both approaches while avoiding their limitations—discrete search cannot scale efficiently in high dimensions because it lacks gradient information, while gradient-based continuous optimization easily traps itself in poor local minima—Tesla uses a two-stage hierarchical planner. First, a coarse discrete search explores the non-convex environment to select a topological decision corridor; second, a continuous gradient-based optimizer refines that corridor into an ultra-smooth, dynamically feasible physical trajectory.

- Trajectory rollouts must anticipate 10 to 15 seconds into the future, spanning dozens of correlated kinematic control variables. - Discrete search methods systematically hop across non-convex topological modes without gradient traps but scale poorly in high-dimensional continuous spaces. - Continuous numerical optimization rapidly converges along smooth gradients but requires a convex bounding corridor to avoid getting pinned in local minima.

---

[01:15:26](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4526s) Hierarchical Trajectory Search and Validation

To illustrate discrete corridor selection, Elluswamy presents an unprotected urban maneuver requiring two consecutive lane changes in rapid succession to execute an upcoming left turn. A naive planner might execute an immediate, aggressive lane change, but hitting the brakes hard causes severe passenger discomfort. Conversely, waiting to accelerate past surrounding traffic down the road results in making the lane change too late, overshooting the turning pocket entirely.

```mermaid flowchart LR accTitle: Hierarchical Planner Architecture accDescr: Flow diagram showing discrete coarse search establishing a convex corridor followed by continuous spline optimization.

subgraph Stage1["Discrete Search"] A["3D Vector Scene"] --> B["Physics-Based Rollouts"] B --> C["Candidate Selection (Safety & Comfort)"] end

subgraph Stage2["Continuous Optimization"] C --> D["Convex Corridor Bounding"] D --> E["Spline Parameterization (Arc Length)"] E --> F["Fine Trajectory (Minimized Jerk & Loss)"] end ```

The system resolves this tradeoff by evaluating thousands of candidate trajectories in fractions of a second using physics-based forward simulation. Each rollout is scored against an explicit cost function balancing collision risk, longitudinal deceleration limits, lateral jerk, and navigation progress. Once the discrete search isolates the optimal sequence, the vehicle executes the maneuver. Telemetry data comparing planned velocity curves against actual recorded execution confirms that the vehicle tracks the planned 10-second trajectory profile with near-perfect fidelity in real-world driving conditions.

- Candidate trajectories are evaluated across multiple criteria, ruling out harsh braking or delayed turns that risk missing routing waypoints. - Physics-based simulation engines simulate thousands of dynamic candidate maneuvers within milliseconds on embedded hardware. - Real-time vehicle telemetry demonstrates tight alignment between planned 10-second longitudinal velocity curves and actual vehicle execution.

---

[01:16:48](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4608s) Joint Multi-Agent Planning in Interactive Driving

Navigating complex human environments requires recognizing that the autonomous vehicle does not operate inside a static vacuum. Safe and natural driving demands anticipating the interactive responses of every surrounding road user. As Elluswamy explains:

> “When driving alongside other agents it's important to not just plan for ourselves but instead we have to plan for everyone jointly and optimize for the overall scenes traffic flow.”

To achieve this, the car does not treat surrounding obstacles as passive bounding boxes moving along static velocity vectors; it actively runs instances of the Autopilot planning algorithm for every relevant nearby vehicle in the scene.

Elluswamy demonstrates this principle using footage of an Autopilot vehicle traversing a narrow, obstructed road corridor flanked by closely parked vehicles, construction cones, and poles. When the first oncoming car approaches, Autopilot briefly decelerates, observes that its own path offers no lateral pull-out space, but calculates that the oncoming vehicle has ample shoulder space and a low enough speed to pull over safely. Rather than phantom-braking or stalling, the system recognizes the mutual game-theoretic dynamic and drives assertively through the opening.

When a second oncoming vehicle approaches at a significantly higher speed, the planner evaluates two branching hypotheses: 1. A yielding hypothesis (represented visually as a green trajectory) where the opposing car slows down. 2. An aggressive bypass hypothesis (represented as a red trajectory) where the driver swerves around parked obstacles directly into Autopilot's lane.

Given the oncoming vehicle's high approach speed and acceleration, Autopilot infers that the driver will not yield and proactively steers into an available side pocket. However, as the oncoming car initiates a turn and alters its yaw rate and deceleration to yield, Autopilot detects this behavioral shift in real time, immediately updates its joint plan, and resumes forward progress. Without this capability to jointly project and react to the intent of other agents, Elluswamy emphasizes that Autopilot would either drive far too timorously or freeze entirely, making autonomous city driving impractical.

- Surrounding dynamic agents are modeled by executing the Autopilot planner from their perspective to predict realistic path intentions. - Game-theoretic joint modeling prevents deadlocks on narrow roads by identifying which vehicle has geometric clearance to pull over and yield. - High-frequency tracking of an oncoming vehicle's subtle yaw rate and longitudinal acceleration allows the planner to adapt within fractions of a second when an opposing driver decides to yield.

---

[01:19:03](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4743s) Continuous Convex Corridor Optimization

Once discrete search and multi-agent intent modeling identify the optimal sequence of actions, the planner constructs a geometric bounding corridor free of non-convex obstacles. Inside this convex corridor, continuous optimization refines the trajectory into an actionable path. The planner initializes a mathematical spline defined in heading, curvature, and acceleration, parameterized explicitly over the arc length of the target path rather than simple time steps.

During optimization, gradient-based numerical routines modify the spline parameters to minimize a combined objective loss function. This objective penalizes proximity to static and dynamic obstacles, excessive total travel time, and sharp vehicle dynamics. Analyzing the lateral acceleration profile generated by this continuous pass reveals smooth, symmetrical trapezoidal curves—a mathematical signature that mirrors the natural driving habits of skilled human drivers while driving lateral jerk down to near zero.

- The coarse discrete search generates a convex spatial corridor that shields the continuous numerical optimizer from local minima traps. - Trajectory splines are parameterized across arc length, optimizing heading, curvature, and acceleration simultaneously. - Objective functions explicitly penalize obstacle margins, travel delays, and comfort metrics, yielding smooth trapezoidal lateral acceleration curves that eliminate steering jerk.

[01:19:53](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4793s) Learning-Based Planning and the Toy Parking Problem

The discussion transitions from structured highway driving to the chaos of real-world urban navigation. Ashok Elluswamy observes that in many parts of the world—including where he grew up—driving is inherently unstructured: cars and pedestrians cut across lanes, drivers brake sharply, and honking is constant. He remarks:

> “We can try to scale up these methods, but it's going to be really difficult to efficiently solve this at runtime. What we instead want to do is use learning-based methods to efficiently solve them.”

To illustrate why traditional heuristics fail, Elluswamy walks through a simplified toy parking simulation. The ego car (rendered in blue) must navigate around curbs, parked vehicles, and orange traffic cones to reach a designated green parking bay. Testing a baseline A* search using standard Euclidean distance to the goal demonstrates catastrophic inefficiency: the planner drives straight toward the target, becomes trapped in a local minimum behind an obstacle, backtracks, and winds up expanding 400,000 nodes before reaching the goal. Augmenting A* with a navigation route helps initially, but encountering unmapped cones still forces the algorithm to backtrack blindly and explore alternative paths without knowing obstacles lie ahead until a collision check fails. That navigation-guided search still requires 22,000 node evaluations. Designing manual heuristics or distance functions to circumvent each obstacle is tedious, hand-tuned, and unable to yield a globally optimal value function.

```mermaid flowchart TD accTitle: Evolution of Planning Approaches in Toy Parking accDescr: Comparison of A* search variants versus neural network Monte Carlo Tree Search node counts.

subgraph Heuristic["Classical Heuristic Search"] A["Standard A* Search (Euclidean Distance Heuristic)"] -->|"Trapped in local minima blind backtracking"| A1["400,000 Nodes Expanded"] B["A* + Navigation Route Heuristic"] -->|"No obstacle awareness checks collision at contact"| B1["22,000 Nodes Expanded"] end

subgraph Learned["Learning-Based Planning"] C["Neural Network Value Function (Global Scene Context)"] -->|"Monte Carlo Tree Search One-shot direct path"| C1["288 Nodes Expanded"] end ```

[01:22:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4935s) Neural Networks and Monte Carlo Tree Search for Trajectory Planning

Because the vision system compresses camera inputs into a vector space containing moving dynamic agents, Elluswamy compares the self-driving problem directly to a multiplayer Atari game. Consequently, techniques like AlphaZero and MuZero can be brought to bear. Tesla trains neural networks to predict state and action distributions that feed into Monte Carlo Tree Search (MCTS). The search is guided by explicit cost functions—including collision risk, passenger comfort, and traversal duration—combined with imitation signals derived from interventions during human driving events.

When tested on the exact same parking scenario, the MCTS planner equipped with a neural network value function navigates directly to the goal in one clean shot without even needing a navigation route heuristic. While alternate branching paths are evaluated, the planner stays locked on the optimal trajectory because the network digests the global context of the entire scene, yielding a value function that steers straight toward the global optimum instead of stumbling into local minima. This approach reaches the destination using only 288 nodes—several orders of magnitude faster than baseline A*. In the target production architecture, raw video streams are crushed into vector space, which is then consumed simultaneously by an explicit planner and a neural network planner alongside intermediate neural features. Together they output a trajectory distribution optimized end-to-end against explicit safety costs and human intervention data, after which downstream explicit planning functions generate the final steering and acceleration commands.

- Baseline A* with Euclidean distance heuristic: ~400,000 nodes expanded. - A* combined with a navigation path heuristic: ~22,000 nodes expanded. - Neural network-guided MCTS using global scene context: 288 nodes expanded. - Search cost functions integrate collision checks, riding comfort, trip duration, and human disengagement/intervention events.

[01:24:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5075s) Manual Labeling: In-House Operations and Vector Space Annotation

Andrej Karpathy returns to address the massive data pipelines required to train these complex networks, noting that model capacity is only as good as the underlying data. He stresses the fundamental relationship between network architecture and dataset scale:

> “So neural networks are just an upper bound; you also need massive datasets to actually train the correct algorithms inside them.”

When Karpathy joined Tesla roughly four years earlier, the company outsourced dataset collection to third-party vendors. The team rapidly discovered that third-party vendors were unsuitable for safety-critical autonomous driving: turnaround latency was crippling and label quality was inadequate. In keeping with Tesla's culture of vertical integration, labeling was brought completely in-house. Tesla built a dedicated organization of over 1,000 professional data labelers based in the United States and co-located near the engineering teams in California, backed by proprietary software tracking latency, per-annotator throughput, and error metrics.

Labeling methodology also underwent a major paradigm shift. Three to four years prior, labelers drew 2D polygons and polylines on single, isolated camera images, a slow and disconnected workflow. Tesla transitioned to 3D and 4D vector space labeling: reconstructing the ground plane driven by the vehicle along with a point cloud, allowing annotators to label bounding boxes and features directly in 3D physical coordinates. These 3D vector labels are then automatically reprojected into all surrounding 2D camera views across time, generating massive multi-camera training targets from a single annotation action.

- Tesla's in-house labeling team grew to over 1,000 dedicated labelers working in close physical proximity with Autopilot engineers. - Custom internal tooling tracks labeling latency, throughput, individual efficiency, and cumulative label growth. - Paradigm shift moved from 2D per-image polygon annotation to 3D/4D vector space labeling with automatic multi-camera reprojection.

[01:27:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5259s) The Transition to Automated Labeling

Despite a 1,000-person dedicated human labeling force, human manual annotation cannot keep up with the data appetite of neural networks containing hundreds of millions of parameters. Karpathy outlines the division of labor between biological and machine intelligence:

> “So people are extremely good at things like semantics, but computers are very good at geometry, reconstruction, triangulation, tracking.”

Because computers excel at dense geometric math while humans excel at high-level semantic identification, Tesla shifted toward automated labeling systems where machines handle heavy reconstruction and humans provide semantic validation or edge-case refinement. The presentation then moves into the architecture of Tesla's auto-labeling pipeline, designed to process millions of driving clips containing dense video and vehicle telemetry at fleet scale.

[01:28:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5308s) Auto-Labeling the Road Surface via Implicit Representations

Auto-labeling begins by gathering 45-second to one-minute video clips accompanied by IMU, GPS, and odometry telemetry, uploaded from both internal engineering vehicles and customer customer cars across the fleet. These raw streams are ingested by server clusters running massive neural networks offline to extract intermediate representations—including semantic segmentation masks, depth estimations, and feature point correspondences. These primitives are then processed through robotics and optimization algorithms to generate high-fidelity ground truth training labels.

```mermaid flowchart LR accTitle: Tesla Auto-Labeling Pipeline accDescr: Pipeline showing data collection, offline server inference, multi-trip optimization, and training label generation.

subgraph FleetData ["Fleet Collection"] A["IMU, GPS & Odometry"] B["Multicam Video Clips"] end

subgraph OfflineServer ["Server-Side Processing"] C["Offline Neural Nets (Depth, Seg, Matching)"] D["Implicit Field (x,y -> z) NeRF-style Optimization"] E["Multi-Trip Alignment (16+ Passes)"] end

subgraph GroundTruth ["Final Supervision"] F["Dense 3D Labels & Kinematics"] G["Video Neural Net Training"] end

A --> C B --> C C --> D D --> E E --> F F --> G ```

A core challenge is accurately labeling the road surface in 3D. Traditional representations like splines or polygonal meshes struggle because topological constraints make them non-differentiable and cumbersome to optimize across complex terrain. Instead, Tesla adopts an implicit representation inspired by the Neural Radiance Fields (NeRF) formulation. The system repeatedly queries ground coordinates $(x, y)$, prompting the network to predict the surface elevation $z$ alongside rich semantic properties such as curbs, lane lines, road boundaries, and drivable space. This reconstructs a continuous 3D coordinate $(x, y, z)$ that can be back-projected into all camera viewpoints. By generating millions of queries and jointly minimizing the reprojection error against 2D image segmentation predictions over space and time, the model produces a temporally consistent ground surface.

While a single vehicle sweep reconstructs the immediate corridor along its path, Tesla stitches multiple passes from different vehicles together into a joint optimization problem. Sixteen separate trips through the same intersection can be aligned using shared static features—such as lane lines and road curbs—so that all passes agree with one another and align with their respective image-space observations. As the speaker clarifies: “Again, the point of this is not to just build HD maps or anything like that; it's only to label the clips through these intersections so we don't have to maintain them forever, as long as the labels are consistent with the videos that they were collected at.” If necessary, human annotators can inspect these outputs to remove residual noise or inject high-level metadata.

- Clips sent to offline compute clusters typically run between 45 and 60 seconds long. - Elevation and semantics are queried continuously in continuous space rather than discretizing into rigid topological meshes. - 16 or more trajectories through a single intersection are fused to reconstruct areas where an individual vehicle never drove.

---

[01:31:21](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5481s) Dense 3D Point Clouds and the Benefit of Hindsight

Beyond the drivable surface, the server-side auto-labeling engine reconstructs dense 3D point clouds of arbitrary static obstacles. Classical structure-from-motion techniques rely on sharp visual textures to establish point correspondences across frames, causing them to fail on uniform surfaces. Tesla's approach overcomes this limitation, resolving high-density point clouds even across featureless asphalt and plain walls.

```mermaid flowchart TD accTitle: Benefit of Hindsight in Auto-Labeling accDescr: How offline future temporal context resolves occlusions and kinematics compared to online inference.

subgraph OnlineInference ["Online In-Car Model"] direction TB P1["Past Frames Only"] --> Guess["Estimated Kinematics (Prone to Occlusion Loss)"] end

subgraph OfflineHindsight ["Offline Auto-Labeler"] direction TB F1["Past Frames"] & F2["Future Frames"] --> Solve["Exact Kinematics (v, a) & Track Stitching Across Occlusions"] end ```

Running these reconstructions offline unlocks an asymmetry that real-time vehicle computers cannot exploit: temporal hindsight. In real time, the in-car neural network can only rely on historical frames to estimate an object's instantaneous velocity and acceleration. Offline servers, by contrast, possess the full recording from start to finish. The speaker highlights this structural advantage: “This is a super useful hack because, say in the car, when the network needs to produce the velocity, it just has to use the historical information and guess what the velocity is. But here, we can look at both the history but also the future, and basically cheat and get the correct answer of the kinematics like velocity, acceleration, etc.”

Access to future trajectory data also resolves tracking loss across prolonged occlusions. By comparing past tracks with reappearing tracks downstream, the pipeline stitches entities together even when hidden behind intermediate geometry. For example, pedestrians on the opposite side of a street remain tracked as a single identity through repeated occlusions by parked and passing vehicles. This supervisory signal directly trains the vehicle's planner, which must preserve occluded obstacles in memory. Tesla aims to generate over one million of these comprehensive auto-labeled clips—containing smooth vehicle paths, verified pedestrian kinematics, and explicit stationary tags for parked cars—to supervise its multi-camera video models.

---

[01:33:36](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5616s) Removing Radar in Three Months with Fleet-Driven Auto-Labeling

The auto-labeling pipeline underwent its trial by fire during the project to transition the vehicle stack off radar sensors, accomplished in roughly three months. Early pure-vision networks struggled during extreme visual edge cases, such as a preceding truck kicking up a dense cloud of snow that blinded forward-facing cameras. The network needed to maintain an active spatial memory of the vehicle ahead despite zero instantaneous optical visibility, but lacked sufficient training samples depicting such rare obscurations.

To solve this, Tesla queried its customer fleet for matching edge-case scenarios, and the fleet surfaced clips capturing rare obscurations, including instances where cargo and debris fell from moving vehicles. “And we send this through our auto-labeling pipeline that was able to label 10K clips within a week; this would have taken several months with humans labeling every single clip here,” the speaker notes. Tesla replicated this workflow across 200 distinct driving conditions, amassing massive datasets that taught the vision networks to retain persistent obstacle positions even when visibility dropped to zero.

- Transition from radar-vision fusion to pure vision was completed in roughly three months. - Fleet triggers identified visual edge cases across 200 separate conditions. - Auto-labeling processed 10,000 targeted video clips in one week, replacing months of manual frame-by-frame labeling.

---

[01:34:52](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5692s) Photorealistic Simulation for Rare Edge Cases and Closed-Loop Control

To demonstrate the capability of their simulation platform, the speaker presents a driving clip containing a Tesla Cybertruck navigating an intersection, challenging the audience to guess its origin. After a brief pause, an audience member correctly guesses it is synthetic. “Someone said it, yes, yes, it’s rendered, it’s our simulation,” the presenter confirms, laughing. “It was hard for me to tell initially, and if I may say so myself, it looks very pretty.”

```mermaid flowchart LR accTitle: Simulation Use Cases accDescr: Three core domains where simulation provides ground truth over real-world data collection.

Sim["Simulation Engine"] --> Rare["Rare Edge Cases (Pedestrians on Highways)"] Sim --> Dense["Extreme Density Labeling (Hundreds of Pedestrians)"] Sim --> Closed["Closed-Loop Simulation (Action-Dependent Dynamics)"] ```

The rendering features photorealistic environmental complexity: uneven ground surfaces scored with cracks, tar seams, and asphalt patches; realistic vehicle dynamics where an articulated semi-truck swings wide and mounts a curb; and naturalistic traffic interactions where simulated cars brake, accelerate smoothly, and avoid collisions. Operating directly within this digital environment, an Autopilot-controlled agent executes unprotected left turns. Because the simulation is synthesized from an underlying vector space, ground truth extraction is mathematically exact. The pipeline produces vehicle 3D bounding cuboids with kinematics, ground-truth depth, surface normals, and dense segmentation masks with zero manual labeling overhead. If engineers devise a new perception task, labeling code can be updated within days.

The presenter outlines three specific operational domains where simulation outmatches physical fleet collection: 1. **Rare, dangerous edge cases:** Scenes that are nearly impossible to harvest safely from fleet driving, such as a couple and their dog running across a high-speed highway. 2. **Dense, occluded scenes:** Crowded downtown intersections where hundreds of pedestrians cross simultaneously. While human annotators would spend hours per clip and auto-labelers struggle with temporal association, the simulator yields ground-truth object cuboids and velocities. 3. **Closed-loop behavior:** Testing scenarios where the ego-vehicle's driving decisions directly affect environmental reactions, requiring deterministic, closed-loop simulation to evaluate policy safety.

[01:37:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5848s) High-Fidelity Sensor Simulation and Photorealism

Ashok Elluswamy explains that the core purpose of Tesla's simulation platform is not simply generating pleasing visuals, but reproducing precisely what onboard vehicle cameras and sensors observe in the real world. Demonstrating side-by-side video clips of an actual camera alongside the simulation across varying exposure steps, the synthetic environment matches the real camera feed down to subtle imaging artifacts. Achieving this degree of fidelity requires rigorous modeling of physical sensor dynamics, including sensor noise, motion blur, optical lens distortions, windshield diffraction patterns, and headlight beam transmissions. Beyond evaluating software, Tesla leverages this fine-grained physical sensor modeling to inform downstream hardware decisions, such as custom lens design, camera specifications, vehicle sensor placement, and headlight transmission characteristics.

```mermaid flowchart TD accTitle: Real Camera Matching Pipeline accDescr: Pipeline showing physical sensor modeling feeding both Autopilot software evaluation and hardware design decisions.

subgraph Inputs["Physical Sensor Modeling"] A["Sensor Noise & Motion Blur"] B["Optical Lens Distortion"] C["Windshield Diffraction & Headlight Optics"] end

subgraph Engine["Rendering Engine"] D["Spatio-Temporal Anti-Aliasing (TAA)"] E["Ray Tracing (Global Illumination)"] F["Neural Rendering"] end

subgraph Applications["Downstream Applications"] G["Autopilot Perception & Policy"] H["Hardware Design Decisions\n(Lenses, Sensor Placement)"] end

Inputs --> Engine Engine --> Applications ```

Visual realism also hinges on eliminating synthetic rendering giveaways. Aliasing artifacts—commonly known in the video game industry as "jaggies"—immediately telegraph to a neural network that it is processing simulated data rather than the physical world. To eliminate these visual cues, the engineering team relies on advanced spatio-temporal anti-aliasing (TAA), ray tracing for physically accurate lighting and global illumination, and neural rendering techniques that push visual realism even closer to physical sensor captures.

- “Again the point of the simulation is not to just produce pretty pictures; it needs to produce what the camera in the car would see and other sensors would see,” Ashok Elluswamy emphasized regarding their strict physical accuracy criteria.

[01:38:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5936s) Procedural Road Generation, Edge Cases, and Failure Replay

To prevent neural networks from overfitting on fixed shapes and dimensions, Tesla populates its simulation world with a vast catalog of diverse assets, ranging from pedestrians wearing varied clothing and exhibiting realistic motion profiles to uncommon road obstacles such as a moose walking directly across the travel lane. The team has built over 2,000 miles of hand-crafted simulated road surfaces—an expanse roughly equivalent to driving across the United States from coast to coast—backed by tooling that lets a single artist generate several miles of roadway in a single day. However, artist-crafted scenarios represent only a small fraction of the training corpus; the vast majority of simulation training data is synthesized procedurally using algorithms. These procedural pipelines dynamically modulate road curvature, foliage density, traffic cones, utility poles, and surrounding vehicles with variable velocities and interaction profiles, producing an endless stream of novel training scenarios.

```mermaid flowchart LR accTitle: Failure Replay and Closed-Loop Synthesis accDescr: Workflow from real-world autopilot failure to procedural 3D reconstruction and synthetic counterfactual re-simulation.

A["Real Fleet Failure Clip"] --> B["Auto-Labeling Pipeline\n(3D Scene Reconstruction)"] B --> C["Synthetic Simulation Scene"] C --> D["Neural Rendering Filter"] D --> E["Autopilot Counterfactual Replay\n(New Outcomes & Closed Loop)"] ```

Because broad procedural generation often creates mundane driving scenarios that the network can already resolve easily, Tesla deploys closed-loop machine-learning routines to identify model vulnerabilities and intentionally synthesize complex variations centered on those exact failure modes. When an Autopilot intervention or failure occurs in the customer fleet, that specific video clip is run through Tesla's offline auto-labeling pipeline to reconstruct a high-precision 3D digital twin of the static environment and all dynamic participants. Autopilot is then replayed inside this synthetic replica, where alternative vehicle behaviors can spawn counterfactual outcomes and new scenarios to ensure the network is permanently hardened against that failure mode. Applying neural rendering models directly on top of these reconstructed scenes yields near-photorealistic outputs. The onboard production networks have already consumed 300 million simulated images and roughly half a billion synthetic labels.

- “This is amazing because we really don't want Autopilot to fail, and when it fails we want to capture it and keep it to that bar,” Ashok stated when walking through the automated failure reconstruction pipeline.

[01:42:09](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6129s) Hardware Integration and In-Car Neural Network Deployment

Milan Kovac takes the stage to explain how Tesla scales data generation and operationalizes neural networks on edge silicon inside customer vehicles. Transitioning Autopilot away from radar dependencies required monumental compute resources: the team generated more than 10 billion labels across 2.5 million video clips by distributing auto-labeling networks and simulation instances across thousands of GPUs, nearly 20,000 CPU cores, and more than 2,000 physical Full Self-Driving (FSD) computers executing hardware-in-the-loop simulation—which Milan humorously noted is actually their smallest compute cluster.

Deploying large vision networks directly onto the production vehicle requires satisfying strict frame rate and latency bounds, which are essential for passenger safety and for calculating precise velocity and acceleration vectors of dynamic actors. The core engineering effort centers on Tesla's proprietary in-house AI compiler, which translates PyTorch model operations directly into hardware-accelerated instructions on the custom FSD chip. The compiler constructs execution schedules designed for maximum throughput under stringent on-chip SRAM memory limits. Furthermore, the FSD computer incorporates two independent compute engines; while one actively dispatches steering and acceleration commands to vehicle actuators, the secondary unit functions as a parallel compute accelerator, with both engines fully redundant and interchangeable across hardware and software boundaries.

- “And tonight I'd just like to start by giving you some perspective into the amount of compute that's needed to power this type of data generation factory,” Milan Kovac observed when introducing the infrastructure scale behind Tesla's vision-only milestone.

[01:44:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6254s) Evaluation Clusters, Developer Tooling, and Training Infrastructure

To maintain developer velocity across hundreds of concurrent model iterations, Tesla built an expansive automated validation pipeline. The system runs more than one million neural network evaluations every week across any new pull request or code change submitted by the autonomy team. These continuous automated regressions execute across an isolated cluster composed of over 3,000 real FSD computers networked together to replicate in-vehicle silicon performance down to the exact clock cycle.

In parallel with automated batch evaluation, Tesla engineers utilize custom visual inspection tools that allow researchers to perform live debugging. These tools run different iterations of a neural network side by side on historical video sequences, rendering bounding volumes, vector predictions, and occupancy grids in real time so engineers can observe minute behavioral regressions or improvements. Fueling the foundation of all these perception networks is an internal training footprint operating just shy of 10,000 dedicated GPUs—a compute footprint surpassing the aggregated GPU capacity of the five highest-ranked publicly known supercomputers in the world. Yet, as the ambition and dimensionality of Tesla's driving models expand, even this compute infrastructure faces significant scaling walls.

- Key fleet validation and training infrastructure metrics: - **Weekly software evaluations:** >1,000,000 automated evaluations per week on incoming code changes. - **Hardware-in-the-loop validation farm:** >3,000 networked in-house FSD computers dedicated to CI/CD evaluation. - **Fleet data generation cluster:** >2,000 FSD computers, ~20,000 CPU cores, and thousands of GPUs. - **Active training infrastructure:** Just shy of 10,000 GPUs dedicated solely to neural network training.

[01:45:40](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6340s) Project Dojo: The Architectural Challenge of AI Training

Ganesh Venkataramanan, who leads Project Dojo, steps up to present Tesla's answer to its escalating compute requirements. Years earlier, anticipating that standard compute platforms would inevitably choke on the data volumes demanded by autonomous driving, Elon Musk instructed the team to design a bespoke high-throughput training supercomputer from scratch. The mandate behind Project Dojo is straightforward: achieve peak AI training performance capable of processing the massively larger neural network architectures envisioned by Andrej Karpathy's team, while maximizing power efficiency and cost-effectiveness.

Ganesh breaks down the fundamental limitations plaguing traditional high-performance distributed architectures. Almost every distributed training computer connects modular compute elements—whether CPUs, GPUs, or bespoke ASICs—across two-dimensional or multidimensional networks. While semiconductor manufacturers find it relatively straightforward to add raw compute units inside a chassis, scaling interconnect bandwidth across distinct nodes remains exceptionally hard, and driving down communications latency across distributed clusters is an even steeper challenge. Addressing this severe communications bottleneck forms the foundational premise of Dojo's architectural design.

- “Elon prefetched this, and a few years back he asked us to design a super fast training computer, and that's how we started Project Dojo,” Ganesh Venkataramanan explained regarding the origin of Tesla's custom supercomputing platform. - “One thing which is common trend amongst this is it's easy to scale the compute; it's very difficult to scale up bandwidth and extremely difficult to reduce latencies,” Ganesh pointed out when diagnosing the core bottleneck of distributed AI training.

[01:47:10](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6430s) Dojo Design Philosophy and the Training Node Architecture

Tesla set out to break the architectural limits of traditional supercomputing by designing Dojo from the inside out as a unified compute plane. The core vision centers on a massive two-dimensional mesh of robust compute elements backed by large pools of local memory and tied together with an ultra-high-bandwidth, low-latency interconnect fabric. At this scale, massive neural networks are partitioned across the plane using model, graph, and data parallelism. Tesla's neural compiler maps workloads to exploit both spatial and temporal locality, containing communications within tight local zones to avoid choking global interconnects. As Ganesh Venkataramanan explained:

> “We wanted to attack this all the way top to the bottom of the stack and remove any bottlenecks at any of these levels.”

The foundational building block of this architecture is the "training node." Selecting the physical boundary of this node involved balancing strict hardware trade-offs: making the node too small would maximize raw clock speed but swamp the system with synchronization and software overhead, whereas making it too large would introduce physical layout complexities and run headlong into memory bottlenecks. Tesla engineered the boundary around physics: engineers determined the absolute maximum physical distance an electrical signal could travel inside a single clock cycle at over 2 GHz, drew a box around that physical perimeter, and packed every available routing channel inside it with wires to maximize bandwidth. Beneath this wiring, they placed machine learning compute engines, a large pool of SRAM, and a programmable core to control execution.

```mermaid flowchart TD accTitle: Dojo Scaling Hierarchy accDescr: Architectural progression from individual training node to the full ExaPOD supercomputer.

subgraph Silicon ["Silicon Level"] A["Training Node (>1 TFLOP, 1.25 MB SRAM)"] -->|"354 Nodes + 576 SerDes"| B["D1 Chip (22.6 TFLOPs, 50B Transistors)"] end

subgraph Packaging ["Wafer & Module Integration"] B -->|"25 D1 Dies on Fan-Out Wafer"| C["Training Tile (9 PFLOPs, 52V DC VRM)"] end

subgraph System ["Cluster & Datacenter Scale"] C -->|"6 Tiles (2x3) per Tray 2 Trays per Cabinet"| D["Dojo Cabinet (100 PFLOPs)"] D -->|"10 Seamless Cabinets"| E["Dojo ExaPOD (1.1 ExaFLOPs, >1M Nodes)"] end ```

The resulting training node is a 64-bit superscalar CPU core optimized around dedicated matrix multiply units and SIMD vector pipelines. Architecturally, it operates as an in-order machine featuring four-wide scalar and two-wide vector execution pipes—though the scalar and vector pipes can execute out of order relative to one another. To keep execution units saturated, each core incorporates four-way multithreading, permitting compute tasks and memory transfers to run simultaneously without stalling the engine.

- **Arithmetic formats supported**: FP32, BFLOAT16, and a proprietary configurable FP8 (CFP8) format. - **On-chip memory**: 1.25 megabytes of high-speed, ECC-protected SRAM per training node. - **Node performance**: Over 1 teraflop of machine learning compute per individual node. - **Custom instruction set architecture (ISA)**: Machine learning-specific instructions including hardware-accelerated transpose, gather, broadcast, and link traversals. - **Modular layout**: Physical node boundaries are designed to abut seamlessly in any 2D direction to create arbitrary compute arrays.

---

[01:51:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6699s) The D1 Chip: Pure Machine Learning Silicon

When 354 individual training nodes are arrayed together into a contiguous grid, they form the core compute engine of Tesla's custom processor, delivering 22.6 teraflops of compute. To feed this matrix without memory starvation, Tesla ringed the perimeter of the array with 576 high-speed, low-power SerDes lanes. This perimeter interface delivers double the off-chip I/O bandwidth of top-tier commercial networking switch chips, which traditionally define the gold standard for communication throughput.

The fully integrated result is the D1 chip. Fabricated on an advanced 7-nanometer semiconductor process node, the D1 packs 50 billion transistors into a compact 645-square-millimeter die. Unlike general-purpose server processors or standard GPUs, the D1 devotes 100 percent of its silicon footprint exclusively to machine learning compute and high-speed communication fabric. There is no dark silicon, no graphics legacy hardware, and no obsolete architectural overhead. As Ganesh summarized the achievement:

> “This chip is like a GPU level compute with a CPU level flexibility and twice the network chip level I/O bandwidth.”

Every layer of the D1 chip was developed in-house by Tesla's semiconductor team, spanning initial instruction set architecture, microarchitecture design, physical floorplanning, tape-out (GDS out), and advanced flip-chip Ball Grid Array (BGA) package integration. On an architectural plot comparing raw teraflops against available I/O bandwidth, the D1 occupies an uncontested design point that drastically outpaces both conventional commercial accelerators and contemporary AI startup hardware.

---

[01:54:06](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6846s) The 9-Petaflop Training Tile: Wafer-Scale Integration and Vertical Power

Because D1 chips feature perimeter interfaces designed to communicate seamlessly without glue logic, Tesla can tile them directly into larger compute planes. At the datacenter interface layer, Tesla connects these planes through custom Dojo Interface Processors (DIPs). The DIP acts as a host bridge to standard datacenter servers over PCIe Gen4, supplies massive pools of shared high-bandwidth DRAM directly to the D1 array, and provides higher-radix network routing across the wider system.

To build the compute plane at scale, Tesla bypassed standard printed circuit board limitations and created the "training tile"—the foundational unit of scale for the Dojo supercomputer. Each training tile integrates 25 known-good D1 dies onto an advanced fan-out wafer process, creating an organic Multi-Chip Module (MCM) that maintains uninterrupted die-to-die bandwidth across all 25 chips. Custom high-density, high-bandwidth interconnects clip directly to the wafer perimeter to export tile-level I/O without pinching communications. The mechanical and electrical scale was so extreme that standard electronic design automation (EDA) software crashed during modeling, forcing Tesla's engineering team to build proprietary simulation tools and algorithms to complete the design.

Power delivery presented an equally daunting challenge: routing thousands of amperes laterally through circuit board traces would have caused massive resistive losses and blocked signal paths. Tesla solved this by implementing vertical power delivery. Engineers designed custom Voltage Regulator Modules (VRMs) that reflow directly onto the fan-out wafer assembly, stacking the power components directly above the silicon dies.

- **Integrated mechanics**: Combines semiconductor die, advanced fan-out packaging, PCB-level component reflow, direct thermal cold plates, and structural load-bearing hardware into one self-contained assembly. - **Power input**: Feeds directly on a 52-volt DC input bus. - **Orthogonal architecture**: The compute interconnect plane runs horizontally (X-Y), while power delivery and liquid cooling flow vertically (Z-axis), ensuring thermal and electrical delivery never bottleneck communication lines. - **Tile compute throughput**: 9 petaflops of machine learning training performance on a single module.

---

[01:58:09](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7089s) Live Hardware Demo and the ExaPOD Cluster Architecture

To prove the concept had moved out of theory and into physical reality, Ganesh held up a complete, fully manufactured 9-petaflop training tile directly to the AI Day audience. He shared that Tesla's hardware team had powered up their first fully functional training tile just the prior week in a benchtop laboratory setup with temporary cooling. To demonstrate early functionality and validate software readiness, the team immediately loaded and executed a neural network created by Andrej Karpathy:

> “And I was told Andre doesn't believe that we could run networks till we could run one of his creations: this is minGPT-2 running on Dojo—do you believe it?”

With the functional tile proven, Tesla scales Dojo into datacenter clusters by chaining modules together without external switches or protocol converters. Six training tiles are joined in a 2x3 matrix to form a training tray. Two trays are packed into a single Dojo cabinet, delivering 100 petaflops of compute. Rather than isolating systems behind chassis and networking switches, the internal interconnect fabric extends straight through the cabinet walls, preserving unified cross-tile bandwidth with zero cliffs or throughput drops.

Ten interconnected cabinets form the complete Dojo ExaPOD. The ExaPOD unites over one million individual training nodes into an unbroken compute fabric producing 1.1 exaflops of BF16/CFP8 compute, creating a dedicated training platform engineered entirely around the scale of Tesla's autonomous vision models.

[02:00:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7228s) Dojo Software Stack, DPUs, and the Compiler Engine

The Dojo hardware architecture links one million compute nodes across an enormous fabric with completely uniform bandwidth, but hardware alone is useless without an equally capable software stack to drive it. Recognizing that neural network workloads vary drastically in scale, the engineering team designed the compute plane to be subdivided and partitioned right from the start. Compute can be partitioned into modular units termed Dojo Processing Units (DPUs). Each DPU consists of one or more D1 chips, interface processors, and one or more host systems, allowing compute resources to scale dynamically up or down based on the exact requirements of any given model or algorithm. Crucially, ML researchers do not need to overhaul their codebase to run on this architecture; they modify their scripts minimally because the compiler suite automatically orchestrates fine-grained parallelism and maps neural network topologies onto the compute plane.

```mermaid flowchart TD accTitle: Dojo Software and Compiler Stack accDescr: Software layers from user-level PyTorch code down to binary generation for D1 hardware. subgraph UserPlane["User Plane"] A["PyTorch Extension API"] end subgraph CompilerSuite["Dojo Compiler Engine"] B["Parallelism Extraction (Data, Model, Graph Parallelism)"] C["Memory Footprint Optimization & Dynamic Control Flow"] D["LLVM Code Generation Backend"] end subgraph ExecutionPlane["Execution & Hardware Control"] E["Driver Stack (Multi-Host / Multi-Partitioning)"] F["Dojo Processing Units (D1 Chips & Interface Processors)"] G["Profilers & Debuggers"] end

A --> B B --> C C --> D D --> E E --> F F -.-> G ```

The Dojo compiler engine relies on multiple advanced techniques to extract data, model, and graph parallelism while applying memory footprint optimizations. Previously, high-bandwidth interconnect limitations restricted model parallelism strictly to individual chip boundaries; with Dojo’s ultra-high-bandwidth fabric, model parallelism can effortlessly span across entire training tiles and beyond. This allows large networks to execute with high utilization even at low batch sizes, opening up completely new performance regimes while simultaneously supporting dynamic control flows like loops and conditional branches. The software suite integrates an extension to PyTorch so practitioners retain familiar user interfaces. Beneath this layer, the compiler generates code on the fly for caching and reuse, routing through an LLVM backend that emits bare-metal binaries for D1 chips. This end-to-end vertical integration produces ultra-optimized execution without needing a single line of handwritten hardware kernels, complete with multi-host driver partitioning, profilers, and debuggers.

- **Dojo Processing Unit (DPU):** Modular slice composed of one or more D1 chips, interface processors, and host machines, scalable up or down per job. - **Compiler Features:** Extracts fine-grained data, model, and graph parallelism; manages dynamic control flow (`if-then-else`, loops); optimizes memory footprint. - **Model Parallelism Expansion:** Extends model parallelism across entire multi-chip training tiles rather than remaining bottlenecked within single-chip boundaries. - **Software Integration:** Standard PyTorch user interface, dynamic JIT binary compilation via an LLVM backend, zero handwritten kernels required.

---

[02:03:51](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7431s) Dojo System Performance and the Next-Generation Roadmap

By vertically integrating every single layer—from custom silicon and packaging to power delivery, cooling, compiler passes, and driver runtime—Tesla eliminates the traditional bottlenecks of distributed compute clusters. The presenter summarizes the culminating impact of Dojo, projecting that it will emerge as the fastest AI training supercomputer in existence while providing unprecedented operational efficiency.

“To sum it all, this is what it will be: it will be the fastest AI training computer, 4x the performance at the same cost, 1.3x better performance per watt that is energy saving, and 5x smaller footprint—this will be Dojo computer.”

The team is already assembling its first complete server cabinets, but they emphasize that the project is merely at its starting line. Architecture work is already underway for a next-generation platform intended to extract another 10x leap in performance across every layer from silicon to software. To pull off that second journey, Tesla closes the compute presentation with an explicit appeal for top talent across chip design, system engineering, and compiler development to join the company.

- **4x Performance:** Same cost profile compared to current state-of-the-art AI training clusters. - **1.3x Better Efficiency:** Higher performance-per-watt yielding substantial energy savings. - **5x Smaller Footprint:** Extreme compute density compared to standard rack server deployments. - **Cabinet Deployment:** Initial production compute cabinets are currently in assembly. - **Next-Gen Dojo Target:** Architecture roadmap targeting an additional 10x performance boost from silicon up through systems.

---

[02:05:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7514s) The Tesla Bot Reveal

Following the Dojo presentation, Elon Musk returns to the stage to pivot beyond Tesla’s automotive fleet into general-purpose robotics. The announcement is introduced with an energetic performance by a dancer dressed in a sleek, black-and-white humanoid robot suit, popping and breakdancing under intense stage lighting to heavy electronic music. Once the music fades and the dancer steps offstage, Musk laughs to the audience: “Now unlike, unlike Dojo, obviously that was not real.” He quickly follows up to establish the serious intent behind the spectacle: “So Dojo is real, the Tesla Bot will be real.”

```mermaid flowchart LR accTitle: Tesla Bot Architecture from Vehicle Stack accDescr: How existing Tesla automotive hardware and software stacks transfer directly to the Tesla Bot humanoid robot. subgraph AutoTech["Tesla Vehicle Foundation"] FSD["Full Self-Driving Computer"] CAM["8 Autopilot Cameras"] NN["World-Model Neural Networks"] ACT["Actuators, Battery, & Power Electronics"] end subgraph HumanoidPlatform["Tesla Bot Platform"] BOT["5'8\" Humanoid Form (125 lbs, Screen Head, 5 mph)"] end FSD --> BOT CAM --> BOT NN --> BOT ACT --> BOT ```

Musk argues that Tesla is already arguably the world's largest robotics company because its vehicles are essentially semi-sentient robots on wheels. The hardware and software built for full self-driving—the inference computer, the multi-camera vision networks, real-time spatial navigation, batteries, and advanced actuators—map directly onto a humanoid form factor. The robot is engineered specifically to navigate spaces constructed for human beings and to eliminate dangerous, repetitive, or uninspiring tasks. With light humor, Musk reassures the audience about physical safety constraints designed directly into the robot's mechanical specifications.

“We're setting it such that it is, at a mechanical level, at a physical level, you can run away from it and most likely overpower it.”

- **Target Prototype:** Tesla aims to build a functional prototype within the following year. - **Height & Weight:** Approximately 5 feet 8 inches tall, weighing roughly 125 pounds. - **Top Movement Speed:** Mechanically capped at around 5 miles per hour. - **Head Interface:** Integrated display screen to provide visual status and useful information. - **Sensing & Compute:** Equipped with 8 autopilot cameras and powered internally by the Tesla Full Self-Driving (FSD) inference computer.

---

[02:09:00](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7740s) Humanoid General Intelligence and the Labor Economy

Musk moves into the deeper philosophical and technical hurdles of general-purpose humanoid robots, noting that true utility requires understanding natural language commands without line-by-line programmatic instruction. A capable assistant must navigate unstructured real-world environments dynamically. The objective is to enable an owner to issue conversational requests—such as telling the machine to pick up a specific bolt and attach it to a car using a wrench, or instructing it to travel to the grocery store and return with a specified grocery list.

This level of autonomous physical capability carries profound societal and economic ramifications. Musk reflects on the foundational nature of human economics, noting that capital goods and equipment are ultimately just distilled labor. If physical labor becomes infinite through generalized robotic automation, the traditional limitations of economic output cease to apply.

“I think essentially in the future physical work will be a choice—if you want to do it you can, but you won't need to do it.”

Because unconstrained automated labor could dramatically disrupt employment, Musk highlights that the long-term societal equilibrium will inevitably require policy interventions like universal basic income, though he jokes: “Yeah, but not right now because this robot doesn't work, so we just need a minute.” He closes the monologue with an open invitation for engineers to join Tesla's robotics team before inviting the evening’s engineering presenters back onto the stage for an open audience Q&A session.

---

[02:11:13](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7873s) Q&A: Open Sourcing and Licensing Autonomous Tech

With the full engineering panel reassembled on stage, Musk asks the technicians to dim the intense stage lights, joking that they feel like interrogation spotlights that leave the panel unable to see into the crowd. The floor opens to attendees for technical questions across Tesla's hardware and software divisions.

An audience member steps up to commend the presentations and asks a high-level strategic question: to what extent is Tesla interested in open-sourcing or publishing its proprietary software, models, and architectures in the future?

Musk addresses the economic reality of the massive capital and engineering investments required to invent systems like FSD and Dojo from scratch, pointing out that open-sourcing the entire core IP would leave the company with no mechanism to finance the multi-billion-dollar R&D efforts unless contributors worked for free. However, Musk draws a clear distinction between proprietary development and industry accessibility, revealing Tesla's openness to commercial licensing.

“But I should say that this is—if other car companies want to license it and use it in their cars, that would be cool; this is not intended to be just limited to Tesla cars.”

---

[02:14:05](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8045s) Q&A: Solving Distributed Compiler Scaling on Dojo

A deep learning accelerator researcher in the audience asks about the Dojo supercomputer's software stack, specifically whether Tesla has truly solved the compiler challenge of scaling and distributing models across thousands of custom nodes, or whether that compiler breakthrough is narrowly restricted to the Dojo hardware architecture. The researcher emphasizes that, from an academic and industry research perspective, distributing workloads efficiently even across a single chip is notoriously difficult.

Musk passes the microphone to Bill Chang, who provides a direct and candid assessment of their compiler roadmap:

“Have we solved the problem? Not yet. Are we confident we will solve the problem? Yes.”

Chang explains that Tesla has successfully validated neural network execution on physical prototype hardware and built accurate performance models demonstrating scalable distribution across larger node fabrics. The central engineering challenge lies in preserving locality: if the compiler can apply balanced mixtures of model parallelism and data parallelism, memory access remains confined within local compute domains. As long as parameter working sets fit within the local on-chip SRAM, activations flow cleanly through the processing pipeline. Chang notes that as node scales expand and future revisions incorporate greater local memory, bandwidth-memory trade-offs will expand their design space, but Tesla's compiler team already sees a clear, solved path for its autonomous driving and vision applications.

[02:15:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8128s) Dojo Timeline and the Acid Test for Success

Addressing an audience inquiry regarding Dojo's modularity, training memory locality, and potential open sourcing, Elon Musk states that the true validation will be operational deployment. Tesla plans to have Dojo operational the following year. Its primary initial role will be training models on vast amounts of video data as efficiently as possible. Musk emphasizes that innovation speed hinges directly on iteration time: reducing training turnaround from several days down to a couple of hours accelerates the overall rate of improvement.

Musk lays down a sharp, unambiguous benchmark for whether the Dojo hardware and software stack has met its objective:

> “The acid test here, and what I've told the Dojo team, is like it's successful if the software team wants to turn off the GPU cluster, but if they want to keep the GPU cluster on, it's not successful.”

```mermaid flowchart LR accTitle: Dojo Success Metric accDescr: Decision flow showing whether Dojo replaces or fails to replace the GPU cluster. subgraph Benchmark ["Acid Test"] A["Dojo Deployment"] --> B{"Software Team Choice"} B -->|"Wants to turn off GPUs"| C["Dojo is Successful"] B -->|"Keeps GPUs running"| D["Dojo is Not Successful"] end ```

[02:17:38](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8258s) Universal Simulation and the Tesla Bot "Optimus"

An audience member asks whether Tesla’s hyper-realistic simulation platform will be leveraged elsewhere across the company. Ian Glow, head of the Autopilot simulation team, explains that achieving full self-driving necessitates expanding simulation beyond pure vehicle dynamics to incorporate every vehicle subsystem, including the battery management system (BMS) and the media control unit (MCU). This deep, holistic integration makes the simulation infrastructure exceptionally valuable across Tesla.

Glow shares that Tesla aims to evolve the tooling into a universal simulation platform. Before broadening it to general use, however, the simulation team will stand up extensive support for the newly announced Tesla Bot. During this explanation, Glow inadvertently slips the internal code name for the robot:

> “Optimus is the code name for the Tesla Bot.”

Musk jokingly quips, “Optimus Prime,” drawing laughter from the crowd. Glow also hints that, further down the line, Tesla has rough ideas for getting its simulation infrastructure into the hands of developers outside the company.

[02:18:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8336s) Multimodal Inputs and Vision Dominance in Autonomous Driving

Audience member Ali Jahanian notes that modern cars are already mobile robots and asks about incorporating multimodal inputs—such as natural language and symbolic reasoning—alongside vision, referencing Andrej Karpathy's open-source miniGPT work.

Musk clarifies Tesla’s core philosophical stance on full self-driving. Driving, he points out, was designed for biological eyes and brains, which means real-world autonomous navigation is predominantly a visual problem solved by silicon neural networks:

> “Driving is fundamentally basically almost entirely vision neural nets; like we're basically it's running on a biological vision neural net and what we're doing here is a silicon camera neural net.”

Musk notes that non-visual modalities still play targeted roles: - **Audio perception:** Detecting emergency vehicle sirens. - **Natural language / interior audio:** Conversing with vehicle occupants. - **External audio understanding:** Interpreting outside speech if a pedestrian or traffic officer is yelling instructions at the vehicle.

[02:21:11](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8471s) Scaling FSD Across International Driving Environments

An attendee asks how Tesla’s autonomous driving stack—which seems primarily trained on United States roadway data—can scale across different countries without having to build everything from scratch. Musk reveals that Tesla actually trains models on video collected from roughly 50 different countries. However, Tesla deliberately focuses initial rollouts on the United States to avoid introducing unnecessary domain complexity before the core problem is solved. When asked why Tesla does not immediately roll out to Canada, Musk notes that Canadian roads are different enough to introduce distinct edge cases.

Andrej Karpathy reinforces that the underlying vision system is fundamentally country-agnostic. Features such as curbs, pavement, drivable free space, and dynamic vehicles behave identically worldwide. Musk adds that the core objective supersedes geographical borders:

> “And there's the the prime directive is don't crash, right, and that's true for every country.”

Musk elaborates on this survival principle: - Even if an unknown object appears—such as an extraterrestrial craft that crash-landed onto the highway—the vehicle must avoid collision. - “You should not need to recognize it in order to not hit it.”

[02:23:20](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8600s) Photometric Multi-View Geometry Accuracy

An attendee asks about the margin of error between synthetically reconstructed 3D geometry and ground-truth geometry in Tesla's auto-labeling pipeline, asking whether it falls within millimeters or centimeters. A member of the Autopilot vision team confirms: - **Geometric reconstruction accuracy:** The standard deviation between synthetic and physical geometry is typically within 3 to 4 centimeters. - **Sensor reliance:** While auxiliary sensors are occasionally mounted on calibration vehicles to establish benchmark baselines, Tesla’s scalable production labeling pipeline relies almost exclusively on passive camera vision.

Musk elaborates that vehicle localization accuracy targets the centimeter level. Because spatial resolution decreases with distance, closer objects are reconstructed with significantly greater precision—matching human driving mechanics, where far-off objects demand less immediate spatial precision until the vehicle approaches.

[02:24:40](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8680s) AI, Manufacturing Automation, and the "Alien Dreadnought"

An attendee asks about the intersection of AI and manufacturing, referencing Musk's past "Alien Dreadnought" concept (an entirely automated factory) and whether the newly announced humanoid Tesla Bot was born out of the "production hell" realization that human labor remains essential.

Musk cautions against taking past extreme statements too literally. Tesla's current factories feature an enormous spread of automation: - **Automation distribution:** Certain sections of the manufacturing line are nearly 100% automated, while other intricate assembly stations remain largely manual. - **Structural castings:** Major manufacturing efficiencies come from mechanical redesign rather than adding robotics. Utilizing massive high-pressure aluminum die-cast machines enables Tesla to cast the entire rear third of the vehicle as a single structural piece, with the front third following. - **Body shop reduction:** By replacing hundreds of stamped, welded metal parts with massive single-piece gigacastings, the total footprint of the body production line drops by 60% to 70%.

Musk concludes that the development of the humanoid Tesla Bot was not prompted specifically by automotive factory floor automation, but rather as a general platform built upon Tesla's advanced real-world AI stack.

[02:26:08](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8768s) Why Tesla Must Build the Humanoid Robot

Elon Musk reflects on the company's trajectory and why developing the Tesla Bot is an almost inevitable step. Rather than setting out with an abstract desire to construct androids, Tesla found itself organically developing all the requisite core technologies—advanced vision, embedded inference computers, spatial navigation, battery packs, and actuators. Because those foundational elements are already being engineered at scale for vehicles, building a humanoid robot became the logical next milestone. Musk stresses that if Tesla does not step up to lead this transition responsibly, another entity inevitably will, making it crucial that Tesla steers the technology to guarantee safety.

Beyond the software and intelligence components, Musk highlights that the single most underappreciated hurdle in robotics is volume manufacturing. While crafting a handful of bespoke research prototypes in a laboratory is manageable, producing capable humanoid robots at a viable price point requires world-class manufacturing prowess. As Elon Musk pointed out, “I should say like also manufacturing volume manufacturing is extremely difficult and underrated and we've gotten pretty good at that.” Bringing down the unit cost so that humanoid robots are not prohibitively expensive remains just as vital as solving the underlying intelligence.

- Tesla's existing engineering pipeline already develops the vision systems, inference compute, and mechanical actuators needed for general robotics. - Safety considerations motivate Tesla to enter humanoid robotics proactively rather than leaving development entirely to outside players. - Scaled, cost-effective volume manufacturing is treated as the primary barrier preventing humanoid robotics from becoming widespread.

---

[02:26:49](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8809s) Scaling Dojo: Power Delivery, Thermals, and $I^2R$ Losses

An audience member asks about the physical constraints of scaling the Dojo supercomputer across cluster racks and multiple data centers, specifically pressing on the immense challenges of thermal dissipation and power delivery. Bill, a Dojo hardware engineer, steps forward to describe the modular architecture engineered to conquer these density bottlenecks. The fundamental building block is the Dojo compute tile, which incorporates integrated dual-sided liquid cooling directly against the top and bottom surfaces. Each tile operates as a self-contained unit under a single cubic foot in volume, designed so that technicians can simply snap it into standard power and liquid coolant lines to achieve instant, fully cooled operation without complex external ducting.

```mermaid flowchart LR accTitle: Dojo Compute Tile Packaging Flow accDescr: Diagram showing how vehicle powertrain technologies feed into Dojo modular cooling and power delivery. subgraph Automotive["Automotive Heritage"] direction TB Powertrain["Powertrain Power Electronics"] CoolingTech["Vehicle Liquid Cooling Systems"] end subgraph DojoTile["Dojo Compute Tile"] direction TB TilePackaging["Compute Tile (under 1 cu. ft.)"] DualCooling["Dual-Sided Liquid Cooling"] LowVolt["Sub-1V High-Amperage Delivery"] end Automotive --> DojoTile DojoTile --> SystemRack["Modular Click-In Rack Integration"] ```

Musk elaborates on how Tesla leveraged its deep institutional knowledge from the automotive sector to make Dojo possible. The team repurposed the power electronics and high-performance cooling architectures originally developed for electric vehicle powertrains and adapted them directly into supercomputer packaging. The primary physics challenge stems from the fact that Dojo chips operate at an extremely low voltage of less than one volt. Pushing tens of thousands of watts at sub-one-volt levels demands colossal electrical current, causing resistive heating losses to skyrocket. As Elon Musk colorfully summarized, “$I^2R$ is what really bites you on the ass.”

- Each compute tile is liquid-cooled on both the top and bottom, packaging compute and power delivery into less than a cubic foot. - System integration is modular: tiles "click" directly into shared cooling and power distribution rails. - Operating Dojo at sub-one-volt rails produces massive amperage, making resistive thermal dissipation ($I^2R$) the core limiting design factor.

---

[02:28:58](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8938s) Dojo’s Purpose: ASICs vs. GPU Emulation and Broader Training Workloads

A questioner asks about the downstream effects of Dojo’s high-throughput hardware: will faster training merely induce engineers to train over massive datasets or build bloated models that become too computationally expensive to run at inference time inside the cars? A Tesla team engineer clarifies that Dojo is not intended solely to train onboard driving networks. Andrej Karpathy’s deep learning pipelines encompass massive server-side models—including heavy auto-labeling networks, synthetic data generators, and foundational multi-modal representations—that operate entirely in the data center to build the datasets that ultimately feed the vehicle stack. Dojo will accelerate all of these upstream training loops.

Musk contextualizes Dojo's broader ambition, stating that while its inaugural role is training neural networks on multi-camera automotive video for the FSD computer, it is fundamentally designed as a general-purpose neural net supercomputer. Musk critiques traditional compute clusters, arguing that CPUs and GPUs were never originally architected for machine learning workloads. Because general-purpose GPUs run neural network mathematics in an inefficient, emulated style, Tesla committed to building Dojo as a dedicated Application-Specific Integrated Circuit (ASIC) from the silicon level up. As Elon Musk asserted, “And just generally any system that is designed for a specific purpose will be better than one that is designed for a general purpose.”

- Dojo's immediate benchmark is outperforming existing high-end GPU clusters at consuming raw multi-camera video for Autopilot training. - Workloads on Dojo extend beyond onboard networks to large-scale data-center auto-labeling and offline curation pipelines. - Modern GPUs execute neural network training effectively in emulation mode, whereas Dojo treats deep learning math as a ground-up native ASIC task.

---

[02:31:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9074s) Unifying Vision and Planning into End-to-End Neural Networks

An attendee asks whether Dojo will enable training networks that cross the architectural boundary separating perception and planner/control, and whether Tesla's onboard FSD hardware can run unified networks within strict real-time latency limits. Ashok Elluswamy answers that training planner networks on Dojo or GPUs is platform-invariant, but eliminating the artificial handoffs between perception and planning actually reduces onboard compute overhead. When vision models are forced to serialize and decode high-dimensional scene features into intermediate explicit states, latency and compute cycles are wasted. An end-to-end network circumvents that overhead by only decoding the specific vehicle trajectories and actions required for driving.

Andrej Karpathy reinforces that end-to-end learning represents Tesla's long-term design philosophy. While current production software still maintains a discrete boundary where perception outputs vector space representations consumed by a C++ planning engine, the historical trend across Tesla's stack has been deep learning steadily absorbing neighboring modules. Karpathy notes that nothing prevents the team from passing latent vision features directly into downstream planning blocks to fine-tune the entire network jointly. Ashok adds that the engineering team must purposefully structure network layers—such as embedding spatial RNNs into perception and integrating search and optimization directly into planning architectures—so that neural networks execute trajectory planning with the speed and determinism of classical C++ algorithms.

As Ashok Elluswamy explained: > “If anything once we make this entire thing end to end it'll be more efficient than decoding a lot of these intermediate states so you should be able to run faster if you make the entire thing end to end neural networks.”

Andrej Karpathy agreed with the overall architectural trajectory: > “Certainly end-to-endness is the guiding principle behind a lot of the network developments and over time in the stack neural networks have taken on more and more functionality and so we want everything to be trained end-to-end because we see that that works best but we are building it incrementally.”

---

[02:33:00](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9180s) Hybrid Architectures, Physics-Based Planning, and the Video Game Metaphor

Following up on planning architectures, another audience member asks whether specialized hardware should be designed for convex optimization and Model Predictive Control (MPC) routines. Ashok explains that Tesla is actively creating a hybrid software architecture. Instead of abandoning mathematical control theory, the team replaces specific bottlenecks in MPC algorithms with learned neural network modules, while leaving established physical dynamics intact. Known kinematic laws and vehicle physics are directly incorporated into the forward pass of the neural network itself. By combining explicit physics models for well-understood dynamics with neural network layers that learn complex environmental interactions, Tesla creates an end-to-end optimizable system that does not discard classical mechanics.

Musk offers an intuitive visual metaphor to explain how the current production stack operates. Once multi-camera surround video is translated by neural nets into a unified, three-dimensional vector space, the autonomous vehicle is essentially navigating inside a video game simulation. Musk points out that game engines in titles like *Grand Theft Auto* have long routed non-player vehicles and pedestrian avatars through 3D environments without collisions using conventional procedural code. While neural network-based planners will ultimately offer superior performance, Musk emphasizes that superhuman safety does not strictly require end-to-end machine learning for the planner today. As Elon Musk stated, “In fact I'm sure you can get to much safer than human with control and planning primarily in C++ with perception vision in neural nets.”

- Tesla is constructing a hybrid planning stack that couples neural network blocks directly with physics-based kinematics inside the forward pass. - Vector space vision turns the driving problem into a spatial navigation task akin to non-player AI pathfinding in *Grand Theft Auto*. - Classical deterministic C++ planning paired with neural net perception is already capable of achieving safety margins well beyond human drivers.

---

[02:34:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9296s) Machine Learning in Manufacturing: A Last-Resort Tool

A member of the audience asks if Tesla applies machine learning and reinforcement learning internally across its broader business operations, such as optimizing factory power consumption, manufacturing logistics, or vehicle design workflows. Musk's response is immediate and counterintuitive for an AI Day presentation: he actively discourages engineers from reaching for machine learning unless all conventional engineering methods have failed. Because machine learning systems are brittle, labor-intensive, and computationally demanding, applying them to standard operational problems often introduces unnecessary complexity.

Musk explains that whenever an engineering team proposes using machine learning to solve an internal workflow problem, he views it as an immediate warning sign. For the vast majority of factory tasks, deterministic software, standard optimization math, and sound first-principles engineering remain vastly superior. Musk jokes that this calculus might change in the future once intelligent humanoid robots can walk factory floors and parse natural language commands, but for now, traditional methods reign supreme. As Elon Musk bluntly declared, “Unless you basically unless you have to use machine learning don't do it.”

- Musk treats the proposal to apply machine learning to factory or operational workflows as an engineering “red flag.” - In approximately 99.9% of manufacturing and design use cases, classical software and analytical modeling remain superior to deep learning. - Machine learning is treated within Tesla as a tool of absolute necessity rather than a convenient catch-all solution.

[02:36:20](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9380s) Simulation Pipeline and Neural Rendering

An audience member asks about the underlying graphical and neural techniques powering Tesla’s simulator, particularly how neural rendering fits into the pipeline. A Tesla engineer breaks down the simulation stack: at the foundation, it uses traditional game engine techniques such as real-time rasterization—very much like Grand Theft Auto—layered with real-time ray tracing. Beyond rasterization and ray tracing, the team has pushed aggressively into neural rendering, noting that the demo imagery shown earlier had literally come in the previous night. The goal is to use neural rendering as an enhancement layer that prevents downstream perception models from ever overfitting to synthetic artifacts.

Tesla is experimenting with approaches similar to those seen in recent photorealism enhancement literature, but with massive operational advantages: far more compute, far more labeled real-world data, tight control over environments, and specialized engineering to run pipelines in real time. The team aims to reach a stage where full models could theoretically be trained entirely inside the simulator, though real-world data will remain primary. Elon Musk underscores that simulation is fundamentally an edge-case engine: as Tesla vehicles become better at accident avoidance, real-world crash logs become exceedingly rare. The simulator can deliberately recreate pre-crash dynamics—such as an unavoidable collision scenario where the vehicle travels at 20 meters per second with only five meters to spare—to train policies on minimizing injury probabilities for vehicle occupants and pedestrians, or to stress-test cars driving the wrong way down highways.

```mermaid flowchart TD accTitle: Tesla Simulation Technology Stack accDescr: Three-layer simulation stack showing rasterization at the base, ray tracing in the middle, and neural photorealism on top, feeding into policy training.

subgraph SIM_STACK["Simulation Engine Stack"] BASE["Base Layer: Real-Time Rasterization (Traditional game engine / GTA-style)"] RT["Middle Layer: Real-Time Ray Tracing"] NEURAL["Top Layer: Neural Rendering (Photorealism enhancement & artifact removal)"] BASE --> RT --> NEURAL end

subgraph USE_CASES["Target Use Cases"] EDGE["Rare Edge-Case Generation (Wrong-way drivers, odd occlusions)"] CRASH["Crash Dynamics & Injury Minimization (e.g., 20 m/s with 5m to impact)"] end

NEURAL --> EDGE NEURAL --> CRASH ```

- Traditional game engine rasterization forms the baseline rendering layer. - Real-time ray tracing sits atop rasterization, followed by neural photorealism enhancement models. - Simulation supplies critical training data for rare failure modes that autonomous vehicles rarely encounter in production fleets.

---

[02:38:48](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9528s) The Target Role of Tesla Bot (Optimus)

Asked about the primary use cases and grand engineering challenges for the newly announced humanoid platform, Elon Musk reiterates the machine's core operational premise: tackling work that humans find repetitive, dull, or hazardous. Internally referred to as Optimus or Tesla Bot, the robot is scoped around basic utility rather than advanced social roles at launch.

Musk states the core criterion simply: “Optimus, it's basically going to start with just dealing with work that is boring, repetitive, and dangerous—basically what is the work that people would least like to do.”

---

[02:39:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9568s) Sim-to-Real Domain Adaptation and the March of Nines

An attendee questions how Tesla avoids the hazards of deploying networks trained on imperfect synthetic data, asking whether explicit domain adaptation techniques are employed to bridge the sim-to-real gap. The simulation team explains that they generate full kinematic video clips directly from the simulator and train models on them immediately. However, simulated data only constitutes a targeted, small fraction of the total dataset, and all evaluations are performed strictly against real-world road footage. To detect and control domain gaps, the team has even repurposed the in-car neural networks to classify whether an image is simulation or reality, using those signals to guide their 3D art and lighting decisions.

Elon Musk emphasizes that real-world camera feeds from the global fleet remain the backbone of their data pipeline because real environments exhibit unmatched entropy: “Overwhelmingly the data set is the real video from the cars on the actual roads—nothing's weirder or has more corner cases than reality.” When the fleet encounters a rare anomaly on the road, Tesla synthesizes approximately a thousand variants of that quirky scenario in simulation to fill the blind spot. Musk frames this relentless iteration as the “march of nines”—the climb from basic reliability to 99.999999% safety, where each added nine increases engineering difficulty by a full order of magnitude.

- Model performance is evaluated strictly on real-world video benchmarks, never on synthetic benchmarks. - In-car neural architectures are retrained as discriminators (sim vs. real) to diagnose visual rendering discrepancies and guide 3D asset creation. - A single bizarre real-world clip often triggers the procedural generation of roughly 1,000 synthetic permutations to harden the model.

---

[02:41:47](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9707s) Humanoid Applications and Companionship

A follow-up question explores whether the humanoid form factor is chosen for specific mechanical tasks, and whether companionship or emotional interaction appears anywhere on the product roadmap. Musk quips about avoiding dystopian science fiction tropes before returning to first principles: human environments—from stairs and doorways to tools and factory floors—were built exclusively for humanoid bipedal bodies with hands.

Musk clarifies that Tesla is taking a strictly literal, functional path first: “We're trying to be as literal as possible: can it do boring, dangerous, repetitive jobs that people don't want to do? And once you can have it do that, then maybe you can do other things too.” He jokes that if an owner wants to treat the robot as a buddy or friend, people will inevitably find creative uses for it, but functional labor automation remains the active engineering roadmap.

---

[02:43:09](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9789s) Model Scaling, Auto-Labeling Distillation, and Predictive Intuition

An audience member asks whether Tesla observes the dramatic performance gains from model scaling that have transformed large language models. The autonomy team confirms that larger models consistently yield superior results provided they are supplied with proportional training data. In the car itself, strict compute and real-time latency budgets limit model capacity, requiring creative optimizations. However, offline auto-labeling clusters face no such real-time constraints. Tesla trains massive, non-deployable models that run offline across multi-trip data to construct high-precision labels, which are then distilled down into the lightweight networks running on vehicle hardware.

Musk and Andrej Karpathy detail the emergent predictive power of these networks. Rather than stalling latency by analyzing single static frames for 200 milliseconds, modern networks distribute computation across a rolling temporal cache of incoming video frames. The resulting temporal perception exhibits an uncanny, superhuman predictive ability: networks accurately reconstruct road curvature hidden around blind corners and infer cyclists or pedestrians who are almost entirely occluded—glimpsed only through bus windows or by the edge of a wheel. Karpathy observes that in the theoretical limit, a neural network with sufficient capacity trained across entire continents could memorize global geometry, functioning as an implicit high-definition map permanently baked directly into the model's weights.

Musk emphasizes the necessity of this architecture: “Without the auto-labeling, I think we would not be able to solve the self-driving problem—it's kind of a funny form of distillation where you're using these very massive models plus the structure of the problem to do this reconstruction and then you distill that into neural networks that you deploy to the car.”

```mermaid flowchart LR accTitle: Offline Auto-Labeling and Distillation Architecture accDescr: Flow diagram showing fleet multi-trip video ingested into massive offline server models, producing auto-labels, and distilled into compact real-time vehicle models.

subgraph OFFLINE["Offline Server Infrastructure"] RAW["Multi-Trip Fleet Video & Kinematics"] BIG["Massive Large-Scale Models (No latency / size constraints)"] LABEL["High-Accuracy 3D Ground Truth Auto-Labels"] RAW --> BIG --> LABEL end

subgraph DEPLOY["In-Vehicle Deployment"] DISTILL["Knowledge Distillation"] SMALL["Compact Multi-Camera Temporal Net (Real-time inference / HW constraints)"] PREDICT["Superhuman Temporal Prediction (Occlusion reasoning & road geometry)"] LABEL --> DISTILL --> SMALL --> PREDICT end ```

- Offline auto-labeling models run without latency limits to generate near-perfect ground truth labels. - Massive models and spatial reconstructions distill into compact networks deployed onto Tesla’s custom in-car silicon. - Temporal networks reason over partial cues (such as a wheel visible beneath a vehicle or a reflection in a window) to predict unseen obstacles around blind turns.

[02:46:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9974s) Tesla Bot Hands and Humanoid Form Factor

An audience member asks about the design constraints of the humanoid robot, wondering how essential it is to keep the human form and whether building five-fingered hands within tight weight limits might force the team into problematic cable-driven setups. Elon Musk responds that this is simply the initial prototype: “I mean, this is just going to be Bot version one, and we'll see.”

Musk clarifies that the robot must be generalized to perform the everyday tasks humans handle in a human-built world. While they could theoretically give it an alternate configuration like two fingers and a thumb, the team plans to stick with five fingers for the first version to evaluate real-world performance. He notes that the hands do not require massive grip strength, but they must be functional enough to carry bags and manipulate everyday tools.

[02:47:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10035s) Visual UX and Public Trust in Autopilot

An attendee relays a concern from an old professor whose primary complaint was that the Autopilot user interface failed to inspire confidence, pointing out jittery visualizations such as flickering object classifications and cars that appear to spin on screen. The questioner asks how Tesla intends to convince owners and the general public that the system is safe and reliable. Musk dismisses the visual glitches as a thing of the past: “A while back cars used to spin; they don't—they don't spin anymore, not in the if you've seen the FSD Beta videos, they—they're pretty solid.”

Andrej Karpathy reinforces Musk's observation, explaining that as Tesla trains its multi-camera networks with vast streams of fleet data, stability is improving rapidly. These spatial multi-camera models are only a few months old, and Karpathy emphasizes that the current UI artifacts are just an intermediate stage on an upward trajectory toward superhuman capability. Musk adds that the trajectory is undeniable: “I mean, it's clearly headed to way better than human, without question.”

[02:48:33](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10113s) Short-to-Medium Economics of the Humanoid Robot

A questioner probes the economic viability of the Tesla Bot. While acknowledging the long-term potential of automating physical labor, the audience member points out that repetitive, boring, and dangerous tasks are typically low-wage roles. The attendee asks how Tesla can replicate its automotive playbook—starting with an expensive, high-margin product like the Roadster before moving downmarket—for a humanoid robot. Musk pauses briefly and sidesteps the financial specifics with a humorous brush-off: “Well, I guess you'll just have to see.”

[02:49:16](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10156s) Occupant vs. Pedestrian Safety and Compute Reaction Speeds

An attendee asks how Tesla's onboard AI resolves ethical dilemmas, specifically prioritizing occupant safety versus pedestrian safety. Musk pushes back on the moral dilemma framing, characterizing it as largely a theoretical construct rather than a practical engineering issue: “The—the thing to appreciate is that from the computer's standpoint, everything is moving slowly.”

Musk argues that high-speed computer perception processes the physical environment so quickly that extreme zero-sum trade-offs are exceptionally rare. To illustrate the vast speed mismatch between biological and machine processing, Musk offers an analogy:

- Rendering an advanced video game like *Cyberpunk* with full ray tracing and neural-net-enhanced graphics at 60 frames per second would take a human roughly a month just to draw a single frame by hand without mistakes. - On the rocketry side at SpaceX, humans are physically incapable of piloting a rocket to orbit because the required attitude control reaction speed is roughly 6 to 7 Hz, which exceeds human biological capability; automated control systems operate far higher, around 30 Hz.

[02:51:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10275s) Hardware 3 Headroom and Hardware 4 on Cybertruck

An audience member asks about compute limits on Hardware 3 (FSD Computer 1), specifically whether larger neural networks are exhausting headroom and at what point Hardware 4 will become necessary. Musk responds that Hardware 3 remains fully capable of delivering unsupervised autonomy: “Well, I'm confident that Hardware 3, or the Full Self-Driving Computer 1, will be able to achieve full self-driving at a safety level much greater than a human, probably, I don't know, at least two or three hundred percent better than a human.”

Musk reveals that Hardware 4 (Full Self-Driving Computer 2) will debut alongside the Cybertruck in roughly a year, bringing an estimated 4x increase in compute performance. He frames the generational jump not as an absolute requirement to make autonomy work, but as a path to widen the safety margin from 300% safer than a human driver to 1000% safer.

```mermaid flowchart LR accTitle: Tesla FSD Hardware Generations accDescr: Comparison between Hardware 3 capabilities and upcoming Hardware 4 target capabilities. subgraph HW3["Hardware 3 (FSD Computer 1)"] A["Existing Fleet Baseline"] --> B["200%–300% Safer Than Human"] end subgraph HW4["Hardware 4 (FSD Computer 2)"] C["Debuts on Cybertruck (~1 Year)"] --> D["~4x Greater Compute"] D --> E["~1000% Safer Than Human"] end ```

[02:52:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10359s) Depth Sensors and Adversarial Camera Attacks

An attendee asks whether the absence of dedicated active depth sensors (such as radar or lidar) leaves the pure-vision neural network vulnerable to adversarial physical exploits, such as someone holding up printed photographs or painting a fake scene. Musk jokes about cartoon tricks: “Yeah, like what, pull some like Wile E. Coyote stuff? You know, like paint the tunnel on the—on the wall? It's like, oops.”

Musk notes that Tesla has not seen serious adversarial attacks in the wild, though he shares an amusing real-world edge case from his own testing: wearing a t-shirt printed with a stop sign and flashing it at the car will cause the vehicle to stop. He explains that when such adversarial quirks surface, the engineering team simply trains the perception networks to recognize context—learning that an image of a stop sign printed on a person's shirt is not an actual roadside regulatory sign.

[02:53:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10434s) Handling Uncertainty, Multi-Modal Planning, and Shadow Mode

Asked how the planning system handles uncertainty and whether it assumes surrounding drivers will always behave predictably, the autonomy team and Musk outline a multi-layered defensive strategy:

- The planner models multi-modal futures rather than betting on a single trajectory, calculating surrounding kinematic envelopes to confirm other vehicles have room to yield before the car commits to a move. - When sensory confidence drops or flicker is detected, the car actively slows down to gather more information; as Musk summarizes: “Aspirationally, the car should be—the less it knows, the slower it goes... speed proportionate to confidence.” - Before any new model is deployed to actively drive the fleet, it runs quietly in "shadow mode" across customer vehicles to compare its hypothetical actions against human steering and braking decisions, flagging divergences where the model would have made a different choice.

Musk explains the fleet learning dynamic: “Effectively, the drivers are training the net; so if the neural net would have controlled and, say, veered right, but the person actually went left, it's like, 'Oh, there's a difference, why was there that difference?'”

[02:56:22](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10582s) Identifying the Highest-Impact AI Stack Component

An audience member asks which single layer across Tesla's autonomous stack—simulation, data collection, planning, or control—would yield the greatest performance jump if improved. The team begins answering by describing the system as an interdependent whole, noting that progress reflects “the area under the curve” across all subsystems, while Musk begins to pinpoint the immediate priority of migrating every remaining network over to the unified spatial-temporal architecture.

[02:57:04](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10624s) Legacy Neural Networks and the Engineering Bottleneck

Andrej Karpathy reflects on the state of the Tesla vision architecture, noting that while the team is transitioning the fleet toward processing full surround video, some legacy single-camera or non-temporal neural networks remain active in production. He emphasizes that this is a temporary situation being addressed rapidly, but ideally every network running on the vehicle will operate on surround video. Karpathy observes that all the fundamental puzzle pieces for success are already in place, but executing the transition requires significantly more engineering talent.

Elon Musk chimes in to reinforce Karpathy's point, stating bluntly that engineering talent is the single bottleneck holding back progress: “There’s just a tremendous amount of work to do to make it work so that’s why we need talented people to join in and solve the problem.” Musk reminds the audience that attracting top software and machine learning engineers to solve these challenges is the primary motivation for hosting Tesla AI Day.

[02:57:49](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10669s) Catastrophic Forgetting and Continuous Online Learning

An audience member asks how Tesla deals with the problem of catastrophic forgetting when training neural networks on massive datasets, and whether the company is considering continuous or online learning directly on customer vehicles so individual drivers could have customized driving styles.

Karpathy responds that while he is familiar with the academic literature on catastrophic forgetting, it is not a problem Tesla encounters in practice because they do not perform continuous learning on customer cars. The deployment pipeline requires strict deterministic validation before pushing software over-the-air:

```mermaid flowchart LR accTitle: Tesla Model Training and Deployment Pipeline accDescr: Pipeline showing model training on servers, fine-tuning, extensive evaluation, and deployment to the vehicle fleet with user driving profile toggles. subgraph Offline["Offline Server Fleet"] A["Collect Fleet Data"] --> B["Train Model Once"] B --> C["Fine-Tune Iteratively"] C --> D["Extensive Safety Evaluation"] end subgraph InCar["Customer Vehicle"] D --> E["Deploy Static Weights"] E --> F["Driver Style Selection (Speed / Urgency)"] end ```

The team trains networks offline on server clusters, fine-tunes them a few times, and conducts extensive evaluations to guarantee stability before any weights are loaded onto production vehicles. Musk clarifies that personal customization does not require local neural network re-training. Instead, driving styles can be handled through user-facing settings similar to instructing a human ride-hail driver:

> “Basically the kind of instructions you’d give to your Uber driver: it’s like ‘I’m late for the flight, please hurry,’ or take it easy, or you know whatever your style is.”

[02:59:12](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10752s) AI Alignment and Narrow Versus Superhuman Intelligence

A questioner brings up the issue of AI safety and alignment, asking whether Tesla worries about the risk that increasingly capable models deployed into the physical world might become misaligned with human intentions, or if the team plans to address alignment only as larger problems emerge.

Musk agrees that AI safety is a genuine concern: “I think that we should be worried about AI.” However, he delineates between general intelligence and the focused scope of Tesla's current projects. Tesla is currently engineering narrow AI: teaching a car to drive significantly safer than a human and training a humanoid robot to handle basic, repetitive physical tasks. Musk acknowledges that once systems reach superhuman general intelligence, “all bets are off” regarding predictability. He explains Tesla's core ethos for autonomous development:

> “What we’re trying to do here at Tesla is make useful AI that people love and and is unequivocally good, that’s our—try to aim for that.”

[03:00:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10835s) Camera Sensors, Hardware 4, and the Elevator Analogy

An attendee asks about optical sensors, pointing out that earlier presentations compared the vision system to a synthetic animal, yet standard cameras remain a crude approximation of the human biological eye. The attendee wonders whether Tesla has evaluated alternative designs such as event-based neuromorphic cameras or intends to manufacture custom camera sensors in-house.

Musk reveals that Tesla will introduce next-generation cameras with the rollout of Hardware 4 (HW4), but stresses that Tesla is still far from exhausting the limits of its existing sensor suite:

> “I’m confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware.”

He notes that future hardware revisions will push safety margins from roughly 300% safer than a human to 1,000% safer. Musk draws a historical parallel to the evolution of building elevators, explaining that early elevators relied on fallible human operators who occasionally made fatal errors:

- **Historical elevator operators**: Manually switched large relays to line up elevator cabs with building floors, occasionally slipping up or getting fatigued and shearing passengers in half. - **Modern automatic elevators**: Passengers walk into a 100-story skyscraper, push a button without hesitation, and trust the automated system entirely. - **The future of transportation**: Driving a car manually will eventually be viewed as an archaic, obsolete practice.

Musk concludes the event by stating that while fully autonomous electric vehicles will become the standard, manual and internal combustion cars will still exist in small numbers as niche hobbies, “just like there are still some horses.” He thanks the audience for attending and shuts down the presentation to enthusiastic applause.

## Notable quotes > [02:09:00](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7740s) I think essentially in the future physical work will be a choice—if you want to do it you can, but you won't need to do it. — Elon Musk

> [01:45:40](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6340s) it's easy to scale the compute; it's very difficult to scale up bandwidth and extremely difficult to reduce latencies — Ganesh Venkataramanan

> [01:24:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5075s) So neural networks are just an upper bound; you also need massive datasets to actually train the correct algorithms inside them. — Andrej Karpathy

> [02:33:00](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9180s) In fact I'm sure you can get to much safer than human with control and planning primarily in C++ with perception vision in neural nets. — Elon Musk

> [03:00:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10835s) I’m confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware. — Elon Musk

> [01:27:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5259s) So people are extremely good at things like semantics, but computers are very good at geometry, reconstruction, triangulation, tracking. — Andrej Karpathy

> [48:44](https://www.youtube.com/watch?v=j0z4FweCy4M&t=2924s) What I find kind of fascinating about this is that we are effectively building a synthetic animal from the ground up. — Andrej Karpathy

## Mentioned Ali Jahanian, AlphaZero, Andrej Karpathy, Ashok Elluswamy, Autopilot, BiFPN, Bill Chang, Cyberpunk, Cybertruck, D1 Chip, Dojo ExaPOD, Dojo Interface Processor (DIP), Dojo Processing Unit (DPU), Elon Musk, FSD Beta, Full Self-Driving (FSD), Ganesh Venkataramanan, Grand Theft Auto, Hardware 3, Hardware 4 (HW4), HydraNet, Ian Glow, LLVM, Milan Kovac, MuZero, Optimus, Project Dojo, PyTorch, RegNet, Roadster, Smart Summon, SpaceX, Spatial RNN, Tesla, Tesla Bot, Tesla Vision, Training Tile, Uber, Wile E. Coyote, YOLO, minGPT-2

## Video description Join us to build the future of AI → https://www.tesla.com/ai 0:00 - Pre-event 46:54 - AI Day Begins 48:44 - Tesla Vision 1:13:12 - Planning and Control 1:24:35 - Manual Labeling 1:28:11 - Auto Labeling 1:35:15 - Simulation 1:42:10 - Hardware Integration 1:45:40 - Dojo 2:05:14 - Tesla Bot 2:12:59 - Q&A

[Source](https://www.youtube.com/watch?v=j0z4FweCy4M)

中文简报

特斯拉发布自研D1芯片与人形机器人以加速推进自主性特斯拉正利用其汽车视觉网络和自动标注流水线打造专用超级计算机及人形机器人，旨在让体力劳动成为可选。特斯拉工程领导层深入解析了公司垂直整合的人工智能技术栈，将车辆视为具备具身智能的人造动物。主要演示详述了特斯拉视觉系统（Tesla Vision）如何借助空间循环神经网络（RNN）和交叉注意力机制，从单摄像头2D空间演进为多摄像头3D矢量空间，从而去掉了毫米波雷达，并介绍了分层多智能体轨迹规划。团队还展示了其庞大的自动标注流水线和逼真的仿真引擎，随后公布了由自研D1芯片和模块化训练单元构架支持的Dojo超级计算机项目。最后，埃隆·马斯克揭晓了人形机器人Tesla Bot（Optimus），目标是实现重复、危险和枯燥体力劳动的自动化。特斯拉发布了自研D1芯片及Project Dojo架构，将25个D1裸片封装为一个9 PFLOPS算力的训练单元，并可扩展至算力达1.1 EFLOPS的ExaPOD超级计算机，以加速神经网络训练。 (111:39) 特斯拉视觉系统已全面转向直接运行于3D矢量空间的多摄像头环视视频网络，集成了空间RNN和特征缓存，从而在量产车中剔除了毫米波雷达。 (69:35) 特斯拉发布了Tesla Bot（Optimus），这是一款高5英尺8英寸、重125磅的人形机器人，旨在通过复用汽车的Autopilot硬件、视觉模型和计算能力，实现重复、危险体力劳动的自动化。 (125:14) 埃隆·马斯克宣布Hardware 4（FSD计算机2代）将在大约一年后随Cybertruck首次亮相，提供约4倍的算力提升，目标是将安全性相比人类驾驶员提高1000%。 (171:15)

Reading summary

Tesla argues that its autonomous driving stack and custom supercomputing represent a unified real-world artificial intelligence ecosystem, combining fleet video, temporal networks, and bespoke silicon to automate physical navigation. Elon Musk framed the presentation around recruiting elite engineering talent to advance both autonomous transport and general-purpose humanoid robotics. Spatial-Temporal Vision in Direct Vector Space (58:10) Tesla transitioned Autopilot from isolated per-image 2D predictions to a multi-camera temporal architecture outputting directly into 3D vector space. Operating on eight camera streams, the vision stack uses RegNet backbones, bidirectional feature pyramid networks (BiFPN), and transformer cross-attention to resolve boundary stitching issues between cameras. To track motion and occlusions, Tesla implemented a Spatial Recurrent Neural Network (RNN) that updates a 2D hidden ground-plane lattice using ego-kinematics. Andrej Karpathy demonstrated that this temporal memory maintains vehicle positions through complete visual dropouts, enabling Tesla to replace automotive radar with pure vision in roughly three months. Interactive Planning in Non-Convex Action Spaces (73:11) Urban autonomous driving requires solving high-dimensional, intensely non-convex planning dilemmas with multiple discrete maneuvers. Autopilot software director Ashok Elluswamy explained that the system models the interactive behavior of surrounding vehicles via joint multi-agent planning rather than treating other cars as passive obstacles. Once discrete search selects an optimal maneuver, the planner establishes a convex spatial corridor free of obstacles, inside which continuous gradient-based routines optimize trajectory splines parameterized across arc length. This process minimizes collision risks, travel time, and lateral jerk, while Monte Carlo Tree Search guided by neural network value functions resolves complex trajectories orders of magnitude faster than baseline A* algorithms. Fleet Auto-Labeling and Photorealistic Sensor Simulation (87:39) To satisfy the data demands of large networks, Tesla shifted from manual annotation to automated server-side labeling and high-fidelity simulation. By querying customer vehicles for specific edge cases across 200 conditions, Tesla's automated pipeline labeled 10,000 targeted clips in a single week. Offline processing exploits temporal hindsight—analyzing past and future frames simultaneously—to calculate precise kinematics and maintain tracking across occlusions, reconstructing dense point clouds within 3 to 4 centimeters of physical geometry. In parallel, Tesla's simulation engine models sensor noise, lens distortion, and windshield diffraction, generating synthetic corner cases to stress-test software and inform vehicle sensor hardware design. Custom Hardware: FSD Silicon and Dojo Supercomputing (107:10) Operationalizing these models requires end-to-end hardware-software co-design. In customer vehicles, a proprietary AI compiler schedules PyTorch operations onto dual custom FSD chips within tight SRAM and latency budgets. For model training, Tesla designed Dojo, a supercomputer built around the custom D1 processor. Dojo tiles combine 25 D1 dies on a fan-out wafer to deliver 9 petaflops of compute per unit with dual-sided liquid cooling. Systems engineer Ganesh Venkataramanan projected Dojo will achieve 4x training performance at equal cost and a 5x smaller footprint than contemporary GPU clusters, though compiler engineer Bill Chang acknowledged that distributed scaling across node fabrics remains work in progress. Extending Physical AI to Humanoid Robotics (129:00) Musk introduced the Tesla Bot (codenamed Optimus), asserting that humanoid robotics is a natural extension of Tesla's investments in vision, inference chips, battery architecture, and actuators. Intended for repetitive, boring, or hazardous labor, Musk suggested generalized robotic automation could eliminate labor scarcity and reshape the economy, ultimately requiring universal basic income. However, Musk conceded that volume manufacturing remains the most underappreciated challenge in robotics and declined to give financial projections. Company engineers emphasized that engineering talent, rather than conceptual limits, remains the operational bottleneck across both vehicle autonomy and robotics. Tesla demonstrated major algorithmic progress in temporal vision, joint planning, and specialized silicon, but several core initiatives remain unverified at scale. Dojo's distributed compiler remains in active development, achieving full self-driving requires conquering the long tail of edge cases through an incremental 'march of nines,' and the Optimus humanoid robot remains an early-stage concept that faces severe manufacturing and economic hurdles before practical deployment.

精炼阅读摘要

Tesla在2021年AI Day展示了其从纯视觉感知、端到端轨迹规划到Dojo定制超级计算硬件的垂直整合技术体系，Elon Musk借此强调公司正转型为现实世界人工智能领域的领导者，并宣布进军通用人形机器人领域。从单帧多任务到多相机时空鸟瞰感知 (58:10) 早期Autopilot主要依赖单帧相机输入与RegNet骨干网及BiFPN进行多任务特征提取（HydraNet），团队通过将主干特征缓存到磁盘来降低微调成本，但单帧图像割裂了物理时空连续性，在目标穿过相邻相机边界时容易产生检测抖动与几何畸变。为了解决这一局限，Tesla转向将8个相机的原始画面输入共享主干网，利用Transformer交叉注意力机制直接在鸟瞰向量空间（BEV）生成特征，免去了依赖不平整路面几何投影的不稳定性。为彻底解决短暂遮挡并平滑运动学估算，Tesla引入了空间循环神经网络（Spatial RNN）构成的时空融合模块。系统将网络隐状态投射到二维地面网格上，结合车辆自身运动学状态仅对具有清晰能见度的区域执行选择性读写更新，从而在隐状态中自主形成道路结构与障碍物的特征级高精地图。这套时序视频架构使得网络在目标完全被遮挡时依然能保留状态记忆，其估算出的距离与速度平滑度达到甚至超越了传统毫米波雷达的数据表现，成为Tesla量产车在约三个月内全面移除物理雷达的关键技术支撑。高维非凸交互式轨迹规划与端到端演化 (73:11) 自动驾驶软件总监Ashok Elluswamy指出，城市道路规划面临10至15秒连续轨迹求解的高维度问题，且存在左右绕行等互相隔离的局部最优解，具有极强的非凸性。为此，Tesla将其他动态交通参与者视为协同演化主体，为周围关键车辆同步运行规划算法，以评估相互博弈的意图。粗粒度离散搜索先构建出避开非凸障碍物的安全走廊，随后利用基于弧长参数化的连续样条优化算法，最小化障碍物贴近度、行驶延误与急动度，生成类似熟练人类驾驶员的平滑梯形横向加速度曲线。规划器同时探索了蒙特卡洛树搜索（MCTS）与预测网络相结合的架构，以类似AlphaZero的方式在全球场景上下文中仅扩展数百个节点即可规避局部陷阱。针对感知与规划的边界，Elluswamy与Andrej Karpathy说明，尽管量产系统仍保留感知输出向量空间再对接C++规划引擎的架构，但长期哲学是走向深度学习端到端整合。直接向规划层传递潜在视觉特征可消除序列化解码高维几何带来的中间计算损耗，团队已通过在无能见度时减速的“置信度定速”策略和在客户车辆上静默运行的影子模式（Shadow Mode）来迭代规划网络。自监督离线自动化标注与超仿真闭环面对数亿参数模型对标注数据的庞大需求，Tesla建立了机器负责几何重建与时序追踪、人类负责高层语义验证的自动化标注管线。离线服务器利用“时间后见之明”（Temporal Hindsight），在拥有完整驾驶片段前后的全局视野下重建密集3D点云，精准标注运动学参数，甚至攻克了传统运动恢复结构（SfM）在缺乏纹理的平整沥青路面上无法匹配的难题。在雷达移除项目中，车队触发机制针对200种工况收集极端边缘案例，自动化标注流水线在一周内标注了1万个视频片段，而在人工标注下该规模需要耗时数月。车端部署受制于计算延迟与SRAM容量，必须通过专有AI编译器将PyTorch模型优化调度为FSD芯片指令；离线集群则无此限制，团队通过训练超大模型生成高精度伪标签，再蒸馏至轻量化车端网络。当真实数据稀缺时，Tesla利用高保真传感器仿真平台合成极端场景，其引擎精细建模了相机运动模糊、镜头畸变、挡风玻璃衍射与大灯配光，仿真不仅为软件提供测试场景，还直接指导车载光学镜头的定制设计。Musk将这种通过合成千种变体以消除真实路况盲区的迭代过程称为攀登99.999999%可靠性的“9的征程”。 Dojo超级计算架构：训练切片与垂直系统集成 (107:10) 为了突破现有超算集群的扩展瓶颈，Tesla从底层物理限制出发设计了Dojo超级计算体系。其核心设计基准是以超过2 GHz时钟周期内电信号传输的极限物理距离界定独立“训练节点”，并在晶圆级先进扇出封装上直接集成25颗已知良好D1芯片，制成无需主板布线、算力达9 PFlops的“训练切片”（Training Tile）。每个切片控制在单立方英尺以内，采用整合式双面液冷散热与垂直供电模块，在亚伏特电压下承载数千安培电流，直接复用了电动车动力总成的功率电子与冷却技术。在系统层面，Dojo通过DIP接口处理器与数据中心主机桥接，并通过编译器套件自动提取数据、模型与图并行，将算力划分为模块化的DPU单元。系统总监Ganesh Venkataramanan声称，相比主流AI训练集群，Dojo在相同成本下可提供4倍算力、1.3倍能效提升并缩小5倍占地空间。然而硬件架构工程师Bill Chang在问答环节承认，跨数千节点的分布式编译扩展问题“尚未解决”，但团队对基于局部性保持与模型并行调度攻克该难题持乐观态度。从车端感知延伸至通用人形机器人Optimus (129:00) Musk正式公布了代号为“Optimus”的Tesla Bot人形机器人计划。他解释称，开发人形机器人并非出于抽象概念，而是因为Tesla在研发全自动驾驶过程中，已经规模化自研了视觉算法、边缘推理芯片、空间导航定位、高能量密度电池组与执行器控制等核心组件，汽车本身即是轮式机器人。Optimus初期的工程定位是接管危险、重复和乏味的体力工作，理解自然语言指令并在非结构化现实环境中行动，例如搬运特定螺栓或执行日常琐事。 Musk从宏观经济学角度提出，所有资本货物本质上都是被提炼的人类劳动，若通用机器人使体力劳动供给趋于无限，经济产出将打破传统限制，届时体力劳动将变成一种个人选择，长期来看社会可能需要全民基本收入（UBI）政策来应对就业重塑。但他同时承认，目前机器人尚不能实际工作，且人形机器人面临被行业严重低估的规模制造门槛；在被问及如何实现机器人从高端到大众市场的商业化路径时，Musk并未给出具体的财务或落地路线图，仅表示“拭目以待”。 Tesla AI Day展示了其通过纯视觉与自研计算硬件构筑的物理世界AI技术闭环，但诸多核心挑战仍悬而未决：Dojo系统的全自动分布式编译器尚未彻底攻克大规模扩展壁垒；完全端到端感知规划模型仍在过渡阶段；被寄予厚望的Optimus人形机器人仍处于初期概念验证期，其工程制造可行性与商业经济模型尚需时间验证。

Detailed notes — zh

## Background 本内容由特斯拉（Tesla）于 2021 年 8 月 20 日在官方 YouTube 频道发布，全长超过 3 小时。官方简介为“加入我们，共同构建人工智能的未来”，整场发布会系统展示了特斯拉在计算机视觉、自动规划、自研训练超算 Dojo 及人形机器人等方向的最新技术进展与工程蓝图。 ## TL;DR 发布会系统性拆解了特斯拉的全栈 AI 技术栈，涵盖基于纯视觉与时空特征融合的 Autopilot 神经网络架构、自动化离线标注与高逼真仿真系统。特斯拉正式揭晓了专为大规模神经网络训练打造的 Dojo 超级计算机与自研 D1 芯片，展示了其极致密度的系统级物理封装与专用编译器。马斯克在后半段压轴发布了代号为 Optimus 的人形机器人（Tesla Bot），并在长达近一小时的硬核 Q&A 中与工程骨干团队深入探讨了端到端控制、Hardware 4 演进、安全边界案例攻坚以及实体经济由劳动力转向完全自动化的终极愿景。 ## Atmosphere & style 整场活动兼具硬核技术研讨会与激进工程狂想的双重氛围，台上多位技术负责人以极高密度的架构图和公式层层剖析底层实现，展现出冷静且严谨的极客气质。马斯克穿插其间，语言风格随性且极富煽动力，既有对物理第一性原理与焦耳热阻抗的生动吐槽，也有对全民基本收入与机器人未来的宏大构想。现场互动坦诚而接地气，工程团队毫不掩饰当下面临的技术难点，让观众能切身感受到硅谷前沿技术攻坚中那种充满荷尔蒙的工程自信与招聘饥渴。 ## Bulletin - [02:00:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7228s) 特斯拉发布专为神经网络训练设计的超级计算机架构 Dojo，其基于自研 D1 芯片与算力瓦片（Training Tile），在同等建设成本下比业界领先方案带来 4 倍 AI 训练性能提升，物理占地空间缩小 5 倍，预计次年正式投产运营。 - [02:05:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7514s) 马斯克宣布内部代号为 Optimus 的首款人形机器人 Tesla Bot，身高约 173 厘米、全身配备 40 个机电执行器关节，直接复用 Autopilot 摄像头与 FSD 车载芯片计算系统，目标在 2022 年推出原型机。 - [02:51:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10275s) 特斯拉规划随 Cybertruck 推出 Hardware 4（HW4）车载芯片与新一代摄像头，算力约达现役 HW3 的 4 倍，旨在将全自动驾驶安全裕量推高至超越人类基准 1000%。 - [02:43:12](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9792s) 自动驾驶感知网络正逐步演进为端到端模型，特斯拉通过云端蒸馏将全地表道路几何特征直接烘焙进网络权重中，以逐步取代传统高精地图。 ## Key takeaways - [02:00:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7228s) Dojo 采用全专用集成电路（ASIC）设计与定制 LLVM 编译器，完全摆脱对手写特定算子内核的依赖，可实现超 100 万计算核心的统一网络互联与线性扩展。 - [02:26:50](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8810s) 特斯拉直接将电动车动力总成的功率电子与双面全液冷散热技术移植到 Dojo 超算中，以解决低工作电压和巨大焦耳热（I²R）损耗。 - [02:15:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8128s) 马斯克确立 Dojo 成功的硬性考核标准：只有当特斯拉软件算法团队主动选择关闭现有 GPU 集群时，Dojo 项目才算真正取得成功。 - [02:05:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7514s) 特斯拉将自身定位为全球最大的机器人公司，通过将汽车领域积累的机器视觉、边缘推理与电池动力总成全面复用至 Optimus 人形机器人中。 - [02:05:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7514s) Tesla Bot 将行走速度上限严格设定为每小时 5 英里，从物理与机械结构层面保障人类能够轻松跑过并制服它，防范失控风险。 - [02:10:04](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7804s) 若人形机器人能够消除重复、危险且枯燥的体力劳动，将打破经济受劳动力总量制约的边界，长期可能倒逼社会推行全民基本收入（UBI）。 - [02:18:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8336s) 自动驾驶系统的最高法则与第一原则是避免发生碰撞，即便前方出现未被分类识别的异常障碍物（如迫降的 UFO），系统也必须无条件刹停避让。 - [02:39:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9568s) 自动驾驶安全性的跃升被称为“9的攻坚战”（march of nines），随着安全性从 99% 向多个 9 逼近，每消除一个数量级的事故概率，工程攻坚难度都会呈指数级倍增。 - [02:36:20](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9380s) 现实中车队越安全导致极端事故数据越稀缺，特斯拉通过游戏引擎、光线追踪及神经渲染构建通用仿真系统，在虚拟世界中重现极端撞击与逆行事故。 - [02:49:16](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10156s) 所谓“电车难题”在工程实践中属于伪命题；计算机以 30Hz 频率感知与推理，在超高反应速度和充裕算力下有足够时间提早规避危险，根本不会陷入两难死局。 - [02:34:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9296s) 特斯拉坚持工程克制原则，严格限制在工业制造流水线盲目滥用机器学习，指出 99.9% 的工程问题采用常规物理法则与机械设计解决更为高效且经济。 - [02:57:04](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10624s) 特斯拉不采用端上持续学习以杜绝灾难性遗忘，所有模型更新均在云端集中训练验证，新模型上线前需通过影子模式在百万级车队中静默推演。 - [02:57:04](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10624s) 当前制约全自动驾驶研发落地的最大瓶颈在于顶尖工程人才短缺，大量遗留单目网络仍需海量人力重构成多摄像头时序环视架构。 - [02:59:12](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10752s) 马斯克以专职电梯员被一键直达现代电梯取代作类比，预言手动驾驶车辆与燃油汽车未来终将退化为类似马术运动的非主流消遣。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=j0z4FweCy4M&t=0s) 开场预热与活动背景在活动正式开始前的预热阶段，现场播放着富有科技感的电子背景音乐，屏幕间歇穿插着引言与音视频片段。其中反复出现的一段经典台词直击技术变革的核心：“我不是你要找的人，你弄错了；我与各地的技术顾问和前沿进展都保持着密切联系，我可以明确告诉你，从来没有人制造出类似的东西。”这段话为整场发布会定下了突破传统、探索未知的基调。经过近四十分钟的等待与音乐铺垫，活动现场响起了掌声，特斯拉人工智能日（Tesla AI Day 2021）正式拉开帷幕。 - 预热阶段穿插对前沿科技与颠覆性创新的引用与呼应 - 现场聚集了大量受邀工程师、研究人员及科技爱好者 ### [46:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=2814s) 埃隆·马斯克致开场辞：特斯拉是一家现实世界人工智能公司特斯拉首席执行官埃隆·马斯克（Elon Musk）登台致歉并正式开场。他略带幽默地解释了活动推迟的原因：“抱歉让大家久等了，感谢各位的到来。刚才遇到了一些技术故障，看来我们真的需要为此研发一些人工智能。”马斯克直截了当地指出了举办本次 AI Day 的核心诉求：特斯拉远不止是一家电动汽车制造公司，更是一家在硬件推断与模型训练端均具备深厚自研实力的现实世界人工智能领军者。马斯克表示，凡是体验过 Full Self-Driving (FSD) Beta 测试版的人，都能直观感受到特斯拉神经网络在学习驾驶过程中的飞速迭代。汽车自动驾驶只是现实世界 AI 的一个具体落地场景，未来这一技术底座还将拓展到更多顺理成章的应用领域。他向全球工程技术人才抛出橄榄枝，正如埃隆·马斯克所言：“我们基本上希望鼓励任何有兴趣在硬件或软件层面解决现实世界人工智能问题的人加入特斯拉，或者考虑加入特斯拉。”随后，他将讲台交给了特斯拉 Autopilot 视觉团队负责人安德烈·卡帕斯（Andrej Karpathy）。 - 特斯拉的核心能力贯穿推断芯片（Inference Hardware）与训练基础设施（Training Hardware） - 活动的核心目标是展示技术栈底层并面向全球顶尖软硬件工程师招聘 ### [48:44](https://www.youtube.com/watch?v=j0z4FweCy4M&t=2924s) 特斯拉视觉与合成生物大脑的构想安德烈·卡帕斯（Andrej Karpathy）登台，开启对 Autopilot 完整技术栈的深度技术剖析。他指出，视觉模块的首要目标是设计一个端到端的神经网络，直接处理车身环绕的 8 个摄像头所采集的原始图像数据，并实时将其转化为自动驾驶核心决策所需的“向量空间”（Vector Space）。在这个三维向量空间中，网络必须高保真地重建出道路边缘、车道线、路沿（Curbs）、交通标志、红绿灯，以及周边所有车辆的三维空间位置、航向角（Orientation）、纵深距离与实时速度。卡帕斯将这一工程比喻为从零构建一个“合成动物”（Synthetic Animal）：车辆本身就像拥有机械躯体、电子电气神经系统的仿生生物，而 Autopilot 就是其大脑，视觉团队负责构建的正是这只合成动物的“合成视觉皮层”（Synthetic Visual Cortex）。生物视觉皮层有着精密的分层布局，光线打在视网膜上后，经过外侧膝状体（LGN）传入大脑枕叶的视觉皮层，沿着 V1、V2、V4、IT 等区域在腹侧流（Ventral Stream）和背侧流（Dorsal Stream）中有序流动。特斯拉在设计车载视觉系统时，同样遵循了这种分层抽取、多通道融合的仿生信息流架构。 ```mermaid flowchart LR accTitle: 特斯拉视觉感知从原始图像到向量空间的处理流 accDescr: 展示8个环视摄像头数据经过合成视觉皮层神经网络处理后输出为3D向量空间的过程 subgraph Input["传感器输入"] CAM["8个环视摄像头 1280x960 12-bit @ 36Hz"] end subgraph Cortex["合成视觉皮层 (Neural Net)"] direction TB REG["RegNet 骨干网络 多尺度特征抽取"] FPN["BiFPN 特征金字塔 跨尺度信息融合"] REG --> FPN end subgraph Output["3D 向量空间 (Vector Space)"] direction TB STATIC["静态拓扑 车道线 / 路沿 / 交通灯"] DYNAMIC["动态目标 车辆位置 / 朝向 / 深度 / 速度"] end CAM --> REG FPN --> STATIC FPN --> DYNAMIC ``` - 输入源：车身分布的 8 个摄像头持续捕获原始视频流 - 核心转换：将 2D 像素阵列在推理过程中实时重构为 3D 向量空间 - 仿生学映射：机械结构对应骨骼肌肉，电气线束对应神经系统，Autopilot 神经网络对应视觉皮层与大脑中枢 ### [51:31](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3091s) 单帧图像时代：RegNet、BiFPN 与 HydraNet 架构演进卡帕斯回顾了大约四年前他刚加入特斯拉时的技术起点。当时系统的主要功能是高速公路单车道保持与前车跟车距离控制，处理完全基于单张图像维度：单个图像送入神经网络，识别并输出向量空间的一小块碎片。当时的图像输入规格为 1280×960 分辨率、12 位整数（12-bit integers），以大约 36Hz 的帧率持续流式输入。为了高效兼顾精度与延迟，特斯拉团队选用了 RegNet（一种通过网络设计空间搜索得到的残差网络变体）作为特征提取的骨干网络（Backbone）。RegNet 形成了从底层到顶层的特征金字塔：底层特征图拥有极高的空间分辨率但通道数较少，负责捕捉纹理与精细细节；顶层特征图空间分辨率较低但通道丰富，富含整幅画面的全局上下文语义。紧随其后，团队采用双向特征金字塔网络（BiFPN）实现跨尺度的上下双向融合，这让底层判断局部物体的神经元能够获得全局消失点上下文，大幅降低了在公路尽头远距离目标上的误判率。在特征融合之上，网络连接了专门的任务头（Task Heads）。以目标检测为例，特斯拉采用类似 YOLO 的一阶段检测头，在图像网格上输出物体的置信位（Binary Bit）及其三维尺寸、边界框偏移（x, y, w, h）、车辆细分类型等属性。随着感知任务激增到车道线拟合、交通灯状态识别等数十个类别，特斯拉迅速演进出了共享单一骨干、分化出众多任务头的经典 HydraNet（九头蛇网络）多任务架构。卡帕斯指出该架构具有两大决定性优势：其一，多任务共享前向传播计算，大幅分摊并降低了车载计算机的推理计算量；其二，实现了任务间的解耦工程化，各算法小组可以独立迭代特定任务的数据集与输出头结构，而不会相互干扰或破坏其他任务的稳定性。 - 原始图像处理能力：1280×960 规格，12 位精度，36Hz 采样流 - 骨干特征提取器：RegNet 结构，优异的延迟-准确率权衡（Latency vs. Accuracy trade-off） - 跨尺度融合机制：BiFPN 让高层消失点语境辅助底层微小分辨率目标的判别 - 架构形态：HydraNet 共享主干网络，下游分支挂接几十个独立的感知任务头（Task Heads） ### [54:40](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3280s) HydraNet 特征缓存与训练工作流 Andrej Karpathy 解释了 HydraNet 架构在工程训练流程中的优势。由于主干网络与各个任务预测头（Heads）之间存在明确的瓶颈层特征，特斯拉团队通过将多尺度特征（multi-scale features）缓存到磁盘，大大降低了模型微调的成本。在日常迭代中，针对单一任务的更新不需要耗时耗资地重新验证所有其他任务；团队只需从缓存的特征出发，只对特定的头部进行微调。整体训练策略采用周期性混合模式：团队偶尔会进行一次全流程端到端（end-to-end）联合训练，将特征重新缓存到磁盘，随后依赖这些缓存特征执行密集的头部微调，过一段时间再进行新一轮端到端联合训练。在数年前的早期阶段，HydraNet 完全基于单帧图像进行处理，并输出极其丰富的预测项。网络直接从 2D 像素中识别静态和动态环境，包括停车标志（stop signs）、停止线（stop lines）、车道边缘线、行驶中的车辆、红绿灯、路缘（curbs）、车辆是否处于停放状态，以及垃圾桶和反光锥桶等各类静态障碍物。 - **解耦微调**：修改单一任务头时无需重新运行并重新验证其余头部，避免了昂贵的算力浪费。 - **特征缓存**：将骨干网提取的多尺度特征持久化落盘，日常研发直接基于缓存特征微调目标任务头。 - **训练节奏**：以“偶尔端到端联合训练更新骨干网，日常高频微调局部预测头”的循环进行迭代。 ### [55:50](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3350s) 传统 2D 图像预测向向量空间投影的失败瓶颈尽管单图 2D 预测取得了不错的静态结果，但随着团队推进 FSD（Full Self-Driving）功能，这一架构很快暴露出致命缺陷。最先暴露出问题的功能是 Smart Summon（智慧召唤）：车辆需要绕过复杂的停车场找到车主，而自动驾驶系统绝对无法直接在 2D 图像像素空间（image space）上进行车辆控制与规控，必须将预测结果投射到自车周围的 3D 向量空间（vector space）中。最初，特斯拉尝试在 C++ 层实现这一投影管道，开发了名为“占用率追踪器”（Occupancy Tracker）的模块。该系统试图将来自各个单相机的 2D 路缘预测跨越相机边界、并在时间维度上进行几何缝合拼接。Karpathy 指出这套方案存在两大核心死穴：“首先，我们很快发现调试占用率追踪器及其所有超参数极其复杂；你根本不想在 C++ 里手动硬编码做这件事，你希望把它放进神经网络里进行端到端训练。”其次，图像空间本质上就不是正确的输出空间，因为精准逆投影需要整张图像上每个像素具备极致准确的深度值，在微小的像素误差下投射到向量空间就会彻底失真、发生严重扭曲，“我们在向量空间里看到的预测非常糟糕，根本无法基于它进行车辆驾驶”。此外，遮挡区域在图像空间中完全没有像素对应物，无法在 2D 空间表达；而在目标检测任务中，大型车辆经常跨越 8 个相机中的 5 个，没有任何单个相机能看到整车全貌，单相机预测后再进行后期融合变得异常困难。 ```mermaid flowchart TD accTitle: 传统图像空间投影与直接向量空间预测对比 accDescr: 展示从传统的单相机2D预测加后期拼接，转变为多相机特征联合端到端输出向量空间的演进。 subgraph OLD["早期方案：图像空间与 C++ 拼接"] A1["单相机输入 (8路图像)"] --> B1["单图像 2D 预测 (像素空间)"] B1 --> C1["C++ Occupancy Tracker (几何反求/时空拼接)"] C1 -.->|"深度微小误差被无限放大"| D1["扭曲畸变的向量空间 (无法驾驶)"] end subgraph NEW["现代方案：端到端向量空间直接预测"] A2["8路相机图像同时输入"] --> B2["共享 Backbone 提取特征"] B2 --> C2["空间特征重表示层 (Transformer)"] C2 --> D2["干净的自顶向下向量空间 (BEV 直接规控)"] end ``` ### [58:10](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3490s) Transformer 驱动的鸟瞰图（BEV）空间变换面对几何投影的失败，特斯拉构想的全新范式是将全部 8 路相机的原始图像同时输入一个统一的大型神经网络，并在网络内部完成特征空间变换，直接端到端输出自车坐标系下的向量空间（鸟瞰图 BEV）。这一重构伴随两个核心难题：第一，必须设计一个可微分（differentiable）的网络结构来实现从 2D 图像特征到 3D 向量空间特征的投影与重构，从而支持全局反向传播；第二，神经网络如果直接输出向量空间数据，就必须拥有真正的向量空间真值标签（Vector Space Labels），这直接颠覆了传统的图像标注流水线。在网络结构层面，直接基于相机内外参做固定几何硬投影无法适应复杂的现实物理世界——路面可能是上坡或下坡，甚至前方存在车辆遮挡，导致某个鸟瞰图坐标对应的实际视觉证据并不落在理论投影像素点上。为了解决这种强数据依赖（data-dependent）的空间对齐问题，特斯拉引入了 Transformer 与多头自注意力机制（Multi-Head Cross-Attention），哪怕仅使用单个 Attention Block 就能取得出色效果。 Karpathy 形象地解释了 Transformer 内部 Query（查询）与 Key/Value（键/值）的交互机制：“本质上，每一个图像切片都在其 Key 中向外广播：‘我是位于大致某个方位的某根柱子的一部分，我看到了这些特征’；而输出向量空间的每一个 Query 则在询问：‘我是位于输出空间该坐标的一个像素，我正在寻找某种类型的特征’。”两者点积交互后，网络自适应地从多视角图像特征中抽取对应的 Value，成功将特征重构至鸟瞰图向量空间。 ```mermaid flowchart LR accTitle: BEV Transformer 空间特征转换机制 accDescr: 展示输出栅格的 Query 与输入相机的 Key 和 Value 如何通过 Cross-Attention 完成空间重表示。 subgraph BEV_SPACE["BEV 目标向量空间"] R["初始化 BEV 输出栅格"] --> PE["正弦/余弦位置编码 (Sin/Cos Positional Encodings)"] PE --> MLP["MLP 投影"] MLP --> Q["Query 向量: '我在BEV坐标(X,Y)，寻找路缘/障碍物'"] end subgraph IMG_SPACE["多相机 2D 图像空间"] CAM["8路图像特征 (Backbone)"] --> KV["Key & Value 向量: '我是左侧相机柱子切片，特征如下'"] end Q --> ATTN["Multi-Head Cross-Attention 交叉注意力交互"] KV --> ATTN ATTN --> FEAT["向量空间特征表示 (BEV Features)"] FEAT --> HEADS["解码预测头 (Heads: 车道线/路缘/动态障碍物)"] ``` ### [01:01:30](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3690s) 虚拟相机校准与向量空间网络落地除了注意力机制，解决工程实现中的标定误差同样至关重要。Karpathy 坦言，产线上下线的每台车其实“都存在轻微的斜视（slightly cockeyed），每台车的相机安装角度都存在机械公差”。如果模型无法精确感知外参变化，图像到向量空间的重构就会出现严重的虚影与模糊。早期的一种直观做法是将所有相机的几何外参拼接起来用 MLP 灌入网络，但特斯拉工程团队发现了一种更优雅且高性能的解法：在网络的最底层引入一层图像校正变换层（Image Rectification Layer）。这一变换层以单车的精确相机标定参数为函数，将所有物理相机的输入实时插值纠正并映射到一个合成的“虚拟标准相机”（synthetic virtual common camera）视场中。为了直观展现差异，Karpathy 展示了车后视侧方后向盲区相机（Repeater Camera）在未校准前的平均叠加画面——未校正的图像堆叠在一起呈现出大面积模糊，而经过校准变换后，车身后侧的倒车镜与轮廓变得极其锐利清晰。当这一整套工程流水线全部跑通后，直接在向量空间输出的多相机统一网络实现了颠覆性突破。对比早前 C++ 拼接算法的扭曲乱象，全新端到端网络输出的预测结果“基本上是白天与黑夜的差距（basically night and day），你终于可以真正基于这套空间表示来直接驾驶了”。该模型同时大幅提升了多视角联合目标检测的稳定性，彻底消除了大车跨相机缝合时的错位现象。 - **物理公差问题**：量产车装配存在微小安装角偏差，若直接做特征投射会导致 BEV 空间特征发散。 - **虚拟标准相机校正**：基于实时内外参，在像素输入级通过 Rectification Transform 将物理画面投影至几何统一的虚拟视场。 - **跨视角连贯性**：消除了大型长车在穿越多相机接缝时的断裂重影，使 3D 边界框与可通行区域呈现出高稳定性的空间连续性。 ### [01:03:05](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3785s) 多相机网络与时间维度的缺失通过实际路测视频，Andrej Karpathy 直观对比了单相机与多相机网络的效果差异。在演示视频中，橙色线框代表单相机（单张图像）预测结果，蓝色线框则代表多相机融合网络的预测。当周围车辆距离极近、单颗摄像头只能捕捉到车体微小边缘（tiny sliver）时，单相机网络往往漏检或估计出偏差极大的位置。此外，当大型车辆在狭窄空间内穿跨不同摄像头的重叠视野交界处时，单相机系统由于缺乏跨传感器的一致性，会在交界处产生大量的抖动与跳变（jank）。多相机网络直接在统一的向量空间（Vector Space）输出结果，彻底消除了这些跨界拼接瑕疵。然而，多相机网络虽然解决了空间维度的跨视角感知问题，但在时间维度上依然是“孤立无援”的——系统依然在每一个孤立的离散时间点上独立运行推断。Karpathy 指出，自动驾驶任务中有极大量的关键预测本质上依赖“视频时序上下文（video context）”：眼前这辆车究竟是静止停放还是正在移动？移动速度有多快？如果一辆车被前方穿行的大货车暂时遮挡，它是否还存在？再比如要准确推断前方的道路拓扑结构，自车往往需要依赖 50 米前刚刚看过的路标或路面指引箭头。为了解决这些纯粹依靠单帧无法回答的问题，团队决定在神经网络架构中正式引入时序处理模块（Video Module）。 ```mermaid flowchart LR accTitle: 特征缓存与时序融合模块流水线 accDescr: 展示多尺度特征经由特征队列缓存并与运动学数据融合后输入时序模块的结构 subgraph S1["空间特征提取"] A["多相机图像"] --> B["多尺度特征提取"] end subgraph S2["时序缓存与融合"] B --> C["Feature Queue (时间/空间队列)"] K["自车运动学数据 (速度/加速度)"] --> C C --> D["Video Module (时序融合网络)"] end subgraph S3["下游解码"] D --> E["多任务输出头 (检测/道路拓扑/深度等)"] end ``` ### [01:04:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=3894s) 特征队列（Feature Queue）：时间与空间的双重缓存机制为了将历史信息送入神经网络，特斯拉设计了一套特征队列（Feature Queue）流水线。该模块将当前多尺度特征、自车运动学数据（Kinematics，包括自车的行驶速度与加速度）以及位置编码进行拼接（concatenate）和编码，按规则推入队列中进行缓存，随后提供给时序模块统一消费。然而，设计特征队列的核心挑战在于“入栈（Push）与出栈（Pop）机制”：系统到底该在什么时机将特征存入队列？ Karpathy 用一段自车接近十字路口的典型挑战场景进行了解释：自车从道路下方驶入交叉路口，横向车流穿行导致前方原本可见的车辆被短暂遮挡，随后自车必须长时间原地等待红灯。如果仅设计一个按时间推进的队列（例如每隔 27 毫秒推入一帧特征），虽然能让神经网络在目标被遮挡瞬间“翻阅”过去的短期记忆，维持对被遮挡车辆的稳定追踪，但却会引发另一个致命缺陷：当自车停在红灯前静止不动数十秒时，固定容量的时间队列会迅速被原地静止的冗余帧占满，从而彻底洗掉并“遗忘”几十米前车辆仍在移动时看到的车道指示标线与路牌信息。为了解决这一矛盾，特斯拉同时部署了双重缓存逻辑： - **基于时间的特征队列（Time-based Queue）**：按固定时间间隔（如每 27ms）记录特征，专门用来捕捉高速动态变化、短时遮挡推理以及即时速度/加速度估计。 - **基于空间的特征队列（Space-based Queue）**：每当车辆实际行驶固定位移时才推入特征，保证自车在红灯前无论静止等待多久，之前行驶过程中记录到的路面拓扑特征与道路标识都不会被冲刷覆盖。 ### [01:06:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4014s) 空间循环神经网络（Spatial RNN）与隐状态地图在选择如何融合队列中的时序信息时，特斯拉工程团队系统探索了 3D 卷积（3D Convolutions）、标准 Transformers、用于降低计算复杂度的轴向注意力网络（Axial Transformers）以及各类经典循环神经网络（RNN）。最终，他们重点打造了一种兼顾精度与车载计算效率的方案：空间循环神经网络（Spatial RNN）。由于地面车辆行驶本质上是在二维平面表面（2D surfaces）上运动，特斯拉将 Spatial RNN 的隐状态（Hidden State）直接组织为一个俯视视角的二维空间网格晶格（2D lattice）。当车辆在道路上行驶时，网络利用自车运动学积分实时追踪自车在隐状态网格中的相对坐标，且仅局部更新自车附近具备可见视野（visibility）的网格节点。这意味着神经网络获得了主动控制内存读写的主动权：“如果旁边有一辆车遮挡了部分路面，网络可以自主选择不向遮挡区域写入；一旦遮挡物驶离、视野开阔，RNN 便会判定当前具备清晰可见度，进而果断写入该空间区域的信息。” - 隐状态经多任务端到端训练收敛后，不同通道自然分化并特化记录不同的道路要素：部分通道追踪道路中心线，部分通道刻画道路边缘与边界，另一些通道则专门记忆路面几何材质。 - 在多个十字路口连续通行的可视化展示中，抽取隐状态前 10 个通道的均值即可清晰看到网络随行进动态记忆、擦除与更新的完整空间地图。 - Karpathy 强调，这种基于单趟行驶构建的方式可以自然拓展：“你能想象不仅是单趟行程，车队中大量不同车辆的多段行程可以协同构建这一地图，本质上就是一种高精地图（HD Map），不同的是它完全不存在于显式几何物体的空间中，而是直接存在于循环神经网络的隐特征空间（Feature Space）里，这非常酷，我以前从未见过。” ### [01:09:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4175s) 时序视频网络在目标遮挡与测速上的性能飞跃引入时序视频模块后，特斯拉在目标检测与运动学估计上取得了质的飞跃。在遮挡对比测试中，道路远端停着两辆车，另一辆横向车辆驶过并短时遮挡住它们。纯单帧网络在视野清晰时能生成边界框，但在目标被遮挡时检测框瞬间丢失；当目标只露出少许边缘处于半遮挡时，单帧网络由于被迫仅凭不完整的像素做出预测，输出了畸变极其离谱的预测框。与之相对，视频时序网络凭借历史特征记忆，在目标完全遮挡时依然能够稳定保持（persist）目标状态；在半遮挡时，网络清楚该区域处于弱可见状态，不会被残缺碎片误导。在特斯拉推行“纯视觉去雷达（Remove the Radar）”的技术攻坚中，时序网络成为了彻底取代毫米波雷达的关键支撑。雷达传统上最核心的优势在于直接利用多普勒效应精准测量物体的深度与瞬时速度。在实测对比中，绿色曲线代表毫米波雷达的深度与速度真值，橙色曲线为单帧视觉网络的估计，蓝色曲线则为视频时序网络的输出。单帧视觉在速度估算上波动剧烈、噪声极大，而结合了空间时间特征融合的视频网络，其预测平滑度与绝对精度不仅与毫米波雷达信号紧密贴合，甚至在许多噪声工况下展现出了超越物理毫米波雷达的鲁棒性。 ### [01:11:02](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4262s) 特斯拉纯视觉端到端网络架构总览与未来改进方向 Andrei Karpathy 展示了融合时间序列后的深度与速度预测效果对比。在单帧图像网络中，系统无法直接提取速度信息，以往只能通过对深度进行微分来间接推导，信号噪声极大；而在引入视频时间模块后，速度预测曲线近乎完全贴合毫米波雷达的实测信号，证明了纯视觉方案在测距和测速质量上已经能够比肩甚至取代雷达。随后，Karpathy 完整梳理了特斯拉当时的视觉神经网络整体架构：底层输入来自 8 颗摄像头的原始图像，首先通过校准矫正层（Rectification Layer）修正相机物理参数，将图像投影到统一的虚拟相机空间；接着使用 RegNet 残差网络提取多尺度特征，并通过 BiFPN 进行多尺度特征融合；随后利用 Transformer 模块将图像空间特征重构映射到输出的 3D 向量空间（Vector Space）；这些特征被送入按时间或空间组织的特征队列中，由空间 RNN（Spatial RNN）等视频模块进行时空关联处理；最终注入 HydraNet 架构的分支主干（Trunk）与各类任务头（Heads），输出道路、静态障碍物及动态物体的全量矢量预测。 ```mermaid flowchart TD accTitle: 特斯拉视觉神经网络整体架构 accDescr: 展示从8路原始图像输入到最终多任务HydraNet头输出的数据处理流水线 subgraph Input ["多相机图像输入与校准"] RAW["原始图像输入 (8路相机)"] --> RECT["校准矫正层 (虚拟相机坐标)"] end subgraph Backbone ["特征提取与空间转换"] RECT --> REG["RegNet 残差网络 (多尺度特征)"] REG --> BIFPN["BiFPN 多尺度特征融合"] BIFPN --> TRANS["Transformer 模块 (图像空间转向量空间)"] end subgraph Temporal ["时空记忆与多任务头"] TRANS --> QUEUE["时空特征队列 (Feature Queue)"] QUEUE --> RNN["视频模块 (Spatial RNN)"] RNN --> HYDRA["HydraNet 多分支任务头 (检测/分割/深度等)"] end ``` Karpathy 指出，该网络在过去三到四年间已经从简单的单帧图像网络演变成如今极为复杂的时空大模型，但团队仍在积极探索进一步优化的空间。目前的时空融合在网络层级中发生得相对靠后，团队正在探索更早期的空域与时域融合方案，例如在底层构建成本体（Cost Volumes）或类似光流的网络模块；另一方面，当前网络的输出端主要采用密集栅格（Dense Rasters），在车载计算平台上对其进行后处理的计算开销非常昂贵。Karpathy 总结道：“我们的输出是密集的栅格，在车端后处理这些密集栅格其实代价相当高昂，而我们显然面临着极其严苛的延迟限制，因此这并不理想。”团队正探索直接以稀疏点集或矢量结构的方式预测道路几何特征，从而彻底免除繁重的栅格后处理流程。 - 输入数据首先通过虚拟相机校正层统一校准，消解各硬件安装位置的物理偏差。 - 特征提取采用 RegNet 与 BiFPN 组合，平衡计算吞吐量与多尺度语义表征能力。 - 输出端正从耗时的密集栅格向稀疏点阵（Sparse Point-by-point）预测演进，以满足严苛的车载毫秒级推理延迟要求。 --- ### [01:13:11](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4391s) 规划与控制面临的挑战：非凸高维动作空间与分层求解策略视觉网络将海量密集视频压缩为 3D 向量空间后，自动驾驶的核心任务便转入规划与控制（Planning and Control）。特斯拉规划控制、自动标注与仿真团队负责人 Ashok Elluswamy 登台接棒讲解。他强调，规划器的终极职责是基于感知的向量空间，在最大化行车安全、舒适性与通行效率的前提下将车辆安全导向目的地。虽然早在 2019 年特斯拉的规划系统就已经能在高速公路上自如巡航、自主变道以及驶入驶出匝道，但非结构化的城市道路驾驶复杂度呈指数级上升——城市路段缺乏规整的车道线，车辆行车轨迹更加自由发散，同时必须实时应对复杂的弯道盲区、穿行车辆以及不可预测的行人和非机动车行为。 Ashok 剖析了自动驾驶规划的核心算法瓶颈：动作空间具有极端的“非凸性”（Non-convex）与“高维性”（High-dimensional）。非凸性意味着存在大量局部最优解，全局一致的解极其难以直接求解，优化算法极易陷入局部死锁；高维性则源于规划器必须对未来 10 到 15 秒内的运动状态进行连续预测，并实时生成包含位置、速度和加速度的时序参数矩阵。传统的离散搜索（Discrete Search）算法由于具备跳跃探索特性，在非凸空间中不易陷入局部极小值，但由于缺乏梯度信息引导，面对高维空间时会因算力爆炸而失效；反之，基于梯度的连续函数优化（Continuous Optimization）在处理高维空间时能极速收敛，但在非凸空间中极易被局部极小值困住。对此，特斯拉设计了分层求解架构：“我们针对这个规划问题的解决方案是将其进行分层拆解：首先利用粗粒度搜索方法化除非凸性并构建出一个凸走廊，然后再利用连续优化技术生成最终的平滑轨迹。” - 规划视界需覆盖未来 10 至 15 秒的时空窗口，实时求解高维参数轨迹。 - 离散搜索擅长规避非凸空间的局部极小值，但在高维度无梯度信息时搜索效率极低。 - 连续优化在梯度引导下求解高效，却依赖良好的初始凸约束以防陷落。 --- ### [01:15:26](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4526s) 候选搜索与博弈交互：全局多智能体联合规划在第一阶段的粗粒度离散搜索中，系统基于物理车辆动力学模型快速模拟成千上万种候选机动动作。Ashok 以双连续变道以完成前方左转的复杂工况为例：规划器最初尝试立即就近变道，但发现减速制动过于剧烈，严重违背舒适性指标；随后尝试加速超车后再切入，但过晚变道会导致车辆彻底错过左转路口。系统在极短时间内模拟评估数千条物理轨迹，权衡安全性、舒适度和导航完成度，选出最优的基准路径。实测数据显示，车辆实际执行的行车速度曲线与规划器提前 10 秒生成的规划曲线近乎完全重叠。在多车混行的狭窄走廊与非结构化路况中，特斯拉规划器打破了孤立自车的传统思路，引入了博弈交互的全局联合规划。Ashok 强调，自动驾驶不能只规划自己，必须联合优化整片场景的交通流：“为了做到这一点，我们实际上是对场景中的每一个相关目标都直接运行一遍 Autopilot 规划器。”在演示视频中，Autopilot 穿行于两侧停满违章车辆、放置路障雪糕筒的狭窄通道。面对第一辆迎面而来的对向车，系统判断自车侧方完全没有避让空间，而对向车具备靠边减速让行的物理条件，Autopilot 遂保持速度果断前进，对方果然主动减速避让；紧接着第二辆对向车以较快速度驶来，系统为该目标车辆同时运行规划器并模拟出两种未来：红色的强行冲刺轨迹与绿色的避让轨迹。由于对方速度与加速度偏高，系统判定其不会轻易让行，主动减速向右借道避让；但在避让过程中，系统通过对向车的偏航率（Yaw rate）和加速度细微变化，敏锐捕捉到对方最终决定刹车礼让，Autopilot 瞬间修正决策重新果断前插通行。Ashok 直言：“如果不这样做，Autopilot 就会变得过于胆怯，也就无法成为一辆具备实用价值的自动驾驶汽车。” ```mermaid flowchart TD accTitle: 交互式多智能体联合规划逻辑 accDescr: 展示自车规划器如何联合推算对向车辆的多条潜在轨迹并动态调整自身路权博弈 subgraph Perception ["环境输入与意图推演"] SENSE["感知检测：对向来车高速驶入狭窄道路"] --> SIM["为对向车辆运行 Autopilot 规划器"] SIM --> HYPO_A["推演意愿 A (绿色)：对方主动减速礼让"] SIM --> HYPO_B["推演意愿 B (红色)：对方高速借道强行通过"] end subgraph Decision ["博弈判断与动态调整"] HYPO_B --> CHECK["对向车速度/加速度偏高：判定为意愿 B"] CHECK --> YIELD["自车决策：主动减速并向侧方避让"] YIELD --> OBSERVE["实时监测对方偏航角速度 (Yaw Rate) 与加速度"] OBSERVE --> SHIFT["捕捉到对方减速减角信号：对方转为礼让"] SHIFT --> ADVANCE["自车瞬间改变决策：重新果断前行通过"] end ``` 在离散搜索与多智能体意图推演确定出无碰撞的安全凸走廊（Convex Corridor）后，规划系统进入第二阶段的连续优化。优化算法在沿规划弧长参数化的航向角与加速度样条曲线（Splines）上进行微积分级别的精细调整，综合考量障碍物距离边界、道路通行时间、纵横向冲击度（Jerk）等联合代价函数。最终生成的横向加速度呈现出极其平滑且符合人体工程学的梯形曲线，使车辆的动态表现与高水平人类驾驶员的操作几无二致。 - 多智能体博弈中，系统基于对向车辆的偏航率（Yaw rate）与加速度动态反推其真实驾驶意图。 - 凸走廊通过离散搜索消除拓扑歧义，样条曲线连续优化则负责压低横向加速度突变。 - 横向加速度曲线被优化为平滑的梯形包络线，彻底消除顿挫感与机械转向痕迹。 ### [01:19:53](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4793s) 经典规划算法在非结构化场景中的局限与泊车玩具问题在规划与控制领域，传统方法通常是建立凸走廊（convex corridor）并在此基础上优化平滑路径。在规整的结构化道路上，这种机制表现尚可，但在世界上许多其他地方——例如主讲人成长的故乡——交通环境极其混乱且非结构化：汽车与行人随意穿插加塞、急刹与鸣笛此起彼伏。如果试图在运行时（runtime）单靠规模化堆砌传统求解器来应对这种现实世界，计算上几乎不可行；Tesla 的思路转向采用基于学习的方法（learning-based methods）来高效求解。为了直观展现传统搜索算法的死穴，演讲者引入了一个玩具级的泊车问题（toy parking problem）：蓝色小车为主车（ego car），目标是停入绿色的停车位，途中必须绕过路缘（curb）、停放的其他车辆以及橙色锥桶。作为基准的经典 $A^*$ 算法使用状态网格空间搜索，并将“到目标的欧氏距离”作为启发式函数（heuristic）： ```mermaid flowchart LR accTitle: A-Star Search Failure Mode accDescr: 经典 A* 算法因贪心直奔目标而陷入死胡同并在局部极值中产生大量回溯 subgraph Classical["经典 A* 搜索缺陷"] A["直奔目标"] --> B["陷入死胡同/局部极值"] B --> C["巨量回溯状态重搜"] C --> D["耗费 40 万节点"] end ``` 由于只认到终点的直线距离，小车一开始直奔目标点，很快就被停放的车辆和障碍物堵进局部极值（local minima），随后只能痛苦地一步步回溯（backtrack），四处搜寻绕行路径，最终足足展开了 400,000 个节点才找到可行解。如果为其引入导航路径启发，让车辆在靠近终点的同时跟随粗略导航，情况虽略有好转，但一旦遭遇未标记的锥桶，搜索算法依然对前方障碍一无所知——它必须真实碰撞后才懂得倒退重来，依然耗费了 22,000 个节点。手工设计针对单个锥桶的距离避障函数不仅繁琐，而且只能解决局部问题，根本无法产生全局最优解。 - 欧氏距离启发式 $A^*$ 算法：耗费 400,000 个搜索节点才完成简单泊车。 - 加入导航路径启发式：耗费 22,000 个搜索节点，依然存在盲目碰撞和低效回溯问题。 - 核心缺陷：人工设计的启发式函数无法提供全局价值函数（global value function）。 --- ### [01:22:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=4935s) 基于神经网络与蒙特卡洛树搜索（MCTS）的规划架构视觉网络输出的是一个包含自车及周围动态障碍物的连续向量空间，这在本质上与 Atari 游戏以及围棋环境高度相似，且属于一个多人交互系统。Tesla 因此借鉴了 DeepMind 解决围棋与 Atari 游戏的 MuZero 和 AlphaZero 技术：训练神经网络直接输出状态与动作概率分布，并将其注入到蒙特卡洛树搜索（MCTS, Monte Carlo Tree Search）中。在同样的泊车场景下，引入神经网络价值函数后，MCTS 展现出惊人的全局直觉：规划器几乎是一步到位、毫不犹豫地直接逼近目标点。屏幕上虽然可视化了其他搜索分支，但主分支径直避开盲区与障碍物。由于神经网络直接吸收了全局上下文（global context），它为搜索提供了正确的全局最小值引导，最终仅搜索了 288 个节点便完成了轨迹规划，相比纯几何启发式算法实现了数个数量级的算力压缩。 ```mermaid flowchart TD accTitle: Neural-Assisted Planner Architecture accDescr: 密集视频特征由视觉系统压缩为向量空间后，与神经网络及显式规划器联合输出轨迹分布 subgraph VisionPipeline["视觉压缩层"] V["密集多摄像头视频数据"] --> S["视觉神经网络系统"] S --> VS["向量空间 (Vector Space)"] S -.-> F["网络中间特征 (Intermediate Features)"] end subgraph HybridPlanner["混合规划系统"] VS & F --> NP["神经网络规划器 (Neural Net Planner)"] VS --> EP["显式规划器 (Explicit Planner)"] NP & EP --> Dist["轨迹分布 (Trajectory Distribution)"] end subgraph LossAndOptimization["端到端成本优化"] Dist --> Cost["端到端轨迹优化"] C1["显式成本 (碰撞/舒适度/时间)"] --> Cost C2["人类接管干预 & 模仿学习数据"] --> Cost Cost --> Cmd["底层显式规划器控制执行 (生成转向与加减速指令)"] end ``` 在量产规划控制框架中，显式规划与学习规划深度融合：端到端优化目标不仅包含显式的碰撞风险、舒适度与通行时间，还直接纳入真实人类驾驶的干预接管事件与模仿学习数据。正如主讲人总结道：“视觉网络将密集的视频数据压缩进向量空间，显式规划器与神经网络规划器共同消费这些表征，生成最终的控制指令。” - MCTS 配合神经网络价值函数：仅耗费 288 个节点即求解成功。 - 成本函数构成：几何硬约束（碰撞检测、距离）、运动舒适度、耗时评估、人类脱困/干预数据（interventions）以及人类模仿轨迹。 --- ### [01:24:34](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5074s) 标注体系演进：从第三方外包到千人机载自研数据团队随着技术细节展开，Andrej Karpathy 重新回到台前，强调模型与数据集之间的本质关系： > “神经网络仅仅确立了性能的上限；许多此类神经网络拥有数以亿计的参数，而这数以亿计的参数必须被正确设定。”（“Neural networks only establish an upper bound on your performance. Many of these neural networks, they have hundreds of millions of parameters, and these hundreds of millions of parameters, they have to be set correctly.” —— Andrej Karpathy）若没有海量、高质、多样的向量空间标注样本，巨大的参数容量就毫无用处。Karpathy 回忆了 Tesla 数据团队的发展史：大约四年前他刚加入 Tesla 时，团队依赖第三方外包商进行数据标注。但他们很快痛苦地意识到，外包模式对于自动驾驶这种关键核心任务而言完全行不通——第三方交付延迟极高，数据质量亦难以达到自动驾驶工程的标准。秉承 Tesla 一贯的全垂直整合（full vertical integration）哲学，工程团队毅然将全部标注任务收归自研，并从零开始自主搭建底层标注软件基础设施。如今，Tesla 已建立起一支超过 1,000 人的全职专业标注团队。这支团队不仅常驻美国本土，更与自动驾驶算法工程师团队紧密在同一物理办公区协同办公。Tesla 内部构建了严密的数据看板，实时监控每个标注人员、每项作业管线在延迟、吞吐量和标注质量上的变化曲线。 - 团队规模：超过 1,000 名专业内部标注员。 - 组织模式：全流程自研软件，标注团队与硅谷算法工程团队同址办公（co-located）。 - 关键监控指标：标注延迟（latency）、数据吞吐量（throughput）以及单人/任务维度的质量统计（quality statistics）。 --- ### [01:26:32](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5192s) 从 2D 图像多边形标注跨越至 4D 向量空间与自动标注序幕数据形式的演化直接驱动了视觉架构的跨越。三至四年前，团队的标注工作完全基于单帧 2D 图像空间（image space）：标注员必须在多张离散的照片上手工勾勒复杂的二维多边形（polygons）和折线（polylines）。这种方式不仅耗时极长，而且生成的二维像素标签根本无法匹配三维世界中自车规划所需的物理向量。为打破吞吐瓶颈，Tesla 标注管线迅速升级至 3D 与 4D（时空维度）的向量空间标注。系统利用自车行驶轨迹对地面几何网格进行初步重建，并生成局部的稀疏点云。标注员不再处理单张相机图片，而是直接在由点云和时间切片组成的三维向量世界中拖拽、修改边界框和空间路径；修改结果随后自动重投影（reproject）到周围所有的相机视图中。这种“一次 3D 标注、全局多目投影”的作业方式，直接将人工标注的吞吐效率提升了数倍。 ```mermaid flowchart LR accTitle: Evolution of Labeling Paradigms accDescr: 从早期2D图像手动勾勒多边形演变为3D/4D向量空间重投影标注，并最终迈入人机协同的自动标注 P1["早期模式: 2D 图像空间 (逐帧手工画多边形/折线，极度缓慢)"] --> P2["中期模式: 3D/4D 向量空间 (单次 3D 点云空间标定，自动重投影至全向相机)"] --> P3["当前阶段: 自动标注管线 (算法做大规模几何重建，人工专攻语义理解)"] ``` 然而，即使 3D 空间标注大幅提效，面对训练千万级深度模型所需的海量场景数据，单纯依靠人工依然存在物理极限。人类的大脑极为擅长高层语义推断（semantics），但在三维几何测量、三角测量（triangulation）、大规模密集追踪与重建方面，机器算法却具备压倒性的计算优势。为了弥补千人团队与天文数字级训练数据之间的缺口，Tesla 决定重兵投入大规模自动标注（Auto Labeling）技术，将高密度的传感器片段（sensor clips）转化为完全自监督、高精度的真值生成管线。 - 2D 图像标注缺陷：耗时长、透视失真严重、无法直接反映 3D 物理空间关系。 - 4D 向量标注特性：在时空重建的地面和点云中作业，自动重投影至各路相机视图。 - 人机分工逻辑：人类处理复杂语义判决，机器算法负责精密几何解算与多帧轨迹追踪。 ### [01:28:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5308s) 基于隐式神经表示与多行程联合优化的路面自动标注特斯拉的离线自动标注管线首先依赖海量的数据采集。工程测试车或客户车队会上传包含 IMU、GPS、测程法（odometry）等传感器数据的视频片段，通常时长在 45 秒到 1 分钟左右。这些片段被汇集到云端服务器集群，通过离线运行大量复杂的神经网络生成中间语义结果（如分割掩码、深度、特征点匹配等），随后再经过一系列机器人学和优化算法，最终提炼出高质量的真值标签用于模型训练。 ```mermaid flowchart TD accTitle: 自动标注管线架构 accDescr: 展示从车辆数据采集、中间网络处理、NeRF隐式重建到跨行程联合优化的离线自动标注完整链路。 subgraph Collection["数据采集阶段"] A["车队 / 工程车片段 (45-60s 视频 + IMU/GPS)"] end subgraph Intermediate["离线网络处理"] B["服务器离线神经网络集群"] C["中间结果 (分割掩码 / 深度 / 点匹配)"] A --> B --> C end subgraph SurfaceRecon["NeRF 隐式路面重建"] D["XY 平面查询"] E["预测 Z 高度 + 语义属性 (路缘/车道线/可行驶空间)"] F["重投影至 8 个环视相机"] G["时空联合优化损失 (对比图像分割)"] C --> D --> E --> F --> G end subgraph MultiTrip["多行程聚合"] H["同一路口不同行程片段 (如16次)"] I["特征对齐与联合大优化 (超出行驶轨迹推断)"] G --> H --> I J["最终高精标注数据集"] I --> J end ``` 在所有标注任务中，首要解决的是路面几何结构。传统做法通常使用样条曲线（splines）或多边形网格（meshes），但受限于拓扑约束，它们在数学上不可微，极难直接用梯度算法优化重建。特斯拉借鉴了类似神经辐射场（NeRF）的隐式表示方法（implicit representation）： - 算法在地面 XY 平面上进行数以百万计的点查询，网络输出对应位置的地面高度 $Z$ 坐标，以及路缘（curbs）、车道线、路面材质和可行驶空间（drivable space）等语义属性。 - 组合生成的 3D 点 $(X, Y, Z)$ 被重新投影回车辆的全部 8 个摄像头视图中，与图像空间的分割预测结果比对误差。 - 在时间和空间维度上对 8 个相机进行联合全局优化，从而生成平滑、时空一致且高度精准的路面重建。不仅单车行驶可以扫出轨迹周围的一片路面，特斯拉还能将同一物理地点、由不同车辆在不同时间录制的多个片段（例如 16 次不同行驶轨迹）汇聚到一次庞大的联合优化中。各行程之间依靠道路边缘、车道线等几何特征相互对齐，不仅要求所有行程间保持一致，还要符合各自的图像观测。“这产生了一种标注路面的高效方法，不仅能标注车辆开过的地方，还能标注车辆未曾驶过的区域。”这一方案的意图不是为了构建高精地图（HD Maps），而仅仅是为了给穿过该路口的视频片段生成精准真值；只要标注与当前采集的视频自洽即可，无需长期维护维护成本。在此基础上，人工标注员只需介入进行简单的噪声清理或补充元数据。 --- ### [01:31:21](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5481s) 稠密 3D 点云与利用“后见之明”的运动学真值标注除了路面重建，离线服务器管线还能任意重建场景中的静态障碍物，输出高精度的 3D 点云。传统视觉特征匹配极其依赖物体表面的纹理，但在平整的路面或无纹理的白墙上往往失效。特斯拉的核心突破在于能够在缺乏纹理的表面上同样生成密集的 3D 点云，使得任意静态障碍物的 3D 几何标注变得轻而易举。在服务器端进行离线处理的另一个巨大技术红利在于可以利用“后见之明”（hindsight）。车载端模型在实时预测障碍物速度时，只能依赖过去的有限帧进行推算和猜测。然而，在离线自动标注中，系统拥有读取整个视频全局的特权： - 系统不仅可以回溯过去，还能直接查看未来的连续帧，相当于“开挂作弊”获取地面真值，直接解算出绝对准确的速度、加速度等运动学参数。 - 当行人在路口被过往车辆反复遮挡时，车载实时感知往往容易丢失追踪，而离线算法可以通过未来出现的轨迹反向缝合各时间段的追踪轨迹（tracklets）。正如演讲者强调的那样：“因为我们知晓未来，我们拥有未来的轨迹，我们可以将它们匹配并关联起来……这对规划模块极为重要，因为规划器需要知道，即使它现在被遮挡了，如果之前看到过某人，它仍然需要将其纳入考量。” 将静态路面、3D 障碍物点云与带有未来真值的连续动态轨迹相结合，特斯拉打造出了全要素一致的时空标注管线。车辆平滑转弯、行人跨遮挡稳定追踪、静止驻车（零速度精准标记）都能以近乎完美的物理连续性输出。特斯拉的目标是生产 100 万个此类高精度标注的视频片段，以此训练多摄像头视频网络，“彻底攻克这一难题（really crush this problem）”。 --- ### [01:33:36](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5616s) 纯视觉去毫米波雷达战役：雪浆遮蔽挑战与数据闭环自动标注管线的威力在特斯拉“移除毫米波雷达”（Remove the Radar）项目中得到了验证。整个项目推进节奏极其紧凑，团队在大约 3 个月内就彻底去掉了前向毫米波雷达。在纯视觉早期版本中，系统在低能见度或严重遮挡场景下容易出现性能退化。例如，前方大型卡车驶过时扬起大片泥雪，瞬间将后车挡风玻璃和视野完全糊住，网络一时间容易丢失对前车的感知追踪。然而，物理常识要求系统必须“记住”那辆车依然在前方行驶。由于真实物理世界中这种特定恶劣工况的数据极为匮乏，早期的神经网络未能表现出良好的记忆保留能力。特斯拉随后向全球客户车队下发影子触发器，搜寻类似极端工况。车队迅速做出了响应，上传了海量“各类杂物/雨雪从其他车辆甩出”的视频片段。得益于离线自动标注管线，特斯拉在一周之内就完成了 1 万个高难视频片段的标注。如果是传统的人工手动标注，这需要数月时间。针对大约 200 种不同的长尾边缘场景，团队重复了这一闭环流程，极快地构建了超大规模数据集。重新训练后的网络具备了卓越的时序记忆，即便视野被暴雪糊住，也能稳定推断出前方被遮蔽车辆的存在与动态。 - 毫米波雷达移除耗时：约 3 个月完成攻坚 - 数据标注吞吐效率：自动标注系统在 1 周内处理 10,000 个复杂视频片段（人工需数月） - 专项长尾工况类别覆盖：针对超过 200 种细分恶劣条件实施数据挖掘 --- ### [01:34:52](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5692s) 仿真系统构建：解决罕见长尾、复杂稠密标注与闭环评估在展示了一段甚至包含特斯拉 Cybertruck 的高度逼真道路视频后，演讲者揭晓了谜底——该片段并非实拍，而是完全由特斯拉自研仿真引擎实时渲染生成的。 ```mermaid flowchart LR accTitle: 仿真系统的核心优势与适用场景 accDescr: 结构化展示仿真在长尾工况采集、复杂稠密标注以及闭环策略验证三方面的核心价值。 subgraph SimulationAdvantages["仿真系统的三大核心价值"] direction TB A["长尾罕见数据生成 (如行人和狗在高速路狂奔)"] B["复杂密集场景秒级真值 (闹市区百人过街/零遮挡误差)"] C["闭环交互式反馈评估 (决策直接改变后续环境状态)"] end SimulationAdvantages --> D["矢量空间直接导出 perfect ground truth (3D Cuboids / 速度 / 深度 / 法线 / 语义分割)"] ``` 特斯拉在该仿真系统上倾注了大量资源，其逼真度体现在多维细节中： - 地面并非一张平坦光滑的贴图，而是布满了真实的裂纹、焦油接缝与修补沥青块的粗糙路面。 - 交通参与者具备拟真的动力学行为：铰接式大型货车（articulated truck）在路口大角度转弯时会压上路缘石，其他社会车辆则具备智能避障、平滑加减速的能力。 - 场景中头顶带有特斯拉 Logo 的车辆正是运行在仿真环境内部的真实 Autopilot 堆栈，正在自主完成无保护左转（unprotected left turn）。由于整个仿真直接构建在原生“矢量空间”（vector space）之上，系统可以毫不费力地直接提取完美的地面真值（ground truth），涵盖车辆 3D 边界框（cuboids）、动力学速度、深度图、表面法向量以及语义分割。演讲者调侃道，哪怕感知负责人 Andrej Karpathy 下周突发奇想提出一项全新的标注任务，团队也只需编写几行代码，即可在矢量空间中直接批量导出对应真值。演讲者总结了仿真大显身手的三大不可替代场景： 1. **真实世界极难采集的数据**：即便拥有百万级车队，某些罕见极端边缘场景依然可遇不可求，例如“一对情侣和他们的狗在周围全是高速行驶车辆的高速公路上狂奔”。这种极端长尾工况虽罕见但现实中确实会发生，Autopilot 必须具备应对能力。 2. **人工与自动标注算法均极难处理的数据**：例如繁忙市中心数百名行人密集穿梭的十字路口。人工标注需要耗费数小时，而离线自动算法也极易在密集的频繁重叠遮挡中发生特征关联错误、算错速度；但在仿真中，底层本就是精确的对象实体，导出边界框和速度属于毫无难度的基础操作。 3. **闭环行为交互（Closed-loop Behavior）**：在强交互场景中，后续数据分布完全取决于自车当前动作做出的决策，只有仿真环境才能安全、可重复地提供这种闭环反馈测试。 ### [01:37:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5848s) 高保真传感器模拟与物理渲染特斯拉模拟系统的核心目标绝非仅仅为了“生成漂亮的画面”，而是要让算法看到与车端真实传感器完全一致的原始输入。模拟器对真实车载相机的曝光、传感器噪声、运动模糊、光学畸变、前大灯投射特性，乃至挡风玻璃上的光衍射图案进行了极度精细的物理建模。这种逼真度甚至允许特斯拉将该模拟器不仅用于 Autopilot 算法验证，还反向用于指导硬件工程设计——例如光学镜头设计、传感器选型与布局规划，以及前大灯透光特性的优化。为了防止神经网络通过渲染瑕疵“走捷径”，特斯拉不遗余力地消除游戏行业中俗称的锯齿（jaggies）等走样瑕疵，专门引入了时空抗锯齿（Temporal Anti-Aliasing）技术与神经渲染（Neural Rendering）手段，并辅以光线追踪（Ray Tracing）来实现真实的局部与全局光照。在场景资产层面，为了杜绝模型因记住少数几辆车的外形尺寸而产生过拟合，资产库中储备了数以千计的模型变体，从穿戴不同衣物且具备拟真动作的行人，到横穿马路的驼鹿，确保了场景的极端多样性。 - **传感器级物理还原**：涵盖曝光范围梯次匹配、光学畸变、运动模糊、传感器噪点、车灯投射与挡风玻璃衍射效应。 - **画质无瑕与高级光照**：通过时空抗锯齿消除渲染瑕疵，结合神经渲染与光线追踪实现逼真漫反射与全局光照。 - **硬件设计协同**：模拟器被直接作为硬件评估平台，参与镜头形状、传感器车身布局和车灯透镜的工程决策。 --- ### [01:38:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=5936s) 算法程序化生成与故障场景闭环重现为了支撑巨量训练需求，团队不仅依靠美术人员手动搭建了 2000 英里的高精道路环境（其长度相当于横跨美国东西海岸），更将重心转向了算法驱动的程序化道路生成（Procedural Generation）。算法能够根据道路曲率、树木茂密度、路锥、电线杆分布以及周围车辆速度与交互等海量参数，近乎无限地派生全新驾驶场景。此外，针对数据中大量“平淡无奇”的普通路况，系统采用机器学习探测技术主动对网络进行扰动测试，精准定位模型的薄弱环节，并在其失效边缘集中合成更多边缘案例，构建主动防御的闭环。更为强大的是基于真实故障的“场景重现与再推演”机制。当真实 Autopilot 车辆在路上遭遇复杂边缘工况或表现不佳时，该视频剪辑会被送入自动标注管线进行 3D 场景与动态物体的高精度重建，进而在模拟器中完全克隆出该物理事件。在此虚拟孪生世界中，Autopilot 能够执行全新的动作策略，推演全新的博弈轨迹。在展示中，Ashok 还特别分享了前一晚刚刚跑通的神经渲染增强管线——通过对合成场景施加神经渲染，画面的质感与车载物理相机实拍几乎难以分辨。Ashok 强调了在车端落地仿真数据的规模，并总结道：“在模拟器中完全重新创建这些场景……这非常不可思议，因为我们绝不希望 Autopilot 再次在同一个地方跌倒，一旦失败，我们就把它捕获并以此作为基准持续守住这条底线。” ```mermaid flowchart TD accTitle: Real-to-Sim Failure Recreation Loop accDescr: 真实故障片段通过自动标注重建为3D仿真并闭环重演的流程图 subgraph Capture["现实捕获"] A["车端真实失效视频剪辑"] end subgraph Reconstruction["自动重建与增强"] B["自动标注管线"] C["3D 场景与动态物体几何重建"] D["神经渲染技术 (Neural Rendering)"] end subgraph SimEngine["模拟与闭环测试"] E["虚拟孪生场景生成"] F["Autopilot 闭环再推演与新策略验证"] G["纳入持续回归测试基准集"] end A --> B B --> C C --> E E --> D D --> E E --> F F --> G ``` - **海量场景生成**：拥有 2000 英里手工数字孪生道路，同时依靠程序化算法无上限生成任意曲率、植被与障碍物组合。 - **闭环主动对抗**：利用机器学习对网络模型施加扰动，自动化探寻失效点并围绕失效工况定向繁衍数据。 - **真实失效克隆**：真实长尾故障录像可自动转为 3D 仿真环境，支持修改主车轨迹并探索“平行世界”下的决策结果。 - **已部署仿真数据规模**：车端训练已融入 3 亿张模拟图像与近 5 亿个合成标签。 --- ### [01:42:09](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6129s) 纯视觉迁移的算力消耗、AI 编译器与车载双芯片架构接替 Ashok 上台的 Milan Kovac 负责车载神经网络集成以及模型训练与评估的基础设施。他首先回顾了几个月前特斯拉全面移除毫米波雷达（Pure Vision push）工程背后的恐怖算力支撑：为了剔除对毫米波雷达的依赖，团队在 250 万个视频片段中生成了超过 100 亿个自动真值标签，背后动用了数千张 GPU、近 20,000 个 CPU 核心，以及串联在模拟回路中的 2,000 多台真实的 FSD 硬件车载电脑——而这仅仅是特斯拉内部规模最小的算力集群。将复杂的 PyTorch 深度学习模型搬上车载硬件，核心挑战在于极端苛刻的延迟（Latency）与帧率（Frame Rate）要求，这直接决定了对周围环境加速度与速度估算的准确度。特斯拉完全自主开发并扩展了专用 AI 编译器，负责将上层算子高效映射至加速硬件单元，并在严苛的片上 SRAM 容量限制下，排布出吞吐量最优的计算指令流水线。在硬件冗余与计算利用上，FSD 电脑配备了两套计算引擎：在任一时刻，仅有一台引擎向车辆底盘输出底层控制指令，而另一台引擎则作为额外的算力扩展节点运行；两台芯片在硬件与软件层面完全对等且可随时无缝互换角色。 - **雷达剔除工程计算规模**：250 万剪辑生成超 100 亿标签，动用数千 GPU、近 2 万 CPU 核心及 2000+ 真实 FSD 硬件在环。 - **核心工程瓶颈**：极低延迟与极高帧率硬指标；自研 AI 编译器负责在受限片上 SRAM 约束下实现算子调度与吞吐最大化。 - **车载双芯片协同机制**：主从引擎架构，单引擎下发底盘控制，副引擎补充运算，支持软硬件全对称互换容灾。 - **每周百万次自动化评估**：专门构建由 3,000 多台 FSD 电脑组成的独立测试集群，针对团队提交的任何代码改动，每周执行超过 100 万次模型评估。 - **训练集群算力规模**：训练侧部署了接近 10,000 块 GPU，Milan 指出仅从 GPU 数量维度来看，该规模就已经超越了当时全球公开排名前五的超级计算机。 --- ### [01:45:34](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6334s) Dojo 项目启航：破解算力扩展中的带宽与延迟瓶颈尽管手握近万块顶级 GPU，面对 Andrej 团队规划的更大、更复杂的视频空间-时间神经网络模型，算力缺口依然巨大。负责 Project Dojo 的特斯拉硬件高管 Ganesh Venkataramanan 登台，揭开了这项旨在彻底重构 AI 训练计算机底层逻辑的宏大计划。该项目始于数年前 Elon Musk 的高瞻远瞩，当时 Musk 预见到了模型规模指数级爆发的趋势，要求团队自研一台极致高速且具备超高能效比的专用 AI 训练计算机。 Ganesh 剖析了传统分布式训练计算机面临的底层物理瓶颈。无论是基于 CPU、GPU 还是专用加速器构建的分布式架构，本质上都是通过某种网络拓扑（例如二维网格网络）将分布在机箱内的计算单元与本地内存互联。在过去的技术演进中，业界往往能轻易通过堆砌芯片来扩展纯浮点运算能力，但真正的系统性瓶颈在于数据流通。正如 Ganesh 所总结的那样：“在这类架构中存在一个普遍趋势：扩展算力很容易，扩展带宽非常困难，而降低延迟更是极其困难。” Project Dojo 的立项正是为了打破这一带宽与延迟的桎梏，为更庞大的模型训练铺平道路。 - **立项初衷**：响应 Elon Musk 对训练算力瓶颈的前瞻预判，目标是实现业内最高 AI 训练性能、极佳功耗比与成本效益。 - **架构核心挑战**：分布式系统中“算力扩展易、带宽扩展难、延迟降低难”的底层物理困局。 - **Dojo 愿景**：从芯片到系统级全面重构，打破传统 GPU 集群互联通信壁垒，直接支撑百亿级参数的时空模型吞吐。 ### [01:47:10](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6430s) Dojo 的设计哲学与训练节点（Training Node）特斯拉团队在构建超级计算机 Dojo 时，设定了明确的设计目标：突破传统计算架构的瓶颈。团队设想了一个大型“计算平面”（Compute Plane），内部密布鲁棒的计算单元，由庞大的内存池支持，并通过极高带宽、超低延迟的 2D 网格（2D Mesh）网络互连。针对超大规模神经网络，系统在架构上提取模型并行、图并行和数据并行；专门研发的神经编译器（Neural Compiler）则充分挖掘空间与时间局部性，将通信范围限制在局部区域，从而压缩全局通信开销，确保带宽利用率随着计算平面的扩大持续扩展。 ```mermaid flowchart TD accTitle: Dojo 计算节点与平面映射逻辑 accDescr: 从最小训练节点到互连计算平面的设计层次示意图 subgraph NodeArch["训练节点 (Training Node)"] A["64-bit 超标量 CPU (4-wide 标量 / 2-wide 向量)"] B["矩阵乘法单元 (Matrix Multiply) SIMD 向量管线"] C["1.25 MB 快速 ECC SRAM"] D["极宽互连导线 (2GHz+ 单周期物理边界)"] A --- B B --- C C --- D end subgraph PlaneLevel["计算平面与编译器"] E["2D Mesh 高带宽低延迟互连"] F["神经编译器 (时空局部性映射)"] NodeArch --> E E --> F end ``` 为了彻底消除各层瓶颈，团队采取自内向外（Inside-Out）的路线，从最基层的微芯片出发。Dojo 的最小计算实体称为“训练节点”（Training Node）。选择其物理尺寸至关重要：如果选得太小，运算固然极快，但同步和软件的额外开销会占据主导；如果选得过大，硬件物理实现的复杂度会激增，最终被内存瓶颈拖垮。因此，团队将主频设在 2GHz 以上，以单时钟周期内信号所能传输的最远距离画出一个方框，把方框边缘密密麻麻布满通信连线，随后在内部植入机器学习计算核心、大容量 SRAM 和可编程核心。演讲者介绍道：“This might be our smallest entity of scale but it packs a big punch: more than one teraflop of compute in our smallest entity of scale.” - **支持数据精度**：FP32、BFLOAT16，以及专为 Dojo 打造的全新格式 CFP8（Configurable FP8）。 - **节点内存与算力**：单个训练节点配备 1.25 MB 带 ECC 校验的超快 SRAM，单节点算力超过 1 TFLOP。 - **架构特性**：64 位超标量 CPU，拥有 4-wide 标量管道和 2-wide 向量管道，支持 4 路超线程（4-way multithreading），可实现计算与数据传输完全重叠。 - **专有 ISA 优化**：定制指令集专门面向机器学习，原生支持转置（Transpose）、收集（Gather）、链接遍历（Link traversals）和广播（Broadcast）等操作。 --- ### [01:51:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6699s) D1 芯片诞生：从网格阵列到定制封装通过高度模块化的设计，特斯拉将 354 个训练节点以 2D 网格无缝拼装，构建出一个芯片内部的完整计算阵列。该阵列不仅能提供极其澎湃的机器学习算力，各节点之间也保持着极高带宽的结构通信。为了让算力不受对外输入输出的制约，团队在计算阵列四周环绕了 576 个高速低功耗 SerDes 通道。演讲者强调指出，其芯片出向 I/O 带宽是当今网络交换机黄金标准芯片的两倍以上。这颗完全由特斯拉团队从微架构、物理设计一直做到 GDS 出图和封装的芯片，被正式命名为 D1。D1 采用 7nm 先进制程，在 645 平方毫米的紧凑面积内集成了 500 亿个晶体管。整块芯片上没有任何“暗硅”（Dark Silicon），也不做任何向后兼容的历史包袱，100% 的硅片面积均服务于机器学习训练与数据传输。演讲者用一句话精准概括了这颗芯片的定位：“This chip is like a GPU level compute with a CPU level flexibility and twice the network chip level I/O bandwidth.” - **芯片工艺与规格**：7nm 制程，集成 500 亿晶体管，芯片面积为 645 mm²。 - **封装形式**：倒装芯片 BGA 封装（Flip-Chip BGA）。 - **核心构成**：单颗 D1 包含 354 个训练节点，环绕 576 个超高速 SerDes 通道。 --- ### [01:54:06](https://www.youtube.com/watch?v=j0z4FweCy4M&t=6846s) 突破物理极限：训练瓦片（Training Tile）与垂直供电将芯片组成超大规模集群时，D1 之间的直连无需任何外部胶水逻辑（Glue Logic）。为了构建容纳约 50 万个训练节点、包含 1500 颗 D1 的计算平面，外围设计了 Dojo 接口处理器（Dojo Interface Processors）。接口处理器一面通过高带宽织网与 D1 计算阵列相连，另一面通过 PCIe Gen4 与数据中心常规主机桥接，同时为整个计算阵列提供高带宽共享 DRAM 以及高基数网络扩展（Higher Radix Network Connection）。为了保持芯片间不损失任何带宽，特斯拉摒弃了常规做法，开发出跨维度的超大封装形态——“训练瓦片”（Training Tile）。一个训练瓦片通过扇出型晶圆级工艺（Fan-out Wafer Process），将 25 颗经过筛选的优质（Known Good Die）D1 裸片严密整合在一起，并定制了超高密度的高带宽连接器。这也是芯片行业迄今为止最大的有机多芯片模块（MCM）。设计过程中业界现有的 EDA 软件和计算集群频繁崩溃崩溃，特斯拉工程师不得不发明全套全新的仿真和布线方法才得以将其实现。 ```mermaid flowchart LR accTitle: 训练瓦片系统集成与物理分层 accDescr: 展现从 25 颗 D1 裸晶、定制 VRM 到整块 9 PFLOP 训练瓦片的垂直层叠架构 subgraph Thermal["散热层"] Cooling["集成式液冷 / 机械构件"] end subgraph Power["垂直供电层"] VRM["定制电压调节模块 (VRM) (52V DC 输入，直接回流焊至晶圆)"] end subgraph Silicon["计算核心层"] Dies["25 颗 D1 裸晶 (扇出型晶圆工艺) 无损带宽互连"] end Thermal --- Power Power --- Silicon ``` 更为棘手的是如何为这样一块密集瓦片输送巨大电能。传统横向走线早已不可行，特斯拉创新性地实现了垂直供电：定制了专用的电压调节模块（VRM），直接回流焊在扇出晶圆表面，将芯片封装与 PCB 回流焊工艺融合在了一起。整块瓦片直接输入 52V 直流电，计算平面与供电和散热系统相互正交。演讲者现场举起刚下线不久的真实硬件，激动地感慨：“I can't believe I'm holding nine petaflops out here.” - **瓦片算力**：单块训练瓦片（Training Tile）提供高达 9 PFLOPs 的峰值算力。 - **验证进展**：就在 AI Day 举办前一周，团队成功点亮了首个功能完备的训练瓦片，在有限风冷的台面试验台上成功跑通了 Andrej Karpathy 编写的精简版 GPT-2 模型（minGPT-2）。 --- ### [02:00:12](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7212s) ExaPOD 诞生：1.1 ExaFLOP 的算力集群在训练瓦片之上，模块化扩展进一步展现出威力。一个托盘（Tray）容纳由 2×3 块训练瓦片构成的训练矩阵（Training Matrix），两个托盘装入一个机柜，即可组成单机柜 100 PFLOPs 的强大系统。特斯拉甚至打破了机柜之间的传统边界，通过自研的无损互连将所有瓦片贯穿互通，消除了机柜间常见的带宽骤降（Bandwidth Divert / Bandwidth Cliffs）。将 10 个机柜组合在一起，就构成了 Dojo 的最终形态——ExaPOD。这套系统汇聚了超过 100 万个精心调优的训练节点，提供了整整 1.1 ExaFLOP 的统一算力，将分布式机器学习集群的密度和互连带宽推向了全新高度。 - **托盘与机柜配置**：单个托盘集成 6 块瓦片（2×3 矩阵）；单机柜配备 2 个托盘，提供 100 PFLOPs 算力。 - **ExaPOD 规模**：由 10 个机柜无损直连组成，包含超过 100 万个计算训练节点，实现 1.1 ExaFLOP 的 FP8 计算能力。 ### [02:00:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7228s) Dojo 软件栈与编译器架构：突破算力扩展的壁垒在 Dojo 硬件集群中，整套网络拥有超过 100 万个节点，且具备全域均匀一致的高带宽。然而，仅靠硬件无法完全解决集群算力的扩展问题，软件层面的设计同样至关重要。Dojo 的计算平面被设计为可高度自由分区的架构，其基础分区单元被称为 Dojo 处理单元（DPU，Dojo Processing Unit）。一个 DPU 由一颗或多颗 D1 芯片、接口处理器（Interface Processor）以及一台或多台主机（Host）组成。DPU 能够根据运行在其上的神经网络算法规模灵活扩容或缩容。对机器学习算法工程师而言，这一层复杂的硬件被软件彻底抽象化：用户无需费力重写代码，仅需对其原有的脚本进行极少量的改动。 ```mermaid flowchart TD accTitle: Dojo Software Stack and Compilation Flow accDescr: 展现从 PyTorch 模型定义到硬件可执行二进制的完整 Dojo 编译与执行管线。 subgraph UserInterface["用户与框架层"] A["PyTorch 扩展接口 (标准 ML 接口)"] end subgraph DojoCompiler["Dojo 编译器套件"] B["计算图提取与动态控制流处理 (Loop / If-Else)"] C["并行度映射引擎 (数据 / 模型 / 图并行)"] D["内存足迹优化与动态代码生成"] E["LLVM 后端二进制生成"] end subgraph RuntimeAndHardware["运行时驱动与硬件分区"] F["多主机 / 多分区驱动栈 (DPU 调度)"] G["D1 芯片与训练晶圆阵列 (Training Tiles)"] end A --> B B --> C C --> D D --> E E --> F F --> G ``` 支撑这种极低迁移成本的核心是 Dojo 的编译器套件。该编译器负责细粒度并行化以及将复杂的神经网络高效映射到计算平面上。由于具备极高的互连织构带宽，原本在传统架构中受限于单芯片边界的模型并行（Model Parallelism），在 Dojo 上被顺畅扩展到了整块训练晶圆（Training Tile）乃至更大规模的集群。编译器通过数据并行、模型并行与计算图并行等多种技术协同提取算力，并在低 Batch Size 场景下高效映射大型网络以释放极致利用率。此外，编译器还内置了对循环（Loops）、条件判断（If-then-else）等高级动态控制流的原生支持。整个软件栈包括无缝对接机器学习工作流的 PyTorch 扩展、能够动态即时生成重用代码的编译器核心、将代码直接下发为机器码的 LLVM 后端、管理多主机和多分区的底层驱动栈，以及配套的性能分析仪（Profiler）和调试工具（Debugger）。这套全自研全栈工具链彻底摆脱了对手写特定算子内核的依赖。 - 集群硬件规模：全系统包含超 1,000,000 个计算核心节点，并维持均匀统一的网络带宽。 - 零手工优化内核：依靠底层 LLVM 编译器后端自动生成高度优化的机器码，“without relying on even single line of handwritten kernel”。 - 宏观性能对比指标：在同等建设成本下，Dojo 带来整整 4 倍的 AI 训练性能提升；能效比达到 1.3 倍（更省电）；而物理占地空间足足缩小了 5 倍。 - 演进规划：团队已在着手第一批机柜的装配测试，并已经着手规划下一代芯片与系统架构，目标是从硅片到系统层面再次实现整整 10 倍（10x）的性能跨越。 --- ### [02:05:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7514s) 特斯拉的人形机器人野望：Tesla Bot 正式公布演讲随后迎来戏剧性转折，一名身着黑白紧身科技服的演员登台跳了一段机械舞，随后埃隆·马斯克（Elon Musk）登台收场并笑着解释道：“Unlike Dojo, obviously that was not real. Dojo is real; the Tesla Bot will be real.”（“显然刚才那段和 Dojo 不同，那不是真的。但 Dojo 是真实的，Tesla Bot 也将会是真实的。”）。马斯克直言，从本质上讲，特斯拉可以说是全球最大的机器人公司，因为特斯拉汽车本质上就是“行驶在轮子上的半感知机器人”（semi-sentient robots on wheels）。 ```mermaid flowchart LR accTitle: Tesla Autopilot Tech Transfer to Tesla Bot accDescr: 特斯拉汽车自动驾驶软硬件技术向人形机器人复用的映射流程图。 subgraph CarTech["特斯拉汽车现有核心技术"] FSD["FSD 芯片 (推理引擎)"] Cams["8 颗 Autopilot 摄像头"] NN["空间与语义世界感知神经网络"] Pack["电池包与动力执行机构设计"] end subgraph BotForm["Tesla Bot 人形形态集成"] Head["面部屏幕显示交互"] Torso["机载 FSD 计算机与轻量骨架"] Joints["40 个机电执行器关节"] Speed["机械限速 5 mph / 轻量级自重"] end CarTech -->|"技术全栈复用"| BotForm ``` 特斯拉在汽车上积累的推理芯片（FSD 计算机）、自动驾驶神经网络（用于感知周围物理世界、进行空间语义理解并规划导航路径）、精密传感器、高能量密度电池以及伺服驱动执行器等技术，完全可以水到渠成地移植到一个人形物理载体中。马斯克宣布，特斯拉计划在明年（2022 年）某个时候推出首款人形机器人的原型机。机器人的目标是消除危险、重复和枯燥的任务，在专为人类打造的物理社会环境中自由通行并协助人类。为了消除大众对科幻电影中 AI 觉醒反叛人类的恐惧，特斯拉从物理和机械层面做出了明确限制：它的最高移动速度被严格设定为大约每小时 5 英里，自重和力量也做了相应克制，确保人类能够轻松跑过它并在必要时能够物理制服它。 - 身高与外形规格：身高约为 5 英尺 8 英寸（约 173 厘米），面部设计有专属信息屏幕用于显示交互与状态信息。 - 感知与计算中枢：头部及身体各方位搭载 8 颗 Autopilot 同款摄像头，并在躯干内集成 FSD 芯片作为主控推理引擎。 - 运动与机械指标：设定行走速度上限为 5 英里/小时（约 8 公里/小时），整机轻量化设计，确保人类在物理层面具备绝对安全控制优势。 - 终极交互目标：无需编写代码或预先指定每一条动作指令，机器人仅凭人类自然语言语音输入即可理解高维意图并执行复杂物理操作（例如：“请拿起那颗螺栓，用那把扳手把它固定在车上”，或是“请去超市帮我买这些生活杂货”）。 --- ### [02:10:04](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7804s) 劳动力的无限供应：宏观经济学范式转移与全场 Q&A 开启马斯克从哲学与宏观经济学层面对 Tesla Bot 的深远影响展开了推演。他指出，人类经济在最底层的物理本质就是“劳动力”（labor），而资本与设备在本质上不过是凝结的劳动力（distilled labor）。如果人形机器人能够让人类彻底告别劳动力短缺，“人类社会经济的总量增长是否还存在任何实际边界？”马斯克反问并给出了倾向于否定的回答。当体力劳动在未来彻底变成一种自由的选择——你想做就做，不想做就不必做——整个人类经济结构将迎来翻天覆地的剧变。正因如此，马斯克预测从长远来看，全球势必需要推行“全民基本收入”（Universal Basic Income, UBI），尽管他幽默地补充道：“当然现在还不着急推行，因为这台机器人目前还干不了活，我们还需要一点时间开发。”随后，马斯克向台下发出诚挚邀请，欢迎各路顶尖软硬件工程人才加入特斯拉团队，并邀请早前主讲的技术骨干集体返台，拉开全场问答环节（Q&A）的序幕。在问答开始的第一个问题中，有现场观众询问特斯拉未来是否有计划将今天的技术研究成果公开发表论文或彻底开源。马斯克十分坦诚地指出，开发这样一套极其庞大的软硬件一体化超级系统需要耗费难以想象的资金：“It is fundamentally extremely expensive to create the system so somehow that has to be paid for. I'm not sure how to pay for it if it's fully open sourced.”（“打造这套系统的成本极其高昂，这些成本必须设法收回；如果将其完全开源，我不知道该怎么赚回研发费用，除非大家都愿意免费来打工。”）不过马斯克话锋一转，郑重表态：特斯拉的核心技术绝不局限于自家车辆，如果其他汽车制造商希望获得 Autopilot 或 FSD 技术的授权并搭载在他们的量产车上，特斯拉持高度欢迎和完全开放的态度。随后，一位深耕深度学习加速芯片领域的研究人员接过了麦克风，直切核心软硬件难点：Dojo 是否真正解决了跨数十万乃至上百万节点的大规模分布式编译器问题，还是说这套机制仅仅适配 Dojo 自身的特殊架构？特斯拉硬件设计负责人 Bill Chang 拿过麦克风直面挑战，用一连串清晰有力的设问与解答回应了业界疑虑：“Have we solved the problem? Not yet. Are we confident we will solve the problem? Yes.”（“我们彻底解决这个问题了吗？还没有。我们有信心能够解决吗？是的。”）。Bill Chang 阐述道，特斯拉目前已在工程样机硬件上成功跑通了神经网络，并建立了扎实的性能仿真模型来验证线性扩展能力。其中的技术攻坚核心在于“保持数据的局部性”（locality）——只要能通过精密的模型并行与数据并行切分，将绝大部分数据和计算保留在本地节点的 SRAM 内，参数在管道内顺畅流动，Dojo 集群就能够无限平滑扩展，特斯拉在自身实际的大规模 AI 业务演进中已经看清了一条通往最终目标的清晰路径。 ### [02:15:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8128s) Dojo 的分布式计算挑战与成败标准在针对 Dojo 硬件架构的交流中，一位提问者指出，模型训练本质上是一种“软扩展”（soft scaling）的应用场景。尽管系统拥有极高的算力和超大带宽互连，但如果计算受限于不同物理位置上的有限局部内存，依然可能遭遇性能瓶颈。提问者特别好奇 Tesla 是如何解决这种局部性与通信开销问题的，并询问相关技术是否有开源的可能。 Elon Musk 回应称“事实胜于雄辩”（the proof's in the pudding），并透露 Dojo 预计将在明年正式投入运行。Dojo 最核心、最首要的应用场景是处理并训练海量的视频数据。Musk 阐述了技术创新的本质公式：创新取决于迭代次数以及每次迭代取得的平均进展。如果能把原本需要几天才能完成的模型训练缩短到几个小时，整个技术迭代的演进速率就会发生质的飞跃。对于 Dojo 研发团队的考核标准，Musk 给出了一个极其务实且非黑即白的判决准绳：“它的试金石，正如我告诉 Dojo 团队的那样：如果软件团队主动想要关掉 GPU 集群，那它就成功了；但如果他们依然想留着 GPU 集群，那它就是不成功的。”（“the acid test here and you know what I've told the Dojo team is like it's successful if the software team wants to turn off the GPU cluster, but if they want to keep the GPU cluster on, it's not successful.”） ```mermaid flowchart LR accTitle: Dojo Success Metric Flowchart accDescr: Visualizes how Dojo training efficiency drives software adoption to determine project success. subgraph IterationSpeed["迭代创新闭环"] A["海量车载视频"] --> B["Dojo 极速训练"] B --> C["训练时间：数天缩减为数小时"] C --> D["快速算法迭代"] end subgraph AcidTest["最终检验标准 (Acid Test)"] D --> E{"软件团队选择"} E -->|"关掉 GPU 集群"| F["Dojo 研发成功"] E -->|"继续依赖 GPU"| G["Dojo 未达预期"] end ``` ### [02:17:38](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8258s) 仿真系统拓展与 Tesla Bot 的内部代号现场观众对逼真的 Autopilot 仿真演示给予了高度评价，并询问仿真系统未来是否有计划拓展到公司内部的其他业务模块。 Autopilot 仿真团队负责人 Ian Glow 出面解答。他指出，随着全面自动驾驶（FSD）的推进，仿真系统必须覆盖车辆越来越多的子系统。目前系统主要在模拟车辆动力学，但未来将进一步整合电池管理系统（BMS）、微控制器单元（MCU）乃至整车的每一个零部件。这种深度的全车软硬件集成，使得 Autopilot 仿真平台在外溢到其他部门时具备极大的通用价值。Glow 表示，长远来看目标是将其打造为一个“通用仿真平台”（universal simulation platform）；但在那之前，团队近期的大量精力将用于搭建针对“Optimus”的支持体系。此外，未来也不排除探索将内部构建的仿真基础设施和酷炫工具开放给外部人员使用的可能性。在此过程中，Elon Musk 顺势揭晓了一个内部机密：“Optimus 是 Tesla Bot 的代号。”（“Optimus is the code name for the Tesla Bot.”）现场随后响起笑声，有人打趣补充道“Optimus Prime（擎天柱）”。 ### [02:18:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8336s) 多模态感知、车内交互与自动驾驶的“第一法则” 来自观众席的 Ali Jahanian 提出了关于多模态融合的思考。他认为汽车本身已经是一个轮式机器人，现在走向人形机器人非常自然；在演示中看到了行人带着狗跑步等复杂场景，符号处理与自然语言模型（例如 Andrej Karpathy 之前开源的 miniGPT）有望与视觉形成互补，因此想了解多模态在车端与仿真中的应用。 Elon Musk 阐明了自动驾驶的物理本质：人类驾驶汽车在生物学层面上依靠的是眼球接收光线并由生物视觉神经网络进行推理，而 Tesla 的技术路径正是使用硅基摄像头加人工神经网络来复现这套生物系统，因此“驾驶在根本上几乎完全是视觉神经网络”（“driving is fundamentally basically almost entirely vision neural nets”）。不过，Musk 也确认系统会融合一定程度的音频感知： - 车外音频：用于及时听辨救护车、消防车、警车等应急特种车辆的警笛声。 - 交互与语义：识别车外是否有人在对着车辆大喊大叫，并理解其语义，或者用于车内乘客的语音交互。当另一位观众问及 FSD 训练数据目前主要来自美国，该模型将如何泛化扩展到全球其他道路体系各异的国家时，Musk 与团队解释道，Tesla 实际上已经在使用来自约 50 个不同国家的数据进行训练。但研发攻坚必须集中精力，因此选择先彻底攻关 Tesla 所在的大本营——美国市场。连加拿大暂时都被放在次要梯队，因为其道路细节也与美国存在微妙差异。先在一个国家解决最硬核的问题，再外推到全球，可以避免引入不必要的初始复杂度。团队成员指出，计算机视觉的核心特征实际上是超越国界的：每个国家都有道路、路沿和车辆，底层模型具有极强的通用性。Musk 随即强调了系统的根本红线：“第一法则就是别撞车，对吧？这对每个国家都成立。是的，这就是最高指令……不管前面是什么，都别撞上去，哪怕那是一架坠毁在高速公路上的 UFO，也绝不能撞上去。你不需要非得认出它是 UFO 才能做到不撞它。”（“the prime directive is don't crash, right? And that's true for every country... whatever it is, don't hit it, even if it's a UFO that crash-landed on the highway, and still don't hit it. You should not need to recognize it in order to not hit it.”） ### [02:23:20](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8600s) 多视几何重构精度与厘米级定位一位提问者关注光度学处理（Photometric process）与多视几何（Multi-view Geometry）重建时的误差范围，询问合成几何体与现实真实几何之间的误差是在毫米级还是厘米级，以及是否结合其他模态传感器来降低误差。 Tesla 工程师明确回应，在空间几何重建中，合成模型与实际几何的标准差通常在 2 到 4 厘米（“within a couple centimeters, three or four centimeters, that's the standard deviation”）。团队的核心原则是寻找高可扩展性的自动化标注方案；在特定测试阶段可能会引入其他传感器用于地面真值（Ground Truth）标定与对齐（benchmark），但在日常生产级系统中，主要依靠纯摄像头管线。 Elon Musk 补充分析了空间精度对自动驾驶决策的实际意义： - 系统的定位精度目标应维持在厘米级水准。 - 空间精度的需求与距离高度相关：对于车辆近处的障碍物和结构，测量精度必须极高；对于远处的物体，误差容忍度相对更大，因为远距离目标不会立即触发规避动作。随着车辆与物体的距离缩短，感知系统测算出的几何精度会自动逐步收敛提升。 ### [02:24:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8679s) 制造自动化与 Tesla Bot 的研发初衷有观众将话题引向 AI 在智能制造中的应用，提及 Musk 过去提出的“外星战舰”（Alien Dreadnought，即极致全自动化工厂）概念，并询问台上展示的人形机器人是否意味着在经历了“产能地狱”（production hell）以及认识到“人类被低估”之后，Tesla 试图用人形机器人来解决车间装配问题。 Musk 澄清说，外界有时会把他随口阐述的观点推向极端。如果实地走访 Tesla 的生产线，会发现不同环节的自动化程度跨度极大：某些工段已经实现了近乎 100% 的全自动化，而个别精细工序则几乎完全依赖人工。但从整体体量来看，工厂的绝大多数制造环节已经实现了高度自动化。 Musk 特别指出，制造环节人工作业的减少主要是通过产品架构和设计工程的变革实现的，而非依靠机器人生搬硬套模仿人类： - 采用超大型高压铝合金压铸技术（一体化压铸）：Tesla 已经成功将车辆后底板的三分之一车身结构一体成型压铸为单一零件。 - 下一步计划是将车身前底板的三分之一也整合为单个大型压铸件。 - 这一架构改进使原有的车身焊装生产线规模大幅削减了 60% 至 70%。针对提问者的核心假设，Musk 明确给出了否定答复：Tesla Bot 人形机器人的立项和研发，“并不是由制造环节的具体需求所驱动的”（“the robot is not prompted by specifically by manufacturing”）。 ### [02:26:08](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8768s) 研发人形机器人的必要性与量产制造的挑战问答环节开始时，埃隆·马斯克（Elon Musk）探讨了特斯拉为何决定制造人形机器人（Tesla Bot / Optimus）。特斯拉并不是为了制造机器人而刻意拼凑技术，而是因为公司在研发自动驾驶的过程中，恰好制造出了实用人形机器人所需的所有核心模块。因此，制造它成了一种水到渠成的自然选择。马斯克直言：“I guess we probably should make it, and if we don't, someone else would well, and so I guess we should make it and make sure it's safe.”（我想我们大概应该制造它，如果我们不做，别人也会做，所以我想我们应该去做，并确保它的安全性。）除了软件和智能，马斯克特别强调了规模化制造（Volume Manufacturing）的关键地位。大规模量产不仅极度困难，而且在业内被严重低估，而特斯拉在过去几年中已经磨练出了极强的制造能力。对于人形机器人而言，制造能力决定了它的最终成本——只有具备高效的制造工程，才能让人形机器人避免沦为造价高昂的实验室玩具，真正走向大众市场普及。 ### [02:26:50](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8810s) Dojo 的散热与电力传输架构：从车载动力总成汲取灵感现场观众向工程团队提问关于 Dojo 超级计算机的物理扩展性问题，特别是计算节点在热力学极限（散热）与电力输送方面的挑战，想了解特斯拉计划如何在机架级别乃至多数据中心之间进行水平扩展。Dojo 工程师比尔（Bill）回应称，Dojo 在设计之初就贯彻了高度模块化的理念。Dojo 的训练瓦片（Compute Tile）采用了上下双面全液冷设计，只要将冷却管路和电力连接器“卡扣”固定好，就能直接实现完整的供电和冷却循环，而这套高集成的单元体积甚至不足一立方英尺。 ```mermaid flowchart TD accTitle: Dojo Compute Tile Thermal and Power Architecture accDescr: 展现特斯拉如何将电动车的三电技术迁移到 Dojo 训练瓦片的超高密度散热与电力设计。 subgraph TechTransfer["技术迁移"] PT["车载动力总成工程 (Powertrain Expertise)"] CL["先进液冷技术 (Advanced Cooling)"] end subgraph DojoTile["Dojo 计算瓦片 (小于 1 立方英尺)"] TopCool["顶层液冷管路 (Liquid Cooled)"] Core["核心计算单元 (ASIC) 运行电压: < 1V / 极高电流"] BottomCool["底层液冷管路 (Liquid Cooled)"] end PT --> Core CL --> TopCool CL --> BottomCool TopCool --- Core BottomCool --- Core ``` 马斯克随后补充，特斯拉在车载功率电子和散热系统上拥有深厚的技术积累。团队直接把电动汽车动力总成（Powertrain）中的功率电子技术以及顶尖的冷却方案搬到了这台超级计算机上。超算算力往往受限于散热极限，马斯克幽默地指出了低电压架构下的物理定律挑战：“$I^2R$ is what really bites you on the ass.”（$I^2R$ 焦耳热才是真正让你吃尽苦头的东西。）在核心计算单元上，芯片以低于 1 伏的极低电压运行，这意味着必须传输极高的电流（Amps），从而产生巨大的欧姆热，高效散热因此成为了系统工程的重中之重。 - Dojo 算力瓦片（Compute Tile）体积小于 1 立方英尺，集成了双面全液冷散热。 - 芯片运行电压不足 1V，面临高电流带来的严重 $I^2R$ 阻抗生热挑战。 - 特斯拉将汽车动力总成和高压功率电子学经验直接迁移到超算基础设施中。 ### [02:28:58](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8938s) Dojo 算力扩展的定位：专用 ASIC 神经网络训练机现场提问转向算法和数据规模的扩展性：随着训练硬件性能的大幅提升，模型规模和训练数据量势必呈指数级增长，这是否会导致模型过于复杂，从而在汽车端侧推理芯片（FSD 芯片）上运行成本过高？另外，Dojo 是否也会直接应用到车端自动驾驶系统？特斯拉自动驾驶团队的一位工程师解释称，由安德烈·卡帕斯（Andrej Karpathy）主导的模型不仅服务于车载终端，还涵盖了庞大的自动数据标注（Auto-labeling）和离线清洗模型；这些复杂模型虽然不直接部署在车上，但产出的结果会反哺车端软件栈，Dojo 将全力加速这些任务。马斯克随后阐述了 Dojo 的硬件战略定位。虽然 Dojo 的首要任务是吞吐海量全景视频数据，训练出供车端推理引擎运行的网络（这也是检验 Dojo 是否比传统 GPU 集群更优秀的试金石），但从本质上讲，Dojo 是一台通用的神经网络训练计算机。马斯克一针见血地指出通用芯片的妥协性：“GPUs are still essentially running neural net training in emulation mode.”（GPU 本质上仍是在以仿真模式运行神经网络训练。）无论 CPU 还是 GPU，其底层指令集和设计初衷都不是为神经网络而生的；相比之下，Dojo 彻底走向专用集成电路（ASIC）路线，全身心服务于神经网络训练这单一目标。根据系统设计的一般法则，专为特定目的量身定制的专用系统，在效率上必然会超越通用架构。 ### [02:31:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9074s) 端到端（End-to-End）自动驾驶演进与混合规划架构现场观众针对此前展示的“感知视觉网络”与“规划控制（Planning & Control）”系统提问：Dojo 是否支持跨越这两个边界训练统一的网络？如果训练出端到端网络，现有车端 FSD 芯片的计算能力能否在严格的毫秒级低延迟要求下完成推理？团队工程师肯定地表示，规划网络完全可以在 Dojo 或 GPU 上训练，模型与底层训练平台无关。更重要的是，端到端架构一旦打通，系统可以直接省去大量对中间状态的显式解码与序列化操作，仅解码驱动车辆所必需的关键输出，执行速度反而会比割裂的传统分段架构更快。安德烈·卡帕斯对端到端的设计理念表示高度赞同：“Certainly, end-to-endness is the guiding principle behind a lot of the network developments, and over time in the stack, neural networks have taken on more and more functionality.”（毫无疑问，端到端是众多网络开发背后的指导原则，随着时间推移，神经网络在整个软件栈中承担了越来越多的功能。）目前特斯拉系统在感知与规划之间以向量空间（Vector Space）作为显式接口，但这只是渐进式工程开发的过渡形态，底层并没有任何技术障碍阻碍团队最终抽取特征进行端到端微调。随后的讨论进一步深入到规划层架构： - 规划算法面临模型预测控制（MPC）和快速求解凸优化（Convex Optimization）的数学需求。 - 团队的探索核心在于将搜索和优化结构嵌入到神经网络拓扑中（正如在感知端利用 Spatial RNN 沉淀时空信息一样），一旦网络结构原生融入优化搜索，规划网络便能拥有媲美甚至超越 C++ 算法的推理速度。 - 未来的技术栈将是“混合系统（Hybrid Stack）”：人类已知明确物理机制的领域保留物理模型，复杂难以解析的领域交由神经网络，两者共同嵌入前向传播（Forward Pass）中。马斯克从系统全局给出了通俗的类比：只要环视视频神经网络能够精准理解环境并输出向量空间，整个外部世界在车机内部就已经被转换成了一个“电子游戏”（比如《侠盗猎车手》（Grand Theft Auto））。在游戏引擎里，即便完全用传统的 C++ 规划与控制代码，也能指挥 NPC 车辆和行人在不发生碰撞的情况下平稳行进。马斯克总结道，虽然终极目标下神经网络做端到端规划可能更优，但他坚信，哪怕仅靠“神经网络感知 + 纯 C++ 规划控制”，特斯拉也完全能够实现远超人类驾驶水平的安全性。 ### [02:34:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9296s) 制造业中克制引入机器学习的工程哲学有观众问及，既然行业内广泛使用强化学习（RL）和机器学习来优化数据中心功耗及各类内部业务流程，特斯拉是否在工厂制造、工业设计或其他工程环节中大范围使用了机器学习。马斯克给出了一个出人意料却极具实战工程风格的回答：他强烈反对在工厂流程中盲目滥用机器学习。马斯克警告道：“Unless you basically have to use machine learning, don't do it. It's usually a red flag when somebody says, 'We want to use machine learning to solve this task.'” （除非你基本上不得不使用机器学习，否则千万别用。当有人说‘我们想用机器学习来解决这个任务’时，通常是一个危险信号。）马斯克解释说，现实中有 99.9% 的工程和制造问题根本不需要用到机器学习，传统的算法、物理法则和常识性工程手段要直接得多。机器学习是一件极度困难、脆弱且维护成本极高的工具，绝不是某种唾手可得的灵丹妙药。只有在常规手段彻底穷尽、非它不可时，才应该动用它。他半开玩笑地补充道，如果未来工厂里普及了能够听懂自然语言指令的人形机器人，工厂的交互和制造逻辑可能会发生变化；但就现阶段而言，特斯拉在工厂制造和生产环节中的铁律是：能少用就少用，最大限度地减少机器学习的存在。 ### [02:36:20](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9380s) 仿真器技术栈：从光栅化到神经渲染与罕见事故模拟一位现场提问者针对展示中结合图形学与神经网络的仿真器提问，询问底层到底采用了哪些渲染与图形技术，尤其是在神经渲染（neural rendering）方面有什么探索。特斯拉仿真团队负责人解释，仿真器的技术栈底层仍然是传统游戏引擎技术，主要是类似《侠盗猎车手》（Grand Theft Auto / GTA）风格的实时光栅化（real-time rasterization）。在此基础之上，他们叠加了实时光线追踪（real-time ray tracing）。至于刚在演示中展示的神经渲染成果，则是“刚出炉”的最新尝试，幻灯片底部的标注甚至就来自昨晚。团队正在深入神经渲染领域试验多种方案，终极目标是将其作为最后画龙点睛的一步（“the cherry on the top”），彻底打破仿真器环境的规律性，让自动驾驶模型永远无法对仿真场景过拟合。特斯拉目前正在尝试类似于学界关于“逼真度增强”（photorealism enhancement）论文中的方法。但与学术界不同，特斯拉拥有庞大得多的真实标注数据、海量的计算资源、对环境的极高控制力，以及顶尖的工程团队来确保这一切能够实时运行。工程师强调，虽然理论上希望做到仅靠仿真就能训练所有内容，但现实中特斯拉永远不需要这么做，因为团队坐拥任何其他对手都不具备的海量真实行驶数据，仿真器主要用于修补现实世界中的细小缝隙。 Elon Musk 随后补充了仿真器的不可替代性：它在极端罕见场景（edge cases）中具有极其重大的价值。例如碰撞发生前数毫秒的极端规避动作。Musk 幽默地指出，这里存在一个讽刺的悖论：“Ironically, the better our cars become at avoiding accidents, the fewer accidents there are so then our training set is small, so then we have to make them crash in the simulation.” 车越安全，事故样本越少，因此必须逼迫车辆在仿真世界中不断撞车。在虚拟场景中，系统可以设定极限参数：距离行人或障碍物仅剩 5 米，车速高达每秒 20 米，求解能将乘员与行人伤亡概率降至最低的唯一动作轨迹。类似高速公路上逆行车辆等极罕见事件，也可以在仿真中批量构建。 ```mermaid flowchart TD accTitle: 仿真器技术栈与极端场景生成 accDescr: 特斯拉仿真器从传统光栅化到光线追踪再到神经渲染的分层架构，以及极端场景仿真的闭环 subgraph Stack["仿真器技术分层 (Simulation Stack)"] A["底层：传统游戏光栅化 (GTA 风格)"] --> B["中层：实时光线追踪 (Ray Tracing)"] B --> C["顶层：神经渲染 (Neural Rendering / 逼真度增强)"] end subgraph RareCases["极端情况合成 (Edge Cases)"] D["现实世界罕见事件 (如逆行、极限避险)"] --> E["仿真器碰撞生成 (生成数千种极端变体)"] E --> F["轨迹与规避规划 (极短距离/极限速度)"] end C --> G["高保真合成数据"] F --> G G --> H["模型训练与评测 (填补现实数据缝隙)"] ``` ### [02:38:48](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9528s) 特斯拉机器人（Optimus）的初始落地场景与定位现场观众接着询问关于人形机器人 Tesla Bot（Optimus）的落地路线，特斯拉究竟选定了哪些早期使用场景，以及在使其具备实用性上面临哪些重大挑战。 Elon Musk 回应，对于代号 Optimus 的特斯拉人形机器人，其首要定位非常直接，那就是处理“无聊、重复且危险”（boring, repetitive, and dangerous）的工作。用一句话概括，就是人类最不愿意去干的重体力或单调枯燥的脏活累活。随后另一位观众在提问中进一步探究人形形态（humanoid form factor）的独特优势，并追问在产品路线上是否考虑过让机器人承担“情感陪伴”（emotion or companionship）的角色。Musk 笑着调侃，首先大家绝对不希望看到它演变成反乌托邦科幻惊悚片里的桥段。他再次重申，团队目前在定位上极其务实： > “Really at this point, we're saying like maybe this robot can just — we're trying to be as literal as possible: can it do boring, dangerous, repetitive jobs that people don't want to do? And you know, once you can have it do that, then maybe you can do other things too.” Musk 也顺势打趣道，如果以后有人想买一台回家当成自己的朋友或者伙伴，也完全可以，相信广大用户届时一定会发挥出各种极具创意的神奇用法。 ### [02:39:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9568s) 仿真与现实的领域自适应（Domain Adaptation）与“9的攻坚战” 提问转回仿真系统：鉴于当前仿真器仍非完美，提问者好奇特斯拉是否在流水线中使用了显式的领域自适应（Domain Adaptation）技术，以弥补合成数据与真实物理世界之间的分布漂移（distribution shift），防止直接部署纯仿真训练模型所带来的安全风险。仿真团队工程师详细披露了数据流： - 仿真器直接导出带有完整运动学（kinematics）状态的连续视频剪辑，模型直接在该数据上进行针对性训练。 - 合成数据在整体训练集中仅占极小的一部分定向细分样本（small targeted segment），模型的评估测试集（evaluation set）则百分之百来自真实世界的路采视频。 - 特斯拉目前并未遭遇严重的仿真过拟合。为了防患于未然，团队甚至拿车机中运行的相同视觉网络进行重新训练，专门用来判别一帧画面是“仿真还是真实”，并将该判别反馈给图形艺术与渲染团队，指导其改进美术和渲染材质。 Elon Musk 紧接着强调，特斯拉整体数据集绝大多数依然是真实车队开出来的真实道路视频，因为“没有什么比现实更古怪、拥有更多边角案例了”。现实世界的路况奇幻莫测。当车队捕获到几张极其怪异、罕见的真实画面时，团队就会在仿真器中基于该奇特模式生成成千上万种形态各异的变体，以补足模型在这些关键死角上的认知盲区。Musk 强调，这一切的终极目标就是在时间推移中无限降低事故伤亡率，这场技术突围被称为“9的攻坚战”（the march of nines）： > “All of this is about over time just reducing the probability of a crash or an injury, and it's called the march of nines — like how do you get to 99.999999% safe, you know? Each nine is an order of magnitude difficulty increase.” ### [02:43:12](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9792s) 模型规模扩展定律、自动标注蒸馏与超人类预测能力针对大语言模型（LLM）中通过 Scaling（扩展参数与计算量）展现出的惊人能力，观众提问特斯拉在自动驾驶视觉与规划网络中是否也观察到了类似的模型缩放效应（Scaling laws）。 Andrej Karpathy 迅速给予了高度肯定的答复：“Absolutely a bigger network typically we see it performs better provided you have the data to also train it with.” Karpathy 指出，虽然车载 FSD 芯片受制于超低延迟（latency）的要求，模型必须精巧设计并施加严格限制，但特斯拉并不只训练跑在车里的网络。在云端的自动标注流水线（auto-labeling pipelines）中，模型尺寸不受任何物理限制： - 云端部署了体量数倍于车载端、性能显著超越车载端的超大参数模型，以实现接近 100% 的重建与标注精度。 - 车载网络实际上是这一庞大系统的“知识蒸馏”（distillation）产物。云端庞大模型结合跨时间、跨视角的几何重建结构，挖掘出 ground truth，然后再将知识浓缩注入到部署至车载 FSD 电脑的轻量级网络中。 - Musk 随后补充指出，计算不能在某一帧画面上死磕占用 200 毫秒，因为新帧在不断涌入，算力必须在时间流中合理分配。随后，Musk 和 Karpathy 揭示了一个令人震撼的现象：神经网络展现出了不可思议的超视距预测能力。 - 即使道路蜿蜒拐弯、视线完全被遮挡，模型也能以极高的精度预测出弯道后方的道路走向。 - 哪怕仅仅从公交车窗缝隙里瞥见自行车的一角，模型也能立即精准脑补并预测出行人或骑行者的存在。 - Karpathy 惊叹道，网络常常在完全不可见的情况下“编造”出周围场景，而这种脑补往往准得令人发毛（eerily good），以至于工程师们甚至会反复核对这是否在训练集中泄露过。 - Karpathy 总结道，从极限角度来看，一个参数量足够巨大的神经网络甚至足以将整个地球的道路特征全部记忆下来，“就像把高精地图（HD Map）直接烘焙进了神经网络的权重之中”。 ### [02:46:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9974s) 人形机器人手的形态与通用物理交互一位观众就特斯拉人形机器人（Tesla Bot）的机械构造提出疑问，询问保留完全拟人化形态的必要性有多高，尤其是制造一副拥有五根手指、同时又必须受制于严格重量限制的机械手极具挑战，甚至可能不得不采用线驱（cable-driven）机制从而引发一系列次生工程难题。Elon Musk 坦率地回应称这仅仅是机器人第 1 版（Version 1），后续还将视情况演进。 Musk 强调，机器人的根本使命在于替代人类完成日常事务，因此必须是一台通用人形机器人。在手部构型上，他打趣说理论上可以做成两根手指加一个大拇指的形态，但团队现阶段仍选择尝试标准的五指构型。“you know for now we'll we'll give it five fingers and and see see if that works out okay,” Musk 表示，这只手并不需要具备惊人的握力，但它的核心指标是能够使用人类日常工具以及搬运提包等物品。 ### [02:47:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10035s) 消除界面抖动与建立全自动驾驶的信任一名观众转述了其大学教授对特斯拉的批评意见：Autopilot 的用户界面（UX）经常出现物体原地打转或分类标签快速闪烁跳动的现象，这让车主感到缺乏安全感。观众借此发问，特斯拉打算如何说服车主、其他道路使用者乃至公众，让他们相信这套全自动驾驶（FSD）系统是真正安全可靠的。 Musk 立即反驳称，汽车在屏幕上自转已经是过去式了，如果在 FSD Beta 的视频中观察，其感知与渲染表现已经相当稳固。Andrej Karpathy 随即补充，随着输入更多训练数据以及全面采用多摄像头联合网络，系统表现正迅速蜕变；这一代新架构其实只上线了几个月，目前远非最终形态。Karpathy 指出，视频中已经包含了构建纯净几何空间所需的一切原始信息，只要数据充足、网络结构得当，系统毫无疑问会远超人类水准。 ```mermaid flowchart LR accTitle: 从早期视觉抖动到多摄像头BEV感知的演进 accDescr: 展示特斯拉如何从单帧不稳定检测进化为多摄像头时间序列空间感知 subgraph EarlyStage["早期单镜头架构"] A["单摄像头输入"] --> B["逐帧目标检测"] B --> C["物体旋转/跳变不稳定"] end subgraph ModernStage["新一代多摄空间架构"] D["8个环视摄像头"] --> E["多摄特征融合与时序网络"] E --> F["矢量化BEV真实空间输出"] end C -.->|"架构重构与大规模数据"| ModernStage ``` ### [02:48:33](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10113s) 人形机器人的商业化路径提问麦克风转到下一位观众，他抛出了一个尖锐的经济学问题：在长远消灭体力劳动的宏大叙事之外，机器人中短期的商业闭环何在？因为现实中重复、危险且枯燥的初级劳动往往报酬微薄，特斯拉无法简单套用当年做车时“先推出昂贵跑车 Roadster、再逐步渗透大众市场”的自上而下路径。对此，Musk 只是意味深长地笑了笑，以一句简短的幽默带过：“well i guess you'll just have to see.” ### [02:49:16](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10156s) 电车难题的伪命题：以超高算力与反应频率碾压人类现场观众提问特斯拉 AI 如何在危急关头权衡“车内乘员安全”与“车外行人安全”，以及这种价值决策是如何写入算法的。Musk 直截了当地指出，这在工程实践中很大程度上是一个伪命题，因为这种极端二选一的情况在现实中极其罕见。其核心原因在于计算机与人类在感知时间尺度上的巨大不对称性。Musk 阐释道：“from the computer standpoint everything is moving slowly.” 在计算机眼里，周围物体的运动极其缓慢。他打了个比方：就像运行《赛博朋克 2077》或现代先进电子游戏时，GPU 能在全光线追踪和神经网络超分渲染下以每秒 60 帧稳定输出，而一个没有犯错的人类光是亲手画出其中一帧画面就得耗费整整一个月。计算机具备超高速的处理能力，而人类非常迟缓。 Musk 进一步援引 SpaceX 的火箭姿态控制作为佐证：人类不可能手动用手柄把火箭开进轨道，因为火箭姿控要求系统在 6 至 7 赫兹的频率下做出反应，人类根本反应不过来；而自动驾驶系统的决策循环甚至达到了 30 赫兹。极高的反应速度和算力余量让系统能够提早规避危险，根本不会陷入非此即彼的绝境。 ### [02:51:15](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10275s) Hardware 3 极限与 Hardware 4 的演进路线一位与会者询问关于 Hardware 3（HW3）算力见顶的传闻，想了解扩大神经网络规模后现有计算模式还剩多少裕量，以及 Hardware 4（HW4）会在何时成为不可或缺的硬件。 Musk 明确表达了对现役 HW3 硬件（FSD 计算机 1 代）的坚定信心。他认为 HW3 足以实现完全自动驾驶，其安全性至少可以比普通人类驾驶员高出 200% 到 300%。随后 Musk 透露了下一代硬件的规划细节： - 规划中的 Hardware 4（FSD 计算机 2 代）预计将随 Cybertruck 一同推出，时间大约在一年左右。 - HW4 的计算性能大约将是 HW3 的 4 倍。 - 升级硬件的驱动力并非因为 HW3 无法上路，而是追求更高的安全性上限——即将其从比人类安全 300% 进一步推升至安全 1000%。 - Musk 类比道，现实中道路上充斥着驾驶技能参差不齐的司机，驾照体系并没有要求每个人都是世界顶尖车手，自动驾驶系统只要达到超越普通人类的稳健门槛即可合法运行。 ### [02:52:39](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10359s) 对抗样本攻击与纯视觉“怀尔狼”陷阱针对特斯拉车辆彻底摒弃毫米波雷达等深度传感器、完全依赖 RGB 摄像头的方案，观众发问是否担心遭遇物理对抗攻击（Adversarial Attacks），比如有人故意打印假照片来欺骗神经网络。 Musk 笑着调侃这就像经典动画片里怀尔狼（Wile E. Coyote）在坚硬的石壁上画一堵假隧道来诱骗哔哔鸟撞墙一样。他表示现实中特斯拉并未遇到太多恶意攻击，不过他分享了一个真实案例：“like right now if you pr most likely if you had like a a t-shirt with a t-shirt with like a stop sign on it which i actually have a t-shirt with a stop sign on it and and then you like flash the car it will it will stop i i proved that.” Musk 亲自验证过，如果身上穿着印有“STOP”停止标志的 T 恤突然对着车晃一下，车辆确实会被迫刹停。然而他也指出，随着模型见到此类边缘案例，团队可以针对性地重新训练网络，使其学会识别这是一名穿着停止标志印花衣服的行人，从而判断它并非真实的路面交通标志。 ### [02:53:54](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10434s) 规划中的多模态不确定性与影子模式训练针对自动驾驶如何预测周围车辆轨迹并进行不确定性规划，Ashok Elluswamy 解释说，特斯拉并不会武断地假定其他驾驶员会老老实实遵守规则。规划系统原生支持“多模态未来”（multi-modal futures），即同时对目标可能采取的多种行为分支进行建模，并结合实际物理规律与运动学约束，在自身行动前验证其他物体的轨迹绝不会与自身发生干涉冲突。系统在感知到不确定性时会采取保守策略并适时让行；但为了保证通行的实用性，系统也会在确保对方能安全让行并做出合理反应的前提下果断切入。 Musk 补充了特斯拉核心的“影子模式”（Shadow Mode）： - 任何新神经网络在正式接管车辆前，都会以静默的影子模式部署在上百万台量产车队中运行。 - 系统在后台实时推演决策（例如在某场景下模型是否会选择向右变道），并与人类司机的真实操作（如人类司机实际上向左打了方向）做比对。 - 一旦产生分歧，系统便会触发差分事件记录，数据回传以分析偏差原因。 - 只要人类司机的最终操作没有演变成事故，整个车队的所有人类驾驶员就相当于在免费、源源不断地为神经网络标注“正确行为模式”。 Elluswamy 随后补充道，车辆内部内置了闪烁度等多种不确定性度量。当系统视线受阻或置信度下降时，车辆会主动减速以采集更多环境信息，绝不会在未知盲区中鲁莽前行。Musk 将这一原则提炼为一句核心控制律：“speed proportionate to confidence”（车速应与系统的置信度成正比）。 ### [02:56:22](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10582s) AI 技术栈的短板与短周期主攻方向一位提问者最后请教，在仿真、数据收集、标注、规划与控制这整套庞大的 AI 技术栈中，如果只能改进其中一项，哪一项能对 FSD 的整体性能产生最显著的杠杆效应。 Elluswamy 认为系统整体性能取决于由各环节构成的“曲线下面积”（Area Under the Curve），任何一个关键节点的优化都会直接扩大面积、抬升系统表现。而在当下短期内，特斯拉最紧迫的任务是将感知侧的所有网络彻底转化为空间与时序统一的大模型。 ### [02:57:04](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10624s) 环绕视频架构的迁移与人才招聘瓶颈 Andrej Karpathy 解释了团队目前在模型架构演进上的状态。虽然 Tesla 正在全面转向环绕视频（surround video）网络架构，但在工程系统中仍保留了一些未接入环绕视频的遗留网络（legacy nets）。Karpathy 指出这只是短期现象，团队正在迅速推进替换，理想状态下所有网络都应基于环绕视频构建。他感叹成功所需的诸多拼图碎片已经就位，但真正的瓶颈在于执行力与工程人力。 Elon Musk 随即附和了 Karpathy 的观点，强调这就是举办本次 AI Day 活动的根本初衷。Musk 坦言：“这才是实际的瓶颈，正如 Andrej 所言，要让这一切运转起来，还有海量的工作要做，所以我们需要才华横溢的人才加入我们来解决这个问题。”（“There's just a tremendous amount of work to do to make make it work so that's why we need talented people to join in and solve the problem.”） - 遗留模型正在加速过渡，目标是将全车所有视觉网络统一到环绕视频架构下。 - 当前制约全自动驾驶落地进度的核心瓶颈是优秀 AI 与系统工程师的数量。 ### [02:57:49](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10669s) 灾难性遗忘与端上持续学习机制一位现场观众提问：在面对海量数据时，Tesla 如何应对神经网络中常见的“遗忘问题”（catastrophic forgetting），以及是否会考虑采用端上学习（online learning）或持续学习（continuous learning），让每位车主拥有个性化的自动驾驶软件版本。 Andrej Karpathy 回应称，学术界讨论的灾难性遗忘并不是 Tesla 实际遇到的问题。Tesla 并没有在车端运行连续学习架构，而是采用中心化训练流程：系统在云端集中训练一次，经过数次微调，随后进行严苛全面的离线评测，确认模型表现稳定、表现优异后，才会打包推送到车辆终端。因此，系统不会因为就地学习而产生不可控的遗忘风险。Musk 补充道，尽管不采用端上模型训练，未来系统仍会提供行为偏好设置，让车主调整驾驶风格。Musk 形容这就像对打车软件司机下达指令一样：“比如‘我赶去机场要迟到了，能不能开快点？’——就像你给 Uber 司机提的要求一样。” - 模型更新采用集中离线训练与微调，规避端上持续学习带来的不稳定性与灾难性遗忘。 - 个性化驾驶体验将通过配置参数实现（如激进/保守风格切换、赶时间模式等），而非在每辆车上单独训练权重。 ### [02:59:12](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10752s) AI 对齐、超级智能与专用 AI 的安全边界针对观众提出的大模型能力提升是否会导致 AI 与人类意图出现“失准”（misalignment）以及 Tesla 部署人形机器人面临的安全顾虑，Elon Musk 阐述了 Tesla 的技术定位与安全哲学。Musk 承认人类确实应该警惕 AI 的长期潜在风险，但他强调 Tesla 目前研发的是专用 AI（narrow AI）。 Musk 解释称，当前的目标非常明确且聚焦：一是让汽车驾驶安全性超越人类，二是让双足人形机器人（Tesla Bot）能够替人类执行基础、重复的日常琐事。尽管 Musk 预测超越人类的超级智能（superhuman intelligence）未来可能出现并带来不可控变数，但 Tesla 现阶段的行动准则非常纯粹：“我们在 Tesla 尝试做的是创造人们喜爱的、毫无疑问是有益的实用 AI，这就是我们的努力方向。”（“What we're trying to do here at Tesla is make useful AI that people love and is unequivocally good that's our, you know try to aim for that.”） ### [03:00:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10835s) 传感器演进：Hardware 4 与“自动驾驶电梯”隐喻最后一位观众提出，人眼不仅仅是连续采集画面的帧序列，普通相机是人眼非常粗糙的近似，询问 Tesla 是否考虑引入事件相机（event camera）或自研更灵活的定制摄像头。Musk 透露，在下一代自动驾驶硬件 Hardware 4（HW4）上，Tesla 确实会引入下一代摄像头。然而 Musk 明确强调，工程团队远未触及当前硬件与摄像头的性能极限：“我有信心，利用现有的摄像头和当前的计算硬件，我们就能实现安全性远超人类的全自动驾驶。”（“I'm confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware.”）软硬件的后续演进，只是为了将安全冗余从超越人类 300% 进一步推升至 1000%。在大会结语中，Musk 描绘了自动驾驶技术的终局图景。他认为未来自动驾驶汽车会像“自动电梯”一样自然普遍。在历史上，电梯曾经由专职操作员手动拉闸控制，操作员疲劳或失误时甚至会造成严重的人身切割惨剧；而如今人们走进百层摩天大楼，只需按一个按钮即可放心到达目的地。Musk 总结道，未来的汽车同样都会变成自动驾驶与纯电动的，燃油车和手动驾驶汽车最终会像马匹一样退居为少数爱好者的消遣。随着 Musk 向全场致谢，Tesla AI Day 2021 正式落下帷幕。 ```mermaid flowchart TD accTitle: Tesla 自动驾驶部署演进与历史隐喻 accDescr: 展示从手动操作时代向自动化系统迁移的历史演进过程 subgraph Historical["历史范式：手动操作时代"] H1["人工电梯员操作继电器闸门"] -->|"疲劳 / 操作失误"| H2["安全事故与机械隐患"] H3["人类手动驾驶燃油车"] -->|"注意力分散 / 反应极限"| H4["每年高发交通事故"] end subgraph Modern["现代与未来范式：自动化与 AI 时代"] M1["现代全自动电梯"] -->|"按键即走"| M2["零心智负担与超高安全性"] M3["Tesla Vision + FSD 算法"] -->|"HW3 / HW4 算力驱动"| M4["全自动纯电出行 (超越人类安全标准)"] end H2 -.->|"技术迭代与系统自动化"| M2 H4 -.->|"AI 驱动与纯视觉替代"| M4 ``` - Hardware 4 将搭载全新一代摄像头，但 HW3 的现有算力与传感器配置已足够实现超越人类安全水平的 FSD。 - 自动化技术普及的终极形态是“心智隐形”：如同手动电梯被一键直达电梯取代一样，手动驾驶燃油车最终将转变为类似马术的小众保留项目。 ## Notable quotes > [02:34:56](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9296s) Unless you basically have to use machine learning, don't do it. It's usually a red flag when somebody says, 'We want to use machine learning to solve this task. — Elon Musk > [02:36:20](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9380s) Ironically, the better our cars become at avoiding accidents, the fewer accidents there are so then our training set is small, so then we have to make them crash in the simulation. — Elon Musk > [02:28:58](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8938s) GPUs are still essentially running neural net training in emulation mode. — Elon Musk > [02:15:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=8128s) the acid test here and you know what I've told the Dojo team is like it's successful if the software team wants to turn off the GPU cluster, but if they want to keep the GPU cluster on, it's not successful. — Elon Musk > [02:31:14](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9074s) Certainly, end-to-endness is the guiding principle behind a lot of the network developments, and over time in the stack, neural networks have taken on more and more functionality. — Andrej Karpathy > [03:00:35](https://www.youtube.com/watch?v=j0z4FweCy4M&t=10835s) I'm confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware. — Elon Musk > [02:39:28](https://www.youtube.com/watch?v=j0z4FweCy4M&t=9568s) All of this is about over time just reducing the probability of a crash or an injury, and it's called the march of nines — like how do you get to 99.999999% safe, you know? Each nine is an order of magnitude difficulty increase. — Elon Musk > [02:10:04](https://www.youtube.com/watch?v=j0z4FweCy4M&t=7804s) It is fundamentally extremely expensive to create the system so somehow that has to be paid for. I'm not sure how to pay for it if it's fully open sourced. — Elon Musk ## Mentioned Tesla, Elon Musk, Dojo, D1, DPU (Dojo Processing Unit), Training Tile, ExaPOD, LLVM, PyTorch, Tesla Bot, Optimus, FSD (Full Self-Driving), Autopilot, Bill Chang, Ian Glow, Ali Jahanian, Andrej Karpathy, Ashok Elluswamy, Hardware 3 (HW3), Hardware 4 (HW4), Cybertruck, SpaceX, Grand Theft Auto (GTA), Cyberpunk 2077 ## Video description Join us to build the future of AI → https://www.tesla.com/ai 0:00 - Pre-event 46:54 - AI Day Begins 48:44 - Tesla Vision 1:13:12 - Planning and Control 1:24:35 - Manual Labeling 1:28:11 - Auto Labeling 1:35:15 - Simulation 1:42:10 - Hardware Integration 1:45:40 - Dojo 2:05:14 - Tesla Bot 2:12:59 - Q&A [Source](https://www.youtube.com/watch?v=j0z4FweCy4M)

Source quotes

[154:56] Elon Musk: Unless you basically have to use machine learning, don't do it. It's usually a red flag when somebody says, 'We want to use machine learning to solve this task. [156:20] Elon Musk: Ironically, the better our cars become at avoiding accidents, the fewer accidents there are so then our training set is small, so then we have to make them crash in the simulation. [148:58] Elon Musk: GPUs are still essentially running neural net training in emulation mode. [135:28] Elon Musk: the acid test here and you know what I've told the Dojo team is like it's successful if the software team wants to turn off the GPU cluster, but if they want to keep the GPU cluster on, it's not successful. [151:14] Andrej Karpathy: Certainly, end-to-endness is the guiding principle behind a lot of the network developments, and over time in the stack, neural networks have taken on more and more functionality. [180:35] Elon Musk: I'm confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware. [159:28] Elon Musk: All of this is about over time just reducing the probability of a crash or an injury, and it's called the march of nines — like how do you get to 99.999999% safe, you know? Each nine is an order of magnitude difficulty increase. [130:04] Elon Musk: It is fundamentally extremely expensive to create the system so somehow that has to be paid for. I'm not sure how to pay for it if it's fully open sourced.

Source quotes

[129:00] Elon Musk: I think essentially in the future physical work will be a choice—if you want to do it you can, but you won't need to do it. [105:40] Ganesh Venkataramanan: it's easy to scale the compute; it's very difficult to scale up bandwidth and extremely difficult to reduce latencies [84:35] Andrej Karpathy: So neural networks are just an upper bound; you also need massive datasets to actually train the correct algorithms inside them. [153:00] Elon Musk: In fact I'm sure you can get to much safer than human with control and planning primarily in C++ with perception vision in neural nets. [180:35] Elon Musk: I’m confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware. [87:39] Andrej Karpathy: So people are extremely good at things like semantics, but computers are very good at geometry, reconstruction, triangulation, tracking. [48:44] Andrej Karpathy: What I find kind of fascinating about this is that we are effectively building a synthetic animal from the ground up.

Original transcript

[0:02] [Music] [0:07] [Music] [1:02] [Music] [1:27] [Music] [1:35] [Music] [1:49] [Music] [2:01] [Music] [2:14] [Music] [2:22] [Music] [2:57] [Music] [4:10] [Applause] [4:16] [Music] [4:26] [Applause] [4:27] [Music] [5:02] [Music] [5:24] [Music] [5:30] so [5:32] [Music] [5:57] [Music] [6:28] [Music] [6:55] [Music] [7:25] [Music] [7:55] [Music] [8:05] [Music] [8:21] [Music] [9:01] [Music] [10:08] [Music] [11:12] [Music] [11:23] [Music] [11:37] [Music] [11:56] [Music] [12:13] [Music] [12:25] [Music] [12:37] [Music] [12:54] i'm not the man you want you've made a [12:56] mistake [12:58] i'm in touch with technological advances [13:00] pretty well everywhere and i can tell [13:01] you that no one has produced anything [13:03] like this [13:10] [Music] [13:20] [Music] [13:44] [Music] [13:52] [Music] [14:40] [Music] [14:46] [Music] [15:08] [Music] [15:37] [Music] [16:13] so [16:21] [Music] [16:45] [Music] [17:05] [Music] [17:38] oh [17:42] [Music] [17:51] so [18:20] [Music] [18:37] wow [18:39] [Music] [18:51] [Music] [19:07] armed [19:17] do [19:22] [Music] [19:42] [Music] [19:48] i'm [19:58] um [20:40] armed [20:52] [Music] [21:15] [Music] [22:02] so [22:05] [Music] [22:23] [Music] [22:47] [Music] [23:17] [Music] [23:30] [Music] [24:04] [Music] [24:25] so [24:26] [Music] [24:34] [Music] [24:49] [Music] [24:59] [Music] [25:14] [Music] [25:21] [Music] [25:26] [Music] [26:21] [Music] [26:46] [Music] [26:54] [Music] [27:08] [Music] [27:19] [Music] [27:33] [Music] [27:41] [Music] [28:16] [Music] [28:28] [Applause] [28:29] [Music] [29:04] so [29:05] [Music] [29:17] so [29:23] [Music] [29:28] [Applause] [29:35] [Music] [29:45] [Applause] [29:46] [Music] [30:18] [Music] [30:42] so [30:43] [Music] [31:45] [Music] [31:50] so [31:52] [Music] [31:57] [Music] [32:06] so [32:14] [Music] [32:20] so [32:24] [Music] [32:43] [Music] [33:07] [Music] [33:14] [Music] [33:16] so [33:17] [Music] [33:24] [Music] [33:40] [Music] [34:20] [Music] [36:31] [Music] [36:42] [Music] [36:56] [Music] [37:15] [Music] [37:44] [Music] [37:51] [Music] [37:56] [Music] [38:13] i'm not the man you want you've made a [38:16] i'm in touch with mistake [38:18] advisors pretty well everywhere and i [38:20] can tell you that no one has produced [38:22] anything like this [38:29] [Music] [38:39] [Music] [39:03] [Music] [39:21] so [39:23] [Music] [39:35] so [39:38] [Music] [39:50] so [39:58] [Music] [40:22] [Applause] [40:23] [Music] [40:43] [Applause] [40:46] [Music] [40:52] [Applause] [40:53] [Music] [41:31] foreign [41:36] so [41:46] [Music] [42:02] [Music] [42:56] um [43:01] [Music] [43:15] so [43:30] so [43:35] [Music] [43:58] [Music] [44:06] you [44:09] [Music] [44:15] [Music] [44:25] armed [44:41] [Music] [44:55] [Music] [45:01] [Music] [45:12] [Music] [45:17] um [45:31] [Music] [45:50] [Music] [45:59] armed [46:09] [Music] [46:32] [Music] [46:48] do [46:54] [Music] [47:09] hello everyone uh sorry for the delay um [47:12] thanks for coming [47:14] and um sorry i had some technical [47:16] difficulties [47:17] really neat ai for this [47:20] so uh what we want to [47:22] show today is that tesla is [47:25] much more than an electric car company [47:27] that we have [47:28] deep ai activity [47:30] in [47:32] hardware on the inference level on the [47:34] training level [47:36] and [47:38] basically we [47:39] i think we're [47:40] i think arguably the leaders in real [47:42] world ai as it applies to the real world [47:45] um [47:46] and those of you who have seen the full [47:48] self-driving [47:49] beta i can appreciate the rate at which [47:52] the tesla [47:53] neural net is learning to to drive [47:56] and um [47:57] so this is a particular application of [47:59] ai but i think there's there's more [48:01] there are more applications uh down the [48:03] road that will make sense and we'll talk [48:05] about that later in the presentation [48:08] but um [48:09] yeah we basically want to encourage [48:11] anyone who is interested in [48:13] solving real-world ai problems [48:15] at either the hardware or the software [48:17] level to join tesla or consider drawing [48:19] tesla [48:20] so let's see we'll start off with andre [48:30] great [48:44] okay hi everyone um welcome [48:46] my name is andre and um i am [48:49] i lead the vision team here at tesla [48:51] autopilot and i'm incredibly excited to [48:53] be here uh to kick off this section [48:55] giving you a technical deep dive into [48:57] the autopilot stack and showing you all [49:00] the under the hood components that go [49:01] into making the car drive all by itself [49:05] so we're going to start off with the [49:06] vision component here [49:07] now in division component what we're [49:09] trying to do is we're trying to design a [49:10] neural network that processes the raw [49:12] information [49:14] which in our case is the eight cameras [49:15] that are positioned around the vehicle [49:17] and they send us images and we need to [49:19] process that in real time into what we [49:21] call the vector space and this is a [49:22] three-dimensional representation of [49:24] everything you need for driving so this [49:26] is the three-dimensional positions of [49:27] lines edges curbs [49:29] traffic signs traffic lights [49:31] cars their positions orientations [49:34] depth velocities and so on [49:39] so here i'm showing a video of [49:42] actually hold on [49:46] apologies [49:47] so here i'm showing the video of the raw [49:50] inputs that come into the stack and then [49:52] neural processes that into the vector [49:54] space and you are seeing parts of that [49:56] vector space rendered in the instrument [49:58] cluster on the car [50:10] now [50:11] what i find kind of fascinating about [50:12] this is that we are effectively building [50:14] a synthetic animal from the ground up so [50:17] the car can be thought of as an animal [50:18] it moves around it senses the [50:19] environment and uh you know acts [50:22] autonomously and intelligently and [50:24] we are building all the components from [50:25] scratch in-house so we are building of [50:27] course all of the mechanical components [50:29] of the body the nervous system which is [50:31] all the electrical components and for [50:32] our purposes the brain of the autopilot [50:34] and specifically for this section the [50:37] synthetic visual cortex [50:39] now the biological visual cortex [50:40] actually has quite intricate structure [50:41] and a number of areas [50:43] that organize the information flow of [50:46] this brain and so in particular in our [50:48] in your visual cortices [50:51] the information hits the [50:53] light hits the retina it goes through [50:54] the lgn all the way to the back of your [50:55] visual cortex goes through areas v1 v2 [50:58] v4 the it the venture on the dorsal [51:00] streams and the information is organized [51:02] in a certain layout and so when we are [51:04] designing the visual cortex of the car [51:07] we also want to design the neural [51:08] network architecture of how the [51:10] information flows in the system [51:13] so the processing starts in the [51:15] beginning when light hits our artificial [51:17] retina and we are going to process this [51:19] information with neural networks now i'm [51:21] going to roughly organize this section [51:24] chronologically so starting off with [51:26] some of the neural networks and what [51:27] they looked like roughly four years ago [51:28] when i joined the team and how they have [51:29] developed over time [51:31] so [51:32] roughly um [51:34] four years ago the car was mostly [51:35] driving in a single lane going forward [51:37] on the highway and so it had to keep [51:39] lane and it had to keep distance away [51:41] from the car in front of us and at that [51:42] time all of processing was only on [51:45] individual image level so a single image [51:47] has to be analyzed by a neural net and [51:48] make little pieces of the vector space [51:51] process that into little pieces of the [51:52] vector space so this processing [51:55] took the following shape we take a 1280 [51:57] by 960 input and this is 12 bit integers [52:00] streaming in at roughly 36 hertz now [52:02] we're going to process that with the [52:04] neural network so instantiate a feature [52:06] extractor backbone in this case we use [52:08] residual neural networks so we have a [52:10] stem and a number of residual blocks [52:11] connected in series [52:13] now the specific class of resnets that [52:14] we use are regnets because we like this [52:17] is like a very rec nets offer a very [52:19] nice design space for neural networks [52:21] because they allow you to very nicely [52:23] trade off latency and uh accuracy [52:26] now these reg nets give us as an output [52:29] a number of features at different [52:31] resolutions in different scales so in [52:33] particular on the very bottom of this [52:34] feature hierarchy we have very high [52:36] resolution information with very low [52:37] channel counts and all the way at the [52:39] top we have low spatial low resolution [52:42] especially but high channel counts so on [52:44] the bottom we have a lot of neurons that [52:46] are really scrutinizing the detail of [52:47] the image and on the top we have neurons [52:49] that can see most of the image and a lot [52:50] of that context have a lot of that scene [52:52] context [52:54] we then like to process this with [52:56] feature pyramid networks in our case we [52:57] like to use by fbns and they get the [53:00] mult they get to multiple scales to talk [53:02] to each other effectively and share a [53:03] lot of information so for example if [53:05] you're a neuron all the way down in the [53:07] network and you're looking at a small [53:08] patch and you're not sure this is a car [53:09] or not it definitely helps to know from [53:11] the top players that hey you are [53:13] actually in the vanishing point of this [53:14] highway and so that helps you [53:16] disambiguate that this is probably a car [53:19] after a buy fbn and a feature fusion [53:21] across scales we then go into task [53:23] specific heads so for example if you are [53:25] doing object detection we have a one [53:27] stage yolo like object detector here [53:30] where we initialize a raster and there's [53:32] a binary bit per position telling you [53:34] whether or not there's a car there and [53:36] then in addition to that if there is [53:38] here's a bunch of other attributes you [53:39] might be interested in so the x y with [53:41] height offset or any of the other [53:42] attributes like what type of a car is [53:43] this and so on [53:45] so this is for the detection by itself [53:47] now very quickly we discovered that we [53:49] don't just want to detect cars we want [53:50] to do a large number of tasks so for [53:53] example we want to do traffic light [53:55] recognition and detection a lane [53:56] prediction and so on so very quickly we [53:58] conversion this kind of architectural [54:00] layout where there's a common shared [54:02] backbone and then branches off into a [54:04] number of heads so [54:05] we call these uh therefore hydronets and [54:07] these are the heads of the hydra [54:10] now [54:11] this architectural layout has a number [54:13] of benefits so number one [54:16] because of the feature sharing we can [54:18] amortize the forward pass inference in [54:20] the car at test time and so this is very [54:22] efficient to run [54:23] because if we had to have a backbone for [54:25] every single task that would be a lot of [54:27] backbones in the car [54:28] number two this decouples all of the [54:30] tasks so we can individually work on [54:32] every one task in isolation and for [54:34] example we can we can upgrade any of the [54:36] data sets or change some of the [54:37] architecture of the head and so on and [54:38] you are not impacting any of the other [54:40] tasks and so we don't have to revalidate [54:42] all the other tasks which can be [54:43] expensive [54:44] and number three because there's this [54:46] bottleneck here and features [54:48] what we do fairly often is that we [54:50] actually cache these features to disk [54:52] and when we are doing these fine tuning [54:54] uh workflows we only fine-tune from from [54:56] the cached features up and only find you [54:58] in the heads so most often in terms of [55:00] our training workflows we will do an [55:02] end-to-end training run once in a while [55:04] where we train everything jointly then [55:06] we cache the features [55:08] at the multiscale feature level and then [55:10] we fine-tune off of that for a while and [55:13] then end-to-end train once again and so [55:15] on [55:16] so here's the kinds of predictions that [55:18] we were obtaining i would say several [55:20] years ago now [55:21] uh from one of these hydronitis so again [55:23] we are processing individual images [55:27] there we go we are processing just [55:28] individual image and we're making a [55:30] large number of predictions about these [55:31] images so for example here you can see [55:33] predictions of the stop signs uh the [55:35] stop lines uh the lines the edges the [55:37] cars uh the traffic lights uh the curbs [55:40] here [55:41] uh whether or not the car is sparked uh [55:43] all of the static objects like trash [55:45] cans cones and so on and everything here [55:47] is coming out of the net [55:48] here in this case out of the hydra net [55:50] so that was all fine and great but as we [55:52] worked towards fsd we quickly found that [55:54] this is not enough so where this first [55:57] started to break was when we started to [55:59] work on smart summon here i am showing [56:01] some of the predictions of only the curb [56:03] detection task and i'm showing it now [56:04] for every one of the cameras [56:07] so we'd like to wind our way around the [56:08] parking lot to find the person who is [56:10] summoning the car now the problem is [56:11] that you can't just directly drive on [56:13] image space predictions you actually [56:15] need to cast them out and form some kind [56:16] of a vector space around you [56:19] so we attempted to do this using c plus [56:21] and developed what we call [56:23] the occupancy tracker at the time [56:27] so here we see that the curb detections [56:29] from the images are being stitched up [56:31] across camera scenes camera boundaries [56:33] and over time [56:35] now there are two pro two major problems [56:36] i would say with the setup number one we [56:38] very quickly discovered that tuning the [56:39] occupancy tracker and all of its hyper [56:41] parameters was extremely complicated you [56:43] don't want to do this explicitly by hand [56:45] in c plus you want this to be inside the [56:47] neural network and train that end to end [56:49] number two we very quickly discovered [56:51] that the image space is not the correct [56:52] output space uh you don't want to make [56:54] predictions in image space you really [56:55] want to make it directly in the vector [56:57] space so here's a way of illustrating [56:59] the issue [57:01] so here i'm showing on the first row the [57:02] predictions of our curves and our lines [57:05] in red and blue and [57:07] they look great in the image but once [57:09] you cast them out into the vector space [57:11] things start to look really terrible and [57:13] we are not going to be able to drive on [57:15] this so [57:16] you see how the predictions are quite [57:18] bad in vector space and the reason for [57:20] this fundamentally is because you need [57:22] to have an extremely accurate depth per [57:24] pixel in order to actually do this [57:25] projection and so you can imagine just [57:28] how high of the bar it is to predict [57:29] that depth so accurately in these tiny [57:32] in every single pixel of the image and [57:34] also if there's any occluded area where [57:36] you'd like to make predictions you will [57:37] not be able to because it's not an image [57:40] space [57:41] concept in that case [57:43] so we very quickly [57:45] real of [57:46] the other problems with this by the way [57:47] is also for object detection if you are [57:50] only making predictions per camera then [57:52] sometimes you will encounter cases like [57:54] this where a single car actually spans [57:56] five of the eight cameras [57:58] and so if you are making individual [58:00] predictions then no single camera since [58:02] sees all of the car and so obviously [58:04] you're not going to be able to do a very [58:05] good job of predicting that whole car [58:07] and it's going to be incredibly [58:08] difficult to fuse these measurements [58:10] so we have this intuition that what we'd [58:12] like to do instead is we'd like to take [58:13] all of the images and simultaneously [58:15] feed them into a single neural net and [58:17] directly output in vector space now this [58:19] is very easily said much more difficult [58:22] to actually achieve but roughly we want [58:24] to lay out a neural net in this way [58:26] where we process every single image with [58:28] a backbone [58:29] and then we want to somehow fuse them [58:32] and we want to re-represent it the the [58:34] features from image space features to [58:36] directly some kind of vector space [58:37] features and then go into the decoding [58:39] of the head [58:41] now [58:42] um so there are two problems with this [58:44] problem number one [58:45] how do you actually create the neural [58:47] network components that do this [58:48] transformation [58:51] and you have to make it differentiable [58:52] so that end-to-end training is possible [58:54] and number two uh [58:56] the if you want vector space predictions [58:58] from your neural net you need vector [59:00] specif based data sets so just labeling [59:02] images and so on is not going to get you [59:04] there you need vector space labels we're [59:05] going to talk a lot more about problem [59:07] number two later in the talk for now i [59:09] want to focus on the neural network [59:10] architectures so i'm going to deep dive [59:12] into problem number one [59:15] so here's the rough problem right we're [59:16] trying to have this bird's eye view [59:18] prediction instead of image space [59:19] predictions so for example let's focus [59:21] on a single pixel in the output space in [59:23] yellow and this pixel is trying to [59:24] decide am i part of a curb or not as an [59:27] example [59:28] and now where should the support for [59:30] this kind of a prediction come from in [59:32] the image space [59:33] well we know roughly how the cameras are [59:34] positioned and their extrinsics and [59:36] intrinsics so we can roughly project [59:38] this point into the camera images and [59:41] you know the evidence for whether or not [59:42] this is a curve may come from somewhere [59:43] here in the images the problem is that [59:45] this projection is really hard to [59:47] actually get correct because it is a [59:48] function of the road surface the road [59:50] surface could be sloping up or sloping [59:51] down or also there could be other data [59:53] dependent issues for example there could [59:55] be occlusion due to a car so if there's [59:57] a car occluding this this viewport this [60:00] this part of the image then actually you [60:02] may want to pay attention to a different [60:03] part of the image not the part where it [60:05] projects and so because this is data [60:07] dependent it's really hard to have a [60:08] fixed transformation for this component [60:10] so in order to solve this issue [60:13] we use a transformer to represent this [60:16] space and this transformer [60:18] uses multi-headed self-attention and [60:20] blocks off it [60:22] in this case actually we can get away [60:23] with even a single block [60:25] doing a lot of this work [60:26] and [60:27] effectively what this does is you [60:29] initialize a raster of the size of the [60:31] output space that you would like and you [60:33] tile it with positional encodings with [60:34] size and coses in the output space and [60:36] then these get encoded with an mlp into [60:38] a set of query vectors [60:40] and then all of the images and their [60:42] features also emit their own keys and [60:44] values and then the queries keys and [60:45] values feed into the multi-headed [60:47] self-attention and so effectively what's [60:49] happening is that every single image [60:50] piece is broadcasting in its key [60:53] what it is what is it a part of so hey [60:56] i'm part of a pillar in roughly this [60:57] location and i'm seeing this kind of [60:58] stuff and that's in the key and then [61:00] every query is something along the lines [61:02] of hey i'm a pixel in the output space [61:04] at this position and i'm looking for [61:06] features of this type [61:07] then the keys and the queries interact [61:08] multiplicatively and then the values get [61:10] pulled accordingly [61:13] and so this re-represents the space and [61:15] we find this to be very effective for [61:16] this transformation so if you do all of [61:19] the engineering correctly this again is [61:20] very easily said [61:22] difficult to do you do all of the [61:24] engineering correctly um [61:27] there's one more there's no problem [61:28] actually before [61:30] i'm not sure what's up with the slides [61:32] so one more thing you have to be careful [61:34] with some of the details here when you [61:35] are trying to get this to work so in [61:36] particular all of our cars [61:38] uh are slightly cockeyed in a slightly [61:41] different way and so if you're doing [61:42] this transformation from image space to [61:44] the output space you really need to know [61:45] what your camera calibration is and you [61:47] need to feed that in somehow into the [61:48] neural net and so you could definitely [61:50] just like concatenate the camera [61:52] calibrations of all of the images and [61:54] somehow feed them in with an mlp [61:56] but actually we found that we can do [61:58] much better by transforming all of the [61:59] images into a synthetic virtual camera [62:02] using a special rectification transform [62:04] so this is what that would look like [62:07] we insert a new layer right above the [62:09] image rectification layer it's a [62:11] function of camera calibration and it [62:13] translates all of the images into a [62:14] virtual common camera so if you were to [62:17] average up a lot of repeater images for [62:19] example which faced at the back [62:21] you would without doing this you would [62:22] get a kind of a blur but after doing the [62:24] rectification transformation you see [62:26] that the back mirror gets really crisp [62:30] so once you do this uh this improves the [62:32] performance quite a bit [62:34] so here are some of the results so on [62:36] the left we are seeing what we had [62:38] before and on the right we're now seeing [62:39] significantly improved predictions [62:41] coming directly out of the neural net [62:42] this is a multi-camera network [62:44] predicting directly in vector space and [62:46] you can see that it's basically night [62:48] and day uh [62:49] you can actually drive on this and uh [62:51] this took some time and some engineering [62:54] and incredible work from the ai team to [62:56] actually get this to work and deploy and [62:57] make it efficient in the car [63:01] this also improved a lot of our object [63:03] detection so for example here in this [63:05] video i'm showing single camera [63:06] predictions in orange and multi-camera [63:08] predictions in blue and basically if you [63:11] if you can't predict these cars if you [63:13] are only seeing a tiny sliver of a car [63:15] so your detections are not going to be [63:16] very good and their positions are not [63:18] going to be good but a multi-camera [63:19] network does not have an issue [63:20] here's another video from a more nominal [63:22] sort of situation and we see that as [63:24] these cars in this tight space cross [63:27] camera boundaries there's a lot of jank [63:29] that enters into the predictions and [63:31] basically the whole setup just doesn't [63:32] make sense especially for very large [63:33] vehicles like this one and we can see [63:35] that the multi-camera networks struggle [63:36] significantly less with these kinds of [63:38] predictions [63:40] okay so at this point we have [63:41] multi-camera networks and they're giving [63:43] predictions directly in vector space but [63:45] we are still operating at every single [63:47] instant in time completely independently [63:49] so very quickly we discovered that [63:51] there's a large number of predictions we [63:52] want to make that actually require the [63:54] video context and we need to somehow [63:55] figure out how to feed this into the net [63:57] so in particular is this car parked or [63:59] not is it moving how fast is it moving [64:01] is it still there even though it's [64:02] temporarily occluded or for example if [64:05] i'm trying to predict the road geometry [64:06] ahead [64:07] it's very helpful to know of the signs [64:09] or the road markings that i saw 50 [64:11] meters ago [64:12] so [64:13] we try to develop [64:15] we try to insert video modules into our [64:16] neural network architecture and this is [64:18] kind of one of the solutions that we've [64:19] converged on [64:20] so we have the multi-scale features as [64:21] we had them from before and what we are [64:23] going to now insert is a feature queue [64:25] module that is going to cache some of [64:27] these features over time [64:28] and then a video module that is going to [64:30] fuse this information temporally and [64:33] then we're going to continue into the [64:34] heads that do the decoding [64:36] now i'm going to go into both of these [64:38] blocks one by one also in addition [64:40] notice here that we are also feeding in [64:42] the kinematics this is basically the [64:44] velocity and the acceleration that's [64:45] telling us about how the car is moving [64:47] so not only are not only are we going to [64:49] keep track of what we're seeing from all [64:50] the cameras but also how the car has [64:52] traveled [64:54] so here's the feature queue and the [64:55] rough layout of it we are basically [64:57] concatenating these features um [65:00] over time and the kinematics of how the [65:02] car has moved [65:03] and the positional encodings and that's [65:05] being concatenated encoded and stored in [65:07] a feature queue [65:08] and that's going to be consumed by a [65:09] video module now there's a few details [65:11] here again to get right so in particular [65:13] with respect to the pop and push [65:14] mechanisms and when do you push and how [65:17] and [65:17] especially when do you push basically so [65:20] here's a cartoon diagram illustrating [65:22] some of the challenges here [65:24] there's going to be the ego cars coming [65:26] from the bottom and coming up to this [65:27] intersection here and then traffic is [65:29] going to start crossing in front of us [65:31] and it's going to temporarily start [65:32] occluding some of the cars ahead and [65:34] then we're going to be stuck at this [65:35] intersection for a while and just [65:36] waiting our turn this is something that [65:38] happens all the time and is a cartoon [65:40] representation of some of the challenges [65:41] here [65:42] so number one with respect to [65:44] the feature queue and when we want to [65:46] push into a queue obviously we'd like to [65:48] have some kind of a time-based queue [65:49] where for example we enter the features [65:51] into the queue say every 27 milliseconds [65:53] and so if a car gets temporarily [65:55] occluded then the neural network now has [65:57] the power to be able to look and [65:59] reference the memory in time and and [66:02] learn the association that hey even [66:03] though this thing looks occluded right [66:04] now there's a record of it in my [66:07] previous features and i can use this to [66:08] still make a detection [66:10] so that's kind of like the more obvious [66:11] one but the one that we also discovered [66:13] is necessary in our case is for example [66:16] suppose you're trying to make [66:17] predictions about the road surface and [66:19] the road geometry ahead and you're [66:21] trying to predict that i'm in a turning [66:23] lane and the lane next to us is going [66:25] straight [66:26] then [66:28] it's really necessary to know about the [66:29] line markings and the signs and [66:30] sometimes they occur a long time ago and [66:33] so if you only have a time-based queue [66:35] you may forget the features while you're [66:37] waiting at your red light so in addition [66:39] to a time-based q we also have a [66:40] space-based view so we push every time [66:42] the car travels a certain fixed distance [66:45] so some of these details actually can [66:47] matter quite a bit and so in this case [66:48] we have a time based key and a space [66:50] base key to cache our features and that [66:52] continues into the video module [66:54] now for the video module we looked at a [66:56] number of possibilities of how to fuse [66:58] this information temporally so we looked [67:00] at three-dimensional convolutions [67:01] transformers axial transformers in an [67:04] effort to try to make them more [67:05] efficient recurrent neural networks over [67:06] a large number of flavors [67:08] but the one that we actually like quite [67:10] a bit as well and i want to spend some [67:11] time on is a spatial recurrent neural [67:13] network video module and so what we're [67:15] doing here is [67:17] because of the structure of the problem [67:19] we're driving on two-dimensional [67:20] surfaces we can actually organize the [67:22] hidden state into a two-dimensional [67:23] lattice and then as the car is driving [67:25] around we update only the parts that are [67:27] near the car and where the car has [67:29] visibility so as the car is driving [67:31] around we are using the kinematics to [67:33] integrate the position of the car in the [67:35] hidden features [67:37] grid and we are only updating the rnn at [67:40] the points where where we have uh that [67:42] are nearby us sort of [67:45] so here's an example of what that looks [67:47] like [67:50] here what i'm going to show you is the [67:51] car driving around [67:53] and [67:53] we're looking at the hidden state of [67:56] this rnn [67:59] and these are different channels in the [68:00] hidden state so you can see that this is [68:03] after optimization and training this [68:04] neural net you can see that some of the [68:06] channels are keeping track of different [68:07] aspects of the road like for example the [68:09] centers of the road the edges the lines [68:11] the road surface and so on [68:14] here's another cool video of this so [68:16] this is looking at the mean of the first [68:18] 10 channels in the hidden state for [68:19] different [68:21] traversals of different intersections [68:23] and all i want you to see basically is [68:25] that there's cool activity as the [68:27] recurrent neural network is keeping [68:28] track of what's happening at any point [68:29] in time and you can imagine that we've [68:31] now given the power to the neural [68:32] network to actually selectively read and [68:34] write to this memory so for example if [68:36] there's a car right next to us and is [68:37] occluding some parts of the road then [68:39] now the network has a has the ability to [68:41] not write to those locations but when [68:43] the car goes away and we have a really [68:44] good view then the recurring neural net [68:46] can say okay we have very clear [68:48] visibility we definitely want to write [68:49] information about what's in that part of [68:51] space [68:53] here's a few predictions that show [68:56] what this looks like [68:58] so here we are making predictions about [68:59] the road boundaries in red intersection [69:01] areas in blue [69:02] road centers and so on so we're only [69:04] showing a few of the predictions here [69:06] just to keep the visualization clean [69:08] um [69:10] and yeah this is this is done by the [69:12] spatial rnn and this is only showing a [69:15] single clip a single traversal but you [69:17] can imagine there could be multiple [69:19] trips through here and basically number [69:20] of cars a number of clips could be [69:22] collaborating to build this map [69:23] basically and effectively an hd map [69:25] except it's not in a space of explicit [69:28] items it's in a space of features of a [69:30] recurrent neural network which is kind [69:31] of cool i haven't seen that before [69:35] the video networks also improved our [69:38] object detection quite a bit so in this [69:39] example i want to show you a case where [69:41] there are two cars [69:43] over there and one car is going to drive [69:45] by and occlude them briefly so look at [69:47] what's happening with the single frame [69:48] and the video predictions as the cars [69:50] pass in front of us [69:55] yeah so that makes a lot of sense so a [69:58] quick [69:59] playthrough through what's happening [70:01] when both of them are in view the [70:02] predictions are roughly equivalent [70:04] and you are seeing multiple orange boxes [70:06] because they're coming from different [70:07] cameras [70:10] when they are occluded the single frame [70:12] networks drop the detection but the [70:13] video module remembers it and we can [70:15] persist the cars and then when they are [70:17] only partially occluded the single frame [70:19] network is forced to make its best guess [70:22] about what it's seeing and it's forced [70:23] to make a prediction and it makes a [70:25] really terrible prediction but the video [70:26] module knows that there's only a partial [70:29] that you know it has the information and [70:31] knows that this is not a very easily [70:33] visible [70:34] part right now and doesn't actually take [70:35] that into account [70:37] we also saw significant improvements in [70:39] our ability to estimate depth and of [70:40] course especially velocity so here i'm [70:42] showing a clip from our remove the radar [70:44] push where we are seeing the radar depth [70:47] and velocity in green and we were trying [70:49] to [70:50] match or even surpass of course the [70:52] signal just from video networks alone [70:54] and what you're seeing here is in [70:57] orange we are seeing a single frame [70:59] performance [71:00] and in blue we are seeing again video [71:02] modules and so you see that the quality [71:03] of depth is much higher and for velocity [71:06] the orange signal of course you can't [71:07] get velocity out of a single frame [71:08] network so we use uh we just [71:10] differentiate depth to get that but the [71:12] video module actually is basically right [71:13] on top of the radar signal and so we [71:16] found that this worked extremely well [71:18] for us [71:20] so here's putting everything together [71:22] this is what our architectural roughly [71:23] looks like today so [71:25] we have raw images feeding on the bottom [71:28] they go through rectification layer to [71:29] correct for camera calibration and put [71:31] everything into a common uh virtual [71:33] camera we pass them through [71:35] regnets residual networks to process [71:38] them into a number of features at [71:39] different scales we fuse the multi-scale [71:41] information with by fbn this goes [71:43] through transformer module to [71:44] re-represent it into the vector space in [71:46] the output space this feeds into a [71:47] feature queue in time or space that gets [71:50] processed by a video module like the [71:51] spatial rnn and then continues into the [71:53] branching structure of the hydra net [71:55] with trunks and heads for all the [71:56] different tasks [71:58] and so that's the architecture roughly [72:00] what it looks like today and on the [72:01] right you are seeing some of its [72:02] predictions uh sort of visualize both in [72:04] a top-down vector space and also in [72:06] images so definitely uh [72:10] this architecture has definitely [72:11] complexified from just a very simple [72:12] image based single network about three [72:14] or four years ago and continues to [72:16] evolve um it's definitely quite [72:18] impressive now there's still [72:18] opportunities for improvements that the [72:20] team is actively working on for example [72:22] you'll notice that our fusion of time [72:23] and space is fairly late in neural [72:25] network terms so maybe we can actually [72:27] do earlier fusion of space or time and [72:29] do for example cost volumes or optical [72:31] flow-like networks on the bottom [72:33] or for example our outputs are dense [72:35] rasters and it's actually pretty [72:36] expensive to post-process some of these [72:38] dense rasters in the car and of course [72:40] we are under very strict latency [72:41] requirements so this is not ideal we [72:43] actually are looking into all kinds of [72:44] ways of predicting just the sparse [72:46] structure of the road maybe like you [72:48] know point by point or in some other [72:50] fashion that is [72:51] that doesn't require expensive [72:52] post-processing [72:54] but this basically is how you achieve a [72:56] very nice vector space and now i believe [72:58] ashok is going to talk about how we can [73:01] run playing control on top of it [73:04] [Applause] [73:11] thank you andre hi everyone my name is [73:13] ashok i lead the planning and controls [73:15] auto labeling and simulation teams [73:19] so like on dimension the visual networks [73:21] take dense video data and then compress [73:23] it down into a 3d vector space the role [73:26] of the planner now is to consume this [73:27] vector space and get the car to the [73:29] destination while maximizing the safety [73:31] comfort and the efficiency of the car [73:34] even back in 2019 our planet was pretty [73:36] capable driver it was able to stay in [73:38] the lanes make lane changes as necessary [73:40] and take exits off the highway but cdc [73:42] driving is much more complicated [73:45] radley there are [73:47] structured lane lines um vehicles do [73:49] much more free from driving then the car [73:51] has to respond to all of [73:53] curtains and crossing vehicles and [73:55] pedestrians doing funny things [74:00] what is the key problem in planning [74:03] number one the action space is very [74:05] non-convex [74:06] and number two it is high dimensional [74:10] what i mean by non-convex [74:12] is there can be multiple possible [74:15] solutions that can be independently good [74:17] but getting a globally consistent [74:18] solution is pretty tricky so there can [74:20] be pockets of local minima that the [74:22] planning can get stuck into [74:24] and secondly uh the high dimensionality [74:26] comes because the car needs to plan for [74:29] the next 10 to 15 seconds and needs to [74:31] produce the position velocities and [74:32] acceleration uh or this entire window [74:35] this is a lot of parameters to produce [74:37] at runtime [74:39] discrete search methods are really great [74:41] at solving non-context problems because [74:42] they are discrete they can they don't [74:44] get stuck in local minima whereas [74:45] continuous function optimization can [74:47] easily get stuck in local minima and [74:49] produce poor solutions that are not [74:50] great [74:52] on the other end for high dimensional [74:53] problems discrete search sucks because [74:57] of the [74:58] it is discrete it does not use any [75:00] graded information so literally has to [75:01] go and explore each point to know how [75:03] good it is whereas continuous [75:05] optimization use gradient-based methods [75:06] to very quickly go to a good solution [75:10] our solution to this printer problem is [75:12] to break it down hierarchically first [75:14] use a code search method to crunch down [75:16] the uh [75:18] non-convexity and come up with a convex [75:20] corridor and then use continuous [75:21] optimization techniques to make the [75:23] final smooth trajectory [75:26] let's see an example of how the search [75:28] operates [75:30] so here we're trying to do a lane change [75:33] in this case the car needs to do two [75:35] back to back lane changes to make the [75:37] left turn up ahead [75:40] for this the car searches over uh [75:43] different maneuvers um [75:46] so the the first one it searches is a [75:48] lane change that's close by but the [75:51] car breaks pretty harshly so it's pretty [75:53] uncomfortable [75:56] the next maneuver it tries does the lane [75:58] change bit late so it speeds up goes [75:59] beyond the other car goes in front of [76:00] the other cars and find it at the lane [76:02] change but now it risks missing the left [76:04] turn [76:06] we do thousands of such searches in a [76:08] very short time span [76:11] because these are all physics-based [76:12] models these features are very easy to [76:14] simulate and in the end we have a set of [76:16] candidates and we finally choose one [76:18] based on the optimality conditions of [76:20] safety comfort and easily making the [76:21] turn [76:23] so now the car has chosen this path and [76:25] you can see that as the car executes [76:27] this trajectory it pretty much matches [76:29] what we had planned the cyan plot on the [76:31] right side here and that one is the [76:34] actual velocity of the car and the white [76:36] line be underneath it is was the plan so [76:39] we are able to plan for 10 seconds here [76:41] and able to match that [76:43] when you see in hindsight so this is a [76:44] well-made plan [76:48] when driving alongside other agents it's [76:50] important to not just plan for ourselves [76:52] but instead we have to plan for everyone [76:54] jointly and optimize for the overall [76:56] scenes traffic flow [76:59] in order to do this what we do is we [77:00] literally run the autopilot planner on [77:02] every single relevant object in the [77:04] scene [77:05] here's an example of why that's [77:06] necessary [77:09] this is an auto corridor i'll let you [77:10] watch the video for a second [77:25] yeah that was autopilot driving an auto [77:27] corridor going around parked cars cones [77:29] and poles uh here there's a 3d view of [77:31] the same thing [77:32] the oncoming car arrives now and [77:34] autopilot slows down a little bit but [77:36] then realizes that we cannot yield to [77:37] them because we don't have any space to [77:39] our side but the other car can heal to [77:41] us instead so instead of just blindly [77:43] breaking here all of our reasons about [77:45] that car uh [77:47] has low enough velocity that they can [77:48] pull over and should yield to us because [77:50] we cannot yield to them and assertively [77:51] makes progress [77:54] a second oncoming car arrives now [77:57] this vehicle has higher velocity [77:59] and like i said earlier we literally run [78:01] the autopilot planner for the other [78:02] object so in this case we run the [78:04] printer for them that object's plan now [78:06] goes around their part their site's [78:08] parked cars and then after they pass the [78:10] parkours goes back to the right side of [78:12] the road for them [78:14] since we don't know what's in the mind [78:15] of the driver we actually have multiple [78:17] possible features for this car here one [78:20] feature is shown in red the other one is [78:21] shown as green [78:22] the green one is a plan that yields to [78:24] us but since this object's velocity and [78:26] acceleration are pretty high we don't [78:28] think that this person is going to yield [78:29] to us and they are actually going to go [78:30] around these sparked cars [78:32] so autopilot decides that okay i have [78:34] space here uh this person is definitely [78:35] going to come so i'm going to pull over [78:38] so as autopilot is pulling over we [78:40] notice that that car has chosen to yield [78:42] to us based on their yaw rate and their [78:44] acceleration and autopilot immediately [78:46] changes his mind and continues to make [78:48] progress [78:49] this is why we need to plan for everyone [78:51] because otherwise we wouldn't know that [78:52] this person is going to go around the [78:53] other park cars [78:55] and come back to their site if you [78:56] didn't do this autopilot would be too [78:58] timid and it would not be a practical [78:59] self-driving car [79:03] so now we saw how the search and [79:04] planning for other people set up [79:06] convicts valley uh finally we do a [79:08] continuous optimization to produce the [79:10] final trajectory that the planet uh [79:12] needs to take [79:13] here the grave uh thing is the convex [79:16] corridor and we initialize the spline in [79:18] heading and acceleration parameterized [79:20] over the arc length of the plan [79:23] and you can see that the [79:24] countermeasuration continuously makes [79:25] fine-grained changes to reduce all of [79:27] its cost some of the costs for example [79:29] are distance from obstacles [79:31] traversal time and comfort for comfort [79:34] you can see that the lateral [79:35] acceleration plots on the right have [79:37] nice trapezoidal shapes uh it's going to [79:39] come up yeah here on the right side the [79:40] green plot um that's a nice trapezoidal [79:43] shape and if you record on a human [79:44] trajectory this is pretty much how it [79:45] looked like the lateral jerk is also [79:47] minimized [79:49] so in summary we do a search for both us [79:51] and everyone else in the scene uh we set [79:53] up a context corridor and then optimize [79:55] for a smooth path [79:56] together these can do some really neat [79:58] things like [79:59] shown above [80:02] but driving looks a bit different in [80:03] other places like where i grew up from [80:05] um [80:07] it's very much more unstructured [80:10] cars and pedestrians cutting each other [80:12] arch braking honking um [80:14] it's a crazy world um [80:16] we can try to scale up these methods uh [80:18] but it's going to be really difficult to [80:20] efficiently solve this at runtime what [80:21] we instead want to do is use learning [80:23] based methods to efficiently solve them [80:25] um and i want to show why this is true [80:28] um so we're going to go from this [80:29] complicated problem to a much simpler [80:31] toy parking problem but still [80:32] illustrates the core of the issue [80:35] here this is a parking lot the ego car [80:38] is in blue and needs to park in the [80:40] green parking spot here so it needs to [80:41] go around the curbs the park cars and [80:44] the cones shown in orange here [80:46] let's do this simple bass line it's a [80:48] star the standard algorithm that uses a [80:50] ladder space search um and and the [80:52] heuristic here is a distance uh the [80:54] euclidean distance to the goal [80:56] so you can see that it directly shoots [80:58] towards the goal but very quickly gets [81:00] trapped in a local minima and it [81:01] backtracks from there [81:03] and then searches a different path to [81:04] try to go around this parkour [81:07] eventually it makes progress and gets to [81:09] the goal but ends up using 400 000 nodes [81:12] for making this [81:14] obviously this is a terrible heuristic [81:16] we want to do better than this so [81:18] if you added a navigation route to it [81:20] and has the car to follow the navigation [81:22] route while being close to the goal this [81:24] is what happens [81:27] the navigation route helps immediately [81:28] uh but still when it enter encounters uh [81:31] cones or other obstacles it basically [81:34] that's the same thing as before [81:35] backtracks and then searches that whole [81:37] new path and the support search has no [81:40] idea that these obstacles exist it [81:41] literally has to go there check if it's [81:43] in collision and if it's in collision [81:44] back up [81:46] the navigation heuristic helped but [81:48] still this took 22 000 notes [81:51] we can design more and more of these [81:53] heuristics to help the search make go [81:55] faster and faster but it's really [81:57] tedious and hard to design a globally [82:00] optimal heuristic [82:02] even if you had a distance function from [82:03] the cones that guided the search this [82:05] would not this would not be this would [82:06] only be effective for the single cone [82:08] but what we need is a global global [82:09] value function so instead of what we [82:11] want to use is neural networks to give [82:13] this heuristic for us [82:15] the visual networks produces vector [82:17] space and we have cars moving around in [82:19] it this basically looks like a atari [82:20] game [82:21] and it's a multiplayer version uh so we [82:24] can use techniques such as mu0 alpha [82:26] zero et cetera that was used to solve go [82:28] and other authority games to solve the [82:29] same problem so we're working on neural [82:32] networks that can produce state and [82:33] action distributions that can then be [82:35] plugged into multicolored research with [82:37] various cost functions [82:38] some of the cost functions can be [82:39] explicit cost functions like distance [82:41] like collisions comfort traversal time [82:43] etc but they can also be [82:45] interventions from the actual manual [82:46] driving events [82:49] we train such a network for this simple [82:50] parking problem so here again same [82:52] problem let's see how mcts researched us [83:01] so here you notice that the [83:03] planet is basically able to in one shot [83:05] make progress towards the goal uh to [83:07] notice that this is not even using a [83:09] navigation heuristic just given the [83:11] scene [83:12] the planet is able to go directly [83:14] towards the goal all the other options [83:15] you're seeing are possible options it's [83:17] not choosing any of them just using the [83:19] option that directly takes it towards [83:20] the goal [83:22] the reason is that the neural network is [83:23] able to absorb the global context of the [83:25] scene and then produce a value function [83:26] that effectively guides it towards the [83:28] global minima as opposed to getting [83:30] certain any local minima [83:32] so this only takes 288 notes and several [83:34] orders of magnitude less than what was [83:36] done in the a star with the equilibrium [83:38] distance heuristic [83:40] so this is what a final architecture is [83:42] going to look like the vision system is [83:44] going to crush down the dense video data [83:46] into a vector space [83:47] uh it's going to be consumed by both an [83:49] explicit planner and a neural network [83:50] planner in addition to this the network [83:52] panel can also consume intermediate [83:53] features of the network [83:55] together this produces a trajectory [83:56] distribution [83:58] and it can be optimized end to end both [84:00] with explicit cost functions and human [84:02] intervention and other imitation data [84:05] this then goes into explicit planning [84:06] function that [84:07] does whatever is easy for that and [84:09] produces the final steering and [84:10] acceleration commands for the car [84:15] with that we need to now explain how we [84:18] train these networks and for training [84:19] these networks we need uh large data [84:21] sets [84:22] um and wait i'm ready to speak briefly [84:24] about manual labeling [84:34] yeah so the data the story of data sets [84:36] is critical of course so far we've [84:37] talked only about neural networks but [84:39] neural networks only establish an upper [84:41] bound on your performance many of these [84:43] neural networks uh they have hundreds of [84:45] millions of parameters and these [84:46] hundreds of millions of parameters they [84:48] have to be set correctly [84:50] if you have a bad setting of parameters [84:51] it's not going to work so neural [84:53] networks are just an upper bound you [84:54] also need massive data sets to actually [84:56] train the correct algorithms inside them [84:58] now in particular i mentioned we want [85:00] data sets directly in the vector space [85:02] and so the really the question becomes [85:03] how can you accumulate because our [85:05] networks have hundreds of millions of [85:06] parameters how do you accumulate [85:08] millions and millions of vector space [85:10] examples that are clean and diverse [85:12] to actually train these neural networks [85:13] effectively [85:15] now so there's a story of data sets and [85:17] how they've evolved on the side of uh [85:20] all of the models and developments that [85:22] we've achieved [85:24] now in particular when i joined roughly [85:26] four years ago we were working with a [85:27] third party to obtain a lot of our data [85:29] sets now unfortunately we found very [85:31] quickly that working with a third party [85:32] to get data sets for something this [85:34] critical was just not going to cut it [85:36] the latency of working with a third [85:37] party was extremely high and honestly [85:40] the quality was not amazing and so in [85:42] the spirit of full vertical integration [85:44] at tesla we brought all of the labeling [85:46] in-house and so over time we've grown [85:49] more than one thousand person uh data [85:51] labeling org that is full of [85:53] professional labelers who are working [85:55] very closely with the engineers so [85:57] actually they're here in the u.s and [85:58] co-located with the engineers here in [86:00] the area as well and so we work very [86:01] closely with them and we also build um [86:04] all of the infrastructure for them from [86:06] scratch ourselves so we have a team we [86:08] are going to meet later today [86:10] that develops and maintains all of this [86:11] infrastructure for data labeling and so [86:13] here for example i'm showing some of the [86:14] screenshots of some of the latency [86:16] throughput and quality statistics that [86:18] we maintain about all of the labeling [86:20] workflows and the individual people [86:22] involved and all the tasks and how the [86:24] numbers of labels are growing over time [86:28] so we found this to be quite critical [86:30] and we're very proud of this [86:32] now in the beginning roughly three or [86:34] four years ago most of our labeling was [86:35] in image space and so you can [86:38] imagine that this is taking quite some [86:39] time to annotate an image like this and [86:41] this is what it looked like [86:43] where we are sort of drawing polygons [86:44] and polylines on top of [86:46] on top of these single individual images [86:48] as i mentioned we need millions of [86:50] vector space [86:51] labels so this is not going to cut it so [86:53] very quickly we graduated to [86:55] three-dimensional or four-dimensional [86:57] labeling where we are directly labeling [87:00] in vector space not in individual images [87:02] so here what i'm showing [87:04] is a clip and you are seeing a very [87:06] small reconstruction you're about to see [87:08] a lot more reconstructions soon but it's [87:10] very small reconstruction of the ground [87:11] plane on which the car drove and a [87:13] little bit of the point cloud here that [87:15] was reconstructed and what you're seeing [87:17] here is that the labeler is uh changing [87:19] the [87:20] labels directly in vector space and then [87:22] we are reprojecting those changes into [87:24] camera images uh so we're labeling [87:26] directly in vector space and this gave [87:27] us a massive increase in throughput for [87:29] a lot of our labels because you're [87:31] labeled once in 3d and then you get to [87:32] reproject [87:34] but even this we realized was actually [87:37] not going to cut it [87:39] so [87:40] because people and computers have [87:41] different pros and cons so people are [87:43] extremely good at things like semantics [87:44] but computers are very good at geometry [87:47] reconstruction triangulation tracking [87:50] and so really for us it's much more [87:52] becoming a story of how do humans and [87:53] computers collaborate to actually create [87:55] these vector space data sets and so [87:57] we're gonna not talk about auto labeling [87:59] which is some of the infrastructure [88:00] we've developed for labeling these clips [88:02] at scale [88:04] [Applause] [88:08] hi again [88:11] so even though we have lots of human [88:13] laborers the amount of training data [88:15] needed for training with networks [88:16] significantly outnumbers them so we try [88:18] to invest in a massive auto labeling [88:20] pipeline here's an example of how we [88:22] label a single clip a clip is a entity [88:25] that has dense sensor data like videos [88:28] imu data gps odometry etc [88:30] this can be 45 second to a minute long [88:32] these can be uploaded by our own [88:33] engineering cars or from customer cars [88:36] we collect these clips and then send [88:38] them to [88:39] our servers where we run a lot of neural [88:41] networks offline to produce intermediate [88:43] results like segmentation masks depth [88:45] point matching etc this then goes [88:47] through a lot of robotics and a [88:49] algorithm to produce a final set of [88:50] labels that can be used to train the [88:52] networks [88:55] one of the first tasks we want to label [88:57] is the road surface typically we can use [88:59] splines or meshes to represent the road [89:00] surface but those are [89:02] because of the topology restrictions are [89:04] not differentiable and not amenable to [89:06] producing this so what we do instead is [89:08] in the style of neural radiance fields [89:09] work from last year which is quite [89:11] popular so we use an implicit [89:13] representation to represent the road [89:14] surface [89:15] here we are querying xy points on the [89:18] ground and asking for the network to [89:19] predict the height of the ground surface [89:22] along with various semantics such as [89:24] curves lane boundaries road surface [89:26] rival space etc [89:28] so given a single xy we get a z together [89:31] these make a 3d point and they can be [89:33] reprojected into all the camera views [89:36] so we make millions of such queries and [89:38] get lots of points these points are [89:40] reprojected into all the camera views [89:42] in we are showing on the top right here [89:45] one such camera image with all these [89:46] points reprojected [89:48] now we can compare this point [89:51] reprojected point with the image space [89:54] prediction of the segmentations [89:56] and jointly optimizing this or all the [89:58] camera views was across space and time [90:00] produced an excellent reconstruction [90:03] here's an example of how that looks like [90:05] so here this is an optimized road [90:06] surface that reproduction to the eight [90:08] cameras that the car has and across all [90:10] of time and you can see how it's [90:12] consistent across both space and time [90:17] so a single car driving through some [90:19] location can sweep out some patch around [90:21] the trajectory using this technique [90:25] but we don't have to stop there [90:27] so here we collect collected different [90:29] clips [90:30] from the same location from different [90:31] cars maybe [90:33] and each of them sweeps out some part of [90:35] the road [90:37] cool thing is we can bring them all [90:38] together into a single giant [90:40] optimization so here these 16 different [90:42] trips are organized using [90:44] aligned using various features such as [90:46] roadages lane lines all of them should [90:48] agree with each other and also agree [90:50] with all of their image space [90:51] observations [90:52] together this is this produces an [90:54] effective way to label the road surface [90:56] not just where the car drove but also in [90:58] other locations that it hasn't driven it [91:01] again the point of this is not to just [91:02] build hd maps or anything like that it's [91:04] only to label the clips through these [91:05] intersections so we don't have to [91:07] maintain them forever as long as the [91:09] labels are consistent with the videos [91:11] that they were collected at [91:14] optionally then humans can come on top [91:15] of this and clean up any noise or [91:17] additional metadata to make it even [91:19] richer [91:21] we don't have to stop at just the road [91:23] surface we can also arbitrarily [91:25] reconstruct 3d static obstacles um here [91:28] this is a reconstructed 3d point cloud [91:30] from our cameras [91:33] the main innovation here is the density [91:34] of the point cloud typically these [91:36] points require texture [91:38] to form associations from one frame to [91:40] the next frame but here we are able to [91:42] produce these points even on textual [91:43] surfaces like the road surface or walls [91:46] and this is really useful to annotate [91:47] arbitrary obstacles that [91:50] we can see on the scene in the world [91:55] one more cool advantage of doing all of [91:56] this on server on the servers offline is [91:59] that we have the benefit of hindsight [92:01] this is a super useful hack because uh [92:04] say in the car then the network needs to [92:06] produce the velocity it just has to use [92:07] the historical information and guess [92:09] what the velocity is [92:11] but here [92:12] we can look at both the history but also [92:14] the future and basically cheat and get [92:17] the correct answer uh of the kinematics [92:19] like velocity acceleration etc [92:22] one more advantage is that we can have [92:23] different tracks but we can switch them [92:25] together that even through occlusions [92:26] because we know the future we have [92:28] future tracks we can match them and then [92:29] associate them so here you can see the [92:31] pedestrians on the other side of the [92:33] road are persisted even through multiple [92:35] occlusions by these cars [92:37] this is really important for the planner [92:38] because the planner needs to know if [92:40] it's if it saw someone it still needs to [92:42] account for them even then they are [92:44] occluded so this is a massive advantage [92:49] combining everything together we can [92:51] produce these amazing data sets that [92:52] annotate all of the road texture [92:55] or the static objects and all of the [92:57] moving objects even through occlusions [92:59] producing excellent kinematic [93:01] labels [93:02] all you can see how the cards turn [93:04] smoothly produce really smooth labels or [93:07] the pedestrians are consistently tracked [93:08] the park cars uh [93:10] obviously zero velocity so we can also [93:12] know that they are parked [93:13] so this is huge for us [93:16] this is one more example of the same [93:17] thing [93:19] you can see how everything is consistent [93:20] we want to produce a million such uh [93:23] labeled [93:24] clips and train our video [93:26] multicam video networks with such large [93:28] data set and really crush this problem [93:31] we want to get the same view that's [93:32] consistent that we're seeing here in the [93:34] car [93:36] we started our first exploration of this [93:38] with the remove the data project uh we [93:40] removed it in a very top a short time [93:42] span i think within three months [93:44] in the early days of the network we [93:45] noticed for example in low security [93:47] conditions the network can suffer [93:49] understandably because obviously this [93:51] truck just dumped a bunch of snow on us [93:52] and it's really hard to see but we [93:54] should still remember that this car was [93:56] in front of us [93:57] but our networks early on did not do [93:59] this because of the lack of data in such [94:01] conditions [94:04] so what we did we added the fleet to [94:05] produce lots of similar clips [94:07] and the fleet responded it did [94:10] so it produces um [94:14] is it play [94:16] yeah it produces lots of video clips [94:19] where shit's falling out of all other [94:21] vehicles [94:22] and we send this through our auto living [94:24] pipeline that was able to label 10k [94:26] clips in within a week this would have [94:28] taken several months with humans [94:30] labeling every single clip here [94:32] so we did this for 200 of different [94:35] conditions and we were able to very [94:37] quickly create large data sets and [94:38] that's how we were able to remove this [94:41] so once we train the networks with this [94:42] data [94:43] you can see that it's totally working [94:45] and keeps the memory that the subject [94:47] was there [94:48] and provides this [94:52] so finally we wanted to actually get a [94:53] cyber truck into a data set [94:55] for the remote radar [94:57] can you all guess where we got this clip [94:58] from i'll give you a moment [95:03] someone said it yes yes it's rendered [95:05] it's our simulation [95:07] it was hard for me to tell initially and [95:09] i if i may if i may say so myself it [95:11] looks pretty it looks very pretty um [95:15] so yeah in addition to auto labeling we [95:17] also invest heavily in using simulation [95:19] for labeling our [95:20] data [95:22] so this is the same scene as [95:24] seen before [95:25] but from a different camera angle [95:29] so a few things that i wanted to point [95:30] out for example the ground surface it's [95:33] not a plane asphalt there are lots of [95:35] cars and cracks and tower seams [95:38] there's some patchwork done on top of it [95:41] vehicles move realistically the truck is [95:42] articulated uh even goes over the curb [95:44] and makes a wide turn um the other cars [95:47] behave smartly they avoid collisions go [95:49] around cars uh and also smooth and [95:51] actual great smooth uh brake and [95:53] accelerate smoothly [95:56] the car here with the logo on the top is [95:58] autopilot actually auto part is driving [95:59] that car and it's making unproductive [96:00] left hand [96:03] and since it's a simulation it starts [96:05] from the vector space so it has perfect [96:06] labels [96:08] here we show a few of the labels that we [96:09] produce [96:10] these are vehicle cuboids with [96:12] kinematics [96:13] depth [96:14] surface normals segmentation but [96:17] andre can name a new task that he wants [96:19] next week and we can very quickly [96:20] produce this because we already have a [96:22] vector space and we can write the code [96:24] to produce these labels very very [96:25] quickly [96:28] so when does simulation help [96:30] it helps number one when the data is [96:31] difficult to source as large as our [96:33] fleet is it can still be hard to get [96:36] some crazy scenes like this couple and [96:37] their dog running on the highway while [96:39] there are other high-speed cars around [96:43] this is a pretty rare scene i'd say but [96:45] still can happen [96:46] and autopilot still needs to handle it [96:48] when it happens [96:50] when data is difficult to label um there [96:52] are hundreds of pedestrians crossing the [96:53] road this could be a mountain downtown [96:56] people crossing the road it's going to [96:57] take several hours for humans to label [96:58] this clip and even for automatic [97:00] labeling algorithms this is really hard [97:01] to get the association right and it can [97:03] produce like bad velocities but in [97:05] simulation this is trivial because you [97:06] already have the objects you just have [97:08] to spit out the cuboids and the [97:09] velocities [97:11] and also finally when we introduce [97:12] closed loop behavior where the cars [97:14] needs to be in a determining situation [97:16] or the data depends on the actions this [97:19] is pretty much the only way to get it [97:20] reliably [97:23] all this is great [97:24] what's needed to make this happen [97:28] number one accurate sensor simulation [97:31] again the point of the simulation is not [97:32] to just produce pretty pictures [97:34] it needs to produce what the camera in [97:36] the car would see and other sensors [97:38] would see so here we are stepping [97:40] through different exposure settings of [97:41] the real camera on the left side and the [97:43] simulation on the right side [97:46] we're able to pretty much match what the [97:47] real cameras do [97:50] in order to do this we had to model a [97:52] lot of the properties of the camera in [97:54] our sensor simulation um starting from [97:57] sensor noise motion blur optical [97:59] distortions even headlight transmissions [98:02] uh [98:03] even like diffraction patterns of the [98:05] windshield etcetera [98:06] we don't use this just for the autopilot [98:08] software we also use it to make hardware [98:10] decisions such as lens design camera [98:11] design sensor placement even headlight [98:14] transmission properties [98:19] second we need to render the visuals uh [98:22] in a realistic manner you cannot have [98:24] what in the game industry called jaggies [98:27] these are aliasing artifacts that are [98:29] dead giveaway that this is simulation we [98:31] don't want them so we go through a lot [98:32] of paints to produce a nice special [98:35] temple a special temporal anti-aliasing [98:38] we also are working on neural rendering [98:40] techniques to make this even more [98:41] realistic [98:47] yeah in addition we also use ray tracing [98:49] to produce realistic lighting and global [98:50] illumination okay that's the last of the [98:52] cop cars i think [98:56] we obviously cannot have uh really just [98:58] four or five cars because the network [98:59] will easily overfit because it knows the [99:01] sizes um so we need to have realistic [99:03] assets like the moose on the road here [99:05] um [99:06] we have thousands of assets in our [99:08] library and they can wear different [99:10] shirts and actually can move [99:11] realistically so this is really cool [99:13] we also have a lot of different [99:14] locations mapped and created uh to [99:16] create these uh sim environments [99:19] we have actually 2000 miles of uh [99:22] road built and this is almost the length [99:24] of the roadway from the east coast the [99:25] west coast of the united states which i [99:27] think is pretty cool [99:28] in addition we have built efficient [99:29] tooling to build several miles more on a [99:32] single day on a for a single artist [99:36] but this is just tip of the iceberg [99:38] actually most of the data that we use to [99:40] train is created procedurally using [99:43] algorithms as opposed to artists making [99:45] these simulation scenarios [99:47] so these are all procedurally created [99:49] roads with lots of parameters such as [99:51] curvature [99:52] various varying trees cones poles cars [99:55] with different velocities and the [99:56] interaction produce an endless stream of [99:58] data for the network [100:00] but a lot of this data can be boring [100:01] because the network might already get it [100:02] correct so what we do is we use also ml [100:04] based techniques to basically put up the [100:06] network to see where it's failing at and [100:08] create more data around the failure [100:10] points of the network so this is in [100:12] closed loop trying to make the network [100:13] performance be better [100:18] we don't want to stop there so actually [100:19] we want to create recreate any failures [100:22] that happens to the autopilot in [100:24] simulation so that we can hold autopilot [100:25] to the same bar from then on [100:28] so here on the left side you are seeing [100:29] a real clip that was collected from a [100:31] car [100:33] it then goes through our auto labeling [100:34] pipeline to produce a 3d reconstruction [100:36] of the scene along with all the moving [100:38] objects [100:40] with this combined with the original [100:42] visual information we recreate the same [100:44] scene synthetically and create a [100:46] simulation scenario entirely out of it [100:48] so and then when we replay autopilot on [100:50] it autopilot can do entirely new things [100:52] and we can form new worlds new outcomes [100:54] from the original failure this is [100:56] amazing because we really don't want [100:58] autopilot to fail and when it fails we [100:59] want to capture it and keep it to that [101:02] bar [101:07] not just that we can actually take the [101:08] same approach that we said earlier and [101:10] take it one step further [101:11] we can use neural rendering techniques [101:14] to make it look even more realistic [101:16] so we take the original original video [101:18] clip [101:19] we create a synthetic uh simulation from [101:21] it [101:22] and then apply neural rendering [101:23] techniques on top of it and it produces [101:24] this which looks amazing in my opinion [101:26] because this one is very realistic and [101:28] looks almost like it was captured by the [101:30] actual cameras [101:31] they saw results from last night because [101:33] it was cool and we wanted to present it [101:35] but yeah [101:37] yeah i'm very excited for what sim can [101:38] achieve [101:39] but this is not all because [101:41] network's trained in the car already [101:42] used simulation data we used 300 million [101:45] images with almost half a billion labels [101:48] and we want to crush down all the tasks [101:49] that are going to come up for uh the [101:51] next several months [101:53] with that i invite milan to see uh to [101:55] explain how we scale these operations [101:57] and really build a label factory [101:59] and spit out millions of labels [102:02] [Applause] [102:09] all right thanks ashok [102:11] hey everyone i'm milan i'm responsible [102:13] for the integration of our neural [102:15] networks in the car and for most of our [102:17] neural network training and evaluation [102:18] infrastructure [102:20] and so tonight i'd just like to start by [102:22] giving you some perspective into the [102:23] amount of compute that's needed to power [102:25] this type of data generation factory and [102:28] so in the specific context of the push [102:30] we went through as a team here a few [102:32] months ago to get rid of the dependency [102:34] on the radar sensor for the pilot we [102:36] generated over 10 billion labels across [102:39] two and a half million clips [102:41] and so to do that we had to [102:43] scale [102:44] our huge offline neural networks and our [102:46] simulation engine across thousands of [102:48] gpus and just a little bit shy of 20 000 [102:51] cpu cars on top of that we also included [102:54] over 2000 actual autopilot full [102:56] self-driving computers in the loop with [102:58] our simulation engine [103:00] and that's our smallest compute cluster [103:07] so i'd like to give you some [103:09] idea of what it takes to take our neural [103:10] networks and [103:12] move them in the car [103:14] and so the [103:15] the two main constraints that we're [103:17] working on there here [103:18] are mostly latency and frame rate [103:22] which are very important for safety but [103:24] also to get proper estimates of [103:26] acceleration and velocity of our [103:27] surroundings [103:29] and so the meat of the problem really is [103:31] around uh the ai compiler that we write [103:34] and extend here within the group that [103:36] essentially maps the compute operations [103:38] for my apart torch model [103:39] to a set of dedicated and accelerated uh [103:43] pieces of hardware and we do that while [103:46] figuring out a schedule that's optimized [103:48] for throat put [103:49] while working on their severe sram [103:51] constraints [103:53] and so by the way we're not doing that [103:55] just on one engine but on across two [103:57] engines on the autopilot computer and [103:58] the way we use those engines here at [104:00] tesla is such that at any given time [104:02] only one of them will actually output [104:04] control commands to the vehicle while [104:06] the other one is used as an extension of [104:08] compute [104:09] but those roles are interchangeable both [104:11] at the hardware and software level [104:14] so how do we interact quickly together [104:16] as a group through these ai development [104:18] cycles well first we have been scaling [104:21] our [104:22] capacity to evaluate our software neural [104:24] network dramatically over the past few [104:25] years and today we're running over a [104:27] million evaluations per week on any code [104:30] change [104:31] that the team is producing and those [104:33] evaluations runs on over three thousand [104:36] actual footstar driving computers that [104:37] are hooked up together in a dedicated uh [104:40] cluster [104:42] and so on top of this we've been [104:43] developing really cool uh debugging [104:45] tools and so here is a video of one of [104:47] our tools which is uh helping developers [104:49] uh iterate through the development of [104:52] neural networks and comparing live the [104:54] outputs from different revisions of a [104:56] same neural network model as reiterating [104:59] life through a video clips [105:02] and so last but not least uh we've been [105:05] scaling our neural network training [105:07] compute dramatically over the past few [105:09] years and today we're barely shy of 10 [105:12] 000 gpus which just to give you some [105:14] sense in terms of number of gpu is more [105:17] than the top five publicly known [105:18] supercomputers in the world [105:20] but that's not enough and so i'd like to [105:22] invite ganesh to talk about the next [105:24] steps [105:27] [Applause] [105:34] thank you milan [105:40] my name is ganesh [105:41] and i lead project dojo [105:44] it's an honor to present this project on [105:46] behalf of the multidisciplinary tesla [105:49] team that is working on this project [105:53] as you saw from milan [105:55] there's an insatiable demand for speed [105:59] as well as capacity for neural network [106:01] training and elon prefetched this and a [106:04] few years back he asked us to design a [106:06] super fast training computer and that's [106:08] how we started project dojo [106:12] our goal is to achieve best ai training [106:15] performance and support all these larger [106:18] more complex models that andres team is [106:21] dreaming of and be power efficient and [106:24] cost effective at the same time [106:27] so we thought about how to build this [106:29] and we came up with [106:31] a distributed compute architecture [106:33] after all all the training computers out [106:36] there are distributed computers in one [106:38] form or the other they have compute [106:40] elements in the box out here connected [106:43] with some kind of network in this case [106:46] it's a two-dimensional network but it [106:48] could be any different network cpu gpu [106:51] accelerators all of them have compute [106:53] little memory [106:55] and network but one thing which is [106:58] common trend amongst this is it's easy [107:01] to scale [107:02] the compute [107:03] it's very difficult to scale up [107:05] bandwidth and extremely difficult to [107:08] reduce latencies and you'll see how our [107:10] design point catered to that how our [107:13] philosophy [107:15] addressed these aspects of traditional [107:17] limits [107:19] for dojo we envisioned a large compute [107:23] plane [107:24] filled with very robust compute elements [107:27] backed with large pool of memory and [107:29] interconnected with very high bandwidth [107:31] and low latency fabric and in a 2d mesh [107:34] format and on to this for extreme scale [107:38] big neural networks will be partitioned [107:41] and mapped to extract different [107:43] parallelism model graph data parallelism [107:47] and then in neural compiler [107:50] of ours will [107:52] exploit spatial and [107:54] temporal locality such that it can [107:57] reduce communication footprint to local [107:59] zones and reduce global communication [108:01] and if we do that [108:03] our bandwidth utilization can keep [108:05] scaling with the plane of compute that [108:08] we desire out here [108:12] we wanted to attack this all the way [108:15] top to the bottom of the stack [108:18] and remove any bottlenecks at any of [108:20] these levels [108:22] and let's start this journey in an [108:24] inside out fashion starting with the [108:26] chip [108:28] as i described [108:29] chips have compute elements our smallest [108:32] entity or scale is called a training [108:34] node [108:35] and the choice of this node is [108:37] very important to ensure seamless [108:40] scaling [108:41] if you go too small it will run fast but [108:44] the overheads of synchronization [108:47] will end software will dominate if you [108:50] pick it too big [108:52] it will have complexities in [108:54] implementation in the real hardware and [108:56] ultimately run into memory bottleneck [108:58] issues [108:59] because we wanted to address [109:03] we wanted to address latency and [109:05] bandwidth as our primary optimization [109:07] point let's see how we went about doing [109:10] this [109:11] what we did was we picked the farthest [109:13] distance a signal could traverse in a [109:16] very clock very high clock cycle in this [109:18] case two gigahertz plus and we drew a [109:21] box around it this is the [109:23] smallest latency that a signal can [109:26] traverse one cycle at a very high [109:28] frequency and then we filled up the box [109:31] with wires to the brink [109:33] this is the highest bandwidth you can [109:35] feed the box with and then we added [109:37] machine learning compute underneath and [109:40] then a large pool of sram [109:42] and last but not the least a [109:44] programmable [109:45] core to control [109:47] and this gave us our high performance [109:50] training node [109:52] what this is is a 64-bit superscalar cpu [109:55] optimized around matrix multiply units [109:58] and vector cmd [110:00] it supports floating point 32 b float 16 [110:04] and a new format cfp8 configurable fp8 [110:09] and it is backed by [110:11] one and a quarter megabyte of fast ecc [110:14] protected sram and the low latency high [110:17] bandwidth fabric that we designed [110:20] this might be our smallest entity of [110:22] scale but it packs a big punch [110:25] more than one teraflop of compute in our [110:28] smallest entity of scale [110:31] so let's look at the architecture of [110:33] this [110:34] the computer architects out here may [110:36] recognize this this is a pretty capable [110:38] architecture as soon as you see this [110:41] it is a super scalar in order cpu [110:44] with four wide vector and two wide vect [110:47] two wide uh four wide scaler and two [110:49] wide vector pipes we call it in order [110:53] although the vector and the scalar pipes [110:55] can go out of order but for the purest [110:57] out there we still call it in order and [111:00] it also has four-way multi-threading [111:02] this increases utilization because we [111:04] could do compute and data transfers [111:06] simultaneously and our custom isa which [111:10] is the instruction set architecture is [111:11] fully optimized for machine learning [111:13] workloads it has features like transpose [111:17] gather [111:18] link traversals broadcast just to name a [111:21] few [111:23] and even in the physical realm we made [111:25] it extremely modular such that we could [111:28] start averting these training nodes in [111:31] any direction and start forming the [111:34] compute plane that we envisioned [111:39] when we click together [111:41] 354 of these training nodes we get our [111:44] compute array it's capable of delivering [111:47] teraflops of machine learning compute [111:51] and of course the high bandwidth fabric [111:53] that interconnects these [111:56] and around this compute array we [111:58] surrounded it with [112:00] high speed low power services 576 [112:04] of them to to enable us to have extreme [112:08] i o bandwidth coming out of this chip [112:11] just to give you a comparison point this [112:14] is more than two times the bandwidth [112:17] coming out of the state-of-the-art [112:19] networking switch chips which are out [112:22] there today and network switch chips are [112:24] supposed to be the gold standards for i [112:27] o bandwidth [112:29] if we put all of it together [112:31] we get [112:33] training optimized chip [112:35] our d1 chip [112:37] this chip is manufactured in seven [112:40] nanometer technology it packs 50 billion [112:43] transistors in a miserly 645 millimeter [112:46] square [112:46] one thing you'll notice 100 of the area [112:50] out here is going towards machine [112:52] learning training and bandwidth there is [112:54] no dark silicon there is no legacy [112:56] support this is a pure machine learning [112:59] machine [113:03] and [113:07] this [113:09] is the d1 chip in a flip chip bga [113:12] package [113:13] this was entirely designed [113:15] by tesla team internally all the way [113:19] from the architecture [113:21] to [113:22] gds out and [113:24] package this chip [113:27] is like a [113:28] gpu level compute with a cpu level [113:31] flexibility and twice the network chip [113:35] level i o bandwidth [113:37] if i were to plot [113:40] the i o bandwidth on the vertical scale [113:42] versus teraflops of compute that is [113:45] available in the state-of-the-art [113:47] machine learning chips are there [113:49] including some of the startups you can [113:51] easily see why our design point excels [113:54] beyond power [113:56] now that we had this fundamental [113:58] physical building block [114:02] how to design the system around it [114:04] let's see [114:06] since d1 [114:09] chips can seamlessly connect without any [114:13] glue to each other we just started [114:16] putting them together [114:18] we just put [114:24] 500 000 [114:26] training nodes together to form our [114:28] compute plane [114:30] this is 1500 d1 chips seamlessly [114:34] connected to each other [114:37] and then [114:38] we add [114:40] dojo interface process [114:42] processors on each end [114:44] this is the host bridge to typical hosts [114:47] in the data centers [114:49] it's connected with pci gen4 on one side [114:53] with a high bandwidth fabric to our [114:55] compute plane [114:56] the interface processors provide not [114:58] only the host bridge but high bandwidth [115:01] dram shared memory for the compute plane [115:05] in addition [115:06] the interface processors [115:08] can also allow us to have a higher radix [115:11] network connection [115:14] in order to achieve this compute plane [115:18] we had to come up with a new way of [115:20] integrating these chips together [115:23] and this is what we call as a training [115:26] tile [115:27] this is the unit of scale for our system [115:31] this is a groundbreaking integration of [115:34] 25 known good d1 dies onto a [115:38] fan out wafer process tightly integrated [115:41] such that it preserves the bandwidth [115:43] between them the maximum bandwidth is [115:45] preserved there and in addition [115:48] we generated [115:50] a [115:51] connector a high bandwidth high density [115:54] connector that preserves the bandwidth [115:56] coming out of this training tile [116:00] and this style gives us nine beta flaps [116:03] of compute [116:04] with a massive i o bandwidth coming out [116:07] of it [116:09] this [116:10] perhaps is the biggest organic mcm in [116:13] the chip industry multi-chip module [116:18] it was not easy to design this there [116:20] were no tools that existed all the tools [116:23] were croaking even our compute cluster [116:25] couldn't handle it we had to our [116:27] engineers came up with different ways of [116:30] solving this they created new methods to [116:33] make this a reality [116:36] now that we had our compute plane [116:38] tile [116:40] with high bandwidth ios [116:42] we had to feed it with power [116:44] and here we came up with a new way of [116:47] feeding power vertically [116:50] we created a custom [116:53] voltage regulator module [116:55] that could be reflowed directly directly [116:58] onto this fan out wafer so what did we [117:01] did out here is we got chip [117:04] package and we brought pcb level [117:07] technology of reflow on fan art wafer [117:10] technology this is [117:12] a lot of integration already out here [117:15] but we didn't stop here [117:16] we integrated [117:18] the entire electrical thermal and [117:21] mechanical pieces out here [117:25] to form [117:26] our training tile fully integrated [117:30] interfacing with a 52 volt dc input [117:35] it's unprecedented this is an amazing [117:38] piece of engineering [117:41] our compute plane is completely [117:43] orthogonal [117:45] to power supply [117:46] and cooling [117:48] that makes [117:50] high bandwidth compute planes [117:53] possible [117:59] what it is [118:00] is a nine petaflop [118:03] training tile this becomes our unit of [118:05] scale for our system [118:09] and this [118:12] is real [118:26] i can't believe i'm holding nine [118:28] petaflops out here [118:39] and in fact [118:41] last week we got our first [118:44] functional training tile [118:46] and on a limited limited cooling [118:49] benchtop setup [118:51] we got some networks running [118:54] and i was told andre doesn't believe [118:56] that we could run networks [118:58] till we could run one of his creations [119:01] this is min gpt2 running on dojo [119:05] do you believe it [119:12] next up [119:14] how to form a compute cluster out of it [119:18] by now you must have realized our [119:20] modularity story is pretty strong [119:24] we just put together some tiles we just [119:26] styled together tiles [119:29] a two by three [119:31] tile in a tray makes our training matrix [119:35] and two trays in a cabinet [119:38] give 100 petaflops [119:40] of compute [119:43] did we stop here no [119:47] we just [119:48] integrated seamlessly we broke the [119:51] cabinet walls we integrated these styles [119:54] seamlessly all the way through [119:56] preserving the bandwidth there is no [119:58] bandwidth divert out here there is no [120:00] bandwidth clips all the tiles are [120:02] seamlessly connected with the same [120:05] bandwidth [120:06] and with this [120:12] we have an exopod [120:15] this is [120:17] one extra flop of compute [120:21] in 10 cabinets [120:24] it's more than a million training nodes [120:26] that you saw we paid meticulous [120:28] attention to that training node and [120:30] there are one million nodes out here [120:34] with uniform bandwidth [120:39] not just the hardware [120:41] the software aspects are so important [120:44] to ensure scaling [120:46] and not every job requires a huge [120:49] cluster so we plan for it [120:52] right from the get go [120:55] our compute plane [120:56] can be subdivided can be partitioned [120:59] into [121:00] units called dojo processing unit [121:04] a dpu [121:05] consists of one or more [121:08] d1 chips [121:11] it also has our interface processor and [121:13] one or more hosts [121:17] and this can be scaled up or down as per [121:20] the needs of any algorithm any network [121:23] running on it [121:26] what does the user have to do they have [121:28] to change their scripts minimally [121:33] and this [121:34] is because [121:35] of our strong compiler suite [121:37] it takes care of fine-grained [121:39] parallelism and mapping the pro problems [121:42] of mapping the neural networks very [121:44] efficiently onto our compute plane [121:48] our compiler is [121:51] uses multiple techniques to extract [121:53] parallelism [121:55] it can transform the networks to achieve [121:58] not only fine-grained parallelism using [122:00] data model [122:02] graph parallelism techniques [122:05] it also [122:07] can do optimizations to reduce memory [122:09] footprints [122:12] one thing because of our [122:14] high bandwidth nature of the fabric is [122:17] enabled out here is model parallelism [122:20] could not have been extended to the same [122:22] level as what we can [122:24] it was limited to chip boundaries now we [122:26] can because of our high bandwidth we can [122:29] extend it to training tiles and beyond [122:32] thus [122:33] large networks can be efficiently mapped [122:36] here at low batch sizes and extract [122:39] utilization and new levels of [122:40] performance [122:42] in addition our compiler is capable of [122:45] handling high level dynamic control [122:48] flows like loops if then else etc [122:52] and our compiler engine is just part of [122:55] our entire software suite [122:58] the stack consists of [123:01] a extension to pytarch that ensures the [123:04] same user-level interfaces that ml [123:07] scientists are used to [123:09] and our compiler generates code on the [123:11] fly [123:12] such that it could be reused for [123:14] subsequent execution it has a llvm [123:17] backend that generates the binary for [123:19] the hardware [123:20] and this ensures we can create optimized [123:24] code [123:25] for the hardware without relying on even [123:28] single line of handwritten kernel [123:32] our driver stack [123:34] takes care of the multi-host [123:36] multi-partitioning that you saw [123:39] a few slides back [123:41] and then we also have [123:43] profilers [123:45] and debuggers in our software stack [123:49] so with all this [123:51] we integrated [123:53] in a vertical fashion we broke the [123:56] traditional barriers to scaling [123:59] and that's how [124:00] we got modularity up and down the stack [124:04] to add to new levels of performance [124:07] to sum it all [124:08] this is what it will be it will be a [124:11] fastest ai training computer [124:14] for x the performance at the same cost [124:17] 1.3 x [124:19] better performance per watt that is [124:21] energy saving and 5x smaller footprint [124:25] this will be dojo computer [124:38] we are not done we are [124:40] assembling our first cabinets pretty [124:43] soon [124:44] and we have a whole next generation plan [124:47] already we are thinking about [124:49] 10x more with different aspects that we [124:52] can do all the way from silicon [124:54] to the system again we will have this [124:57] journey again we're recruiting heavily [125:00] for all of these [125:02] areas thank you very much [125:14] and next up [125:15] elon will update us on what's beyond our [125:19] vehicle feet fleet for ai [125:29] [Music] [125:35] oh [125:39] [Music] [126:07] so [126:08] [Music] [126:31] oh [126:36] [Music] [126:51] thank you [126:53] [Applause] [126:59] now unlike unlike dojo obviously that [127:02] was not real [127:04] uh [127:05] so doj is real uh the tesla bot will be [127:07] real [127:08] um but [127:09] basically if you think about what we're [127:11] doing right now with the cars tesla is [127:13] arguably the world's biggest robotics [127:15] company because our cars are [127:17] like [127:17] semi-sentient robots on wheels [127:20] and with [127:22] the [127:23] full self-driving computer essentially [127:24] the the inference engine on the car [127:26] which will keep evolving obviously and [127:29] uh dojo [127:30] and all the [127:33] neural nets recognizing the world [127:35] understanding how to navigate through [127:36] the world [127:37] uh it it kind of makes sense to put that [127:39] onto a humanoid form [127:42] they're also quite good at sensors and [127:44] batteries and [127:46] actuators [127:47] so [127:48] we think we'll probably have [127:50] a prototype sometime next year that [127:56] basically looks like this [127:57] and it's intended to [128:03] be friendly of course [128:06] and uh navigate through a world uh built [128:09] for humans [128:10] and [128:11] uh eliminate dangerous repetitive and [128:13] boring tasks [128:14] um we're setting it such that it is um [128:19] at a mechanical level at a physical [128:20] level you can run away from it [128:26] and and most likely overpower it [128:29] so [128:30] hopefully that doesn't ever happen but [128:33] um you never know [128:36] so it's uh it'll be a you know [128:39] a light [128:40] a light [128:42] yeah five miles an hour you can get run [128:44] faster than that'd be fine [128:46] [Laughter] [128:50] um so [128:51] yes it's a [128:53] round [128:54] uh 5.8 [128:56] um [128:57] has sort of a screen where the head is [128:59] for useful information um but as [129:01] otherwise basically got the autopilot [129:03] system in it so it's got cameras got [129:05] eight cameras and [129:14] full-size driving computer [129:15] and making use of all of the same tools [129:17] that we use in the car [129:19] so [129:20] um i mean things i think that are really [129:22] hard about uh [129:23] having a useful humanoid robot is cannot [129:26] navigate through the world without being [129:28] explicitly trained [129:30] i mean without explicit like uh [129:33] line-by-line instructions [129:35] um [129:36] can you can you talk to it and say you [129:37] know please uh pick up that bolt [129:40] and uh attach it to [129:43] the car with that wrench [129:45] and it should be able to do that [129:48] um [129:49] it should be able to you know please you [129:50] know please go to the store and get me [129:52] the following groceries [129:54] um that kind of thing [129:56] so [129:58] yeah i think we can do that [130:00] um [130:03] and [130:04] yeah [130:05] this i think will be quite quite [130:07] profound because if you say like what is [130:10] the economy it is uh [130:12] at the foundation it is labor so [130:15] what happens when there is uh [130:18] you know [130:19] no shortage of labor [130:22] um that's why i think long-term that [130:23] there will need to be universal basic [130:25] income [130:27] yeah [130:29] but but not right now because this robot [130:31] doesn't work so [130:33] we just didn't need a minute [130:36] so [130:36] um [130:38] yeah but i think essentially in the [130:39] future physical work will be a choice if [130:42] you want to do it you can but you won't [130:43] need to do it [130:45] and [130:46] yeah i think it obviously has profound [130:48] implications for the economy because [130:51] given that the economy at its [130:52] foundational level is labor i mean [130:54] capital is capital equipment it's just [130:56] distilled labor [130:58] then [131:00] is there any actual limit to the economy [131:02] maybe not [131:05] so [131:06] yeah [131:09] join our team and help build this [131:12] all right so i think we'll we'll have [131:13] everyone come back on the stage and you [131:15] guys can ask questions if you'd like [131:18] yeah [131:21] [Applause] [131:37] you [131:45] [Music] [131:57] all right there we go [132:00] [Music] [132:02] all right cool you guys brought me back [132:03] on [132:04] i just like stage [132:06] try to get [132:07] the camera angle from there [132:08] but not from the side [132:16] all right cool [132:17] so we'll probably turn the lights back [132:19] on [132:20] and uh [132:24] yeah great um [132:26] so we're having to answer any questions [132:27] you have about anything on the software [132:30] hardware side where things are going [132:32] and yeah fire away [132:35] um we have because the lights are like [132:37] interrogation lights so we actually [132:38] cannot see ah there we go great [132:43] all right cool [132:57] i can just okay there we go first off i [132:59] mean thanks to all the presenters that [133:00] was just super cool to see everything [133:02] i'm just curious at a high level and [133:03] this is kind of a question for really [133:05] anyone who wants to take it um to what [133:08] extent are you interested in publishing [133:10] or open sourcing anything that you do [133:13] for the future [133:27] um well i mean it is fundamentally [133:30] extremely expensive to create uh the [133:33] system so [133:35] uh [133:36] somehow that has to be paid for i'm not [133:38] sure how to pay for it if it's fully [133:39] open sourced [133:40] um [133:43] yeah unless people want to work for free [133:50] but but i should say that uh this is [133:53] if other car companies want to license [133:56] it and use it in their cars that would [133:58] be cool this is not intended to be just [133:59] limited to tesla cars [134:05] is for the dojo supercomputer so did you [134:08] solve the compiler problem of scaling to [134:10] these many nodes or is or [134:14] if it is solved is it only applicable to [134:16] dojo [134:17] because [134:19] i'm doing research in [134:21] deep learning accelerators and getting [134:23] the correct scalability [134:25] or the distribution even in one ship is [134:29] extremely difficult from the [134:31] research projects perspective so i was [134:33] just curious [134:35] excuse me mike for bill [134:38] you [134:40] have we solved the problem not yet are [134:43] we confident we will solve the problem [134:45] yes [134:46] we have demonstrated networks on [134:47] prototype hardware now we have models [134:49] performance models showing the scaling [134:52] the difficulty is as you said how do we [134:54] keep the localities [134:56] if we can do enough model parallel [134:57] enough data parallel to keep most of the [135:00] things local we just keep scaling we [135:02] have to fit the parameters in our [135:03] working set in our sram that we have [135:06] and we flow through the pipe [135:11] there's plenty of opportunities [135:13] sorry as we get further scale for [135:15] further processor nodes have more local [135:16] memory [135:17] memory trade also bandwidth we can do [135:20] more things but as we see it now [135:22] the applications that tesla has we see a [135:24] clear path [135:28] and our our modularity story means we [135:30] can have different ratios different [135:32] aspects created out of it i mean this is [135:35] something that we chose for our [135:37] applications [135:38] internally [135:43] sure [135:45] the locality portion of it given that [135:47] training is such a soft scaling [135:49] application [135:50] uh even though you have all this compute [135:52] and have a high bandwidth [135:55] high bandwidth interconnect [135:57] it it could not [135:59] give you that performance because you [136:01] are doing computations on limited memory [136:03] at different locations so i was that's [136:06] very curious to me when you said it [136:07] solved because i i just jumped onto the [136:09] opportunity and would love to know more [136:12] given that how much you can open source [136:15] yeah [136:18] yeah i guess the proof's in the pudding [136:20] um [136:21] so we should have dojo operational next [136:23] year um and [136:25] um i think we'll we'll obviously use it [136:27] for [136:28] uh training video training it's i mean [136:30] fundamentally this is about like um [136:33] the the [136:34] primary application initially is we've [136:36] got vast amounts of video and how we [136:38] train vast amounts of video uh [136:40] as efficiently as possible [136:42] and [136:45] uh also shorten the amount of time like [136:47] if you're trying to train train to a [136:48] task um [136:50] like just in general innovation is um [136:53] how many iterations and what is the [136:55] average progress between each iteration [136:57] and so if [136:59] if you can reduce the time between [137:00] iterations uh the rate of improvement is [137:03] is much better [137:04] so [137:05] um you know if it takes like sometimes a [137:07] couple days for a model to train versus [137:10] a couple hours that's a big deal [137:13] but the the asset test here and um you [137:16] know what i've [137:18] told the dojo team is like it's [137:20] it's successful if the [137:23] uh [137:24] software team wants to turn off the gpu [137:26] cluster [137:27] but if they want to keep the gpp cluster [137:29] on [137:30] it's not successful [137:32] so yeah [137:38] hi [137:39] right over here [137:41] i love the presentation thank you for [137:43] getting us out here loved everything [137:44] especially the simulation part of the [137:46] presentation i was wondering uh it [137:48] looked very very [137:50] very uh realistic are there any plans to [137:52] maybe expand simulation to other parts [137:53] of the company in any way [137:58] hi i'm ian glow uh i manage the [138:00] autopilot simulation team [138:02] so as we go down the path to [138:05] full self driving we're gonna have to [138:06] simulate more and more of the vehicle [138:09] currently we're simulating vehicle [138:10] dynamics but we're going to need bms [138:12] we're going to need the mcu we're going [138:13] to need every single part of the vehicle [138:15] integrated and that actually makes the [138:16] autopilot simulator really useful for [138:18] places outside of autopilot so i want to [138:20] expand or we want to expand eventually [138:23] to being a universal simulation platform [138:25] but i think before that we're going to [138:27] be spinning up a lot of optimist support [138:29] and then a little bit further down the [138:30] line that we have some rough ideas on [138:33] potentially how to get uh [138:34] the simulation infrastructure and some [138:36] of the cool things we've built into the [138:37] hands of people outside of the company [138:40] optimus is the code name for the tesla [138:42] bot [138:44] oops [138:49] optimus uprine [138:50] [Laughter] [138:56] hi [138:57] this is ali jahanian [138:59] thank you for the great presentation and [139:01] putting all of these cool things [139:03] together [139:04] yeah for a while i have been thinking [139:06] that uh the car is already a robot so [139:09] why not a humanity robot [139:12] and i'm so happy that today [139:14] you mentioned that you are going to [139:16] build such thing [139:18] especially i think that this can uh give [139:21] opportunity for rays of uh putting multi [139:24] modality together for instance we know [139:27] that in [139:29] the example that you are showed that [139:31] there was a dog and [139:33] with some passengers or running together [139:38] the [139:39] language [139:40] and symbolic processing can really help [139:42] for [139:43] visualizing that [139:45] so i was wondering [139:48] if i could hear a little more about [139:51] this type of putting modalities together [139:54] including language and vision because i [139:57] have been working with for instance mini [139:59] gpt's and andre put out there [140:02] and [140:04] yeah i had i didn't hear much about [140:07] other modalities that's going into the [140:09] car or at least in the simulation [140:12] is is there any comment [140:14] that [140:15] you could tell us [140:16] well driving is fundamentally uh [140:20] basically [140:22] almost entirely vision neural nets uh [140:24] like we're basically it's running on a [140:26] biological vision [140:27] neural net and what we're doing here is [140:30] a silicon camera neural net um [140:33] so [140:34] and there are there's there is some [140:35] amount of uh audio uh you know you want [140:37] to hear uh if um there's like emergency [140:40] vehicles or [140:42] uh [140:42] you know uh [140:45] i guess converse with the [140:47] people in the car [140:48] um [140:50] you know if somebody's yelling something [140:52] at the at the car that car needs to [140:54] understand what that is [140:56] um [140:57] so [140:59] you know all things that are necessary [141:00] for it to [141:01] be fully autonomous [141:04] yeah [141:05] thank you [141:11] hi uh thank you for all the great work [141:13] that you've shown uh my question is for [141:15] the team uh because the data that was [141:18] shown was seems to be predominantly from [141:20] the united states that the the fsd [141:22] computer is being trained on but as it [141:24] is being as it gets rolled out to [141:26] different countries which have their own [141:28] road systems and [141:29] challenges that come with it how do you [141:31] think that it's going to scale like like [141:34] i'm assuming like the ground up is not a [141:36] very viable solution so how does it [141:38] transfer to different countries [141:41] uh well there is we actually do train on [141:45] using data from [141:47] probably like 50 different countries [141:49] um but we have to pick [141:53] you know in as we're trying to advance [141:54] full self driving we need to pick one [141:56] country and since we're located here we [141:58] pick the us um [142:00] and then we get a lot of questions like [142:02] why not even canada like well because [142:03] the roads are a little different in [142:04] canada different enough um [142:07] and so when trying to solve a hard [142:08] problem uh you want to [142:11] uh say like okay what's the [142:14] let's not add additional complexity [142:16] right now uh let's just solve it for the [142:18] us and then we'll extrapolate to the [142:20] rest of the world but we do use video [142:21] from all around the world yeah i think a [142:23] lot of a lot of what we are building is [142:25] very country agnostic fundamentally all [142:27] the computer vision components and so on [142:29] don't care too much about country [142:30] specific uh [142:32] sort of features every you know [142:34] different countries have roads and they [142:35] have curbs and they have cars and [142:37] everything we're building is fairly [142:38] general for that [142:40] yeah and there's the the prime directive [142:42] is don't crash right and that's true for [142:44] every country yes this is the prime [142:46] directive [142:48] um [142:49] and um [142:50] even right now the car is pretty good at [142:52] not crashing um [142:54] and so [142:55] just basically um [142:57] whatever it is don't hit it even if it's [142:59] a ufo that crash landed uh on the [143:02] highway and still don't hit it [143:06] you should not need to recognize it in [143:07] order to not hit it [143:09] so that's very important [143:20] and i want to ask that [143:23] when you do the photometric process [143:25] multiview geometry how much of an error [143:27] do you see is it like one millimeter one [143:30] centimeter so i'm just if it's not [143:32] confidential [143:35] sorry question what is the what's it [143:37] what's the difference between the [143:38] synthetic [143:40] sure [143:42] what is the difference between the [143:43] synthetically created geometry to the [143:45] actual geometry [143:47] yeah it's usually within a couple [143:49] centimeters three or four centimeters [143:52] that's the standard deviation [143:55] merge with different kind of modalities [143:58] to bring down that error [144:02] we primarily try to find scalable ways [144:04] to label um in some occasions we use [144:06] other sensors to [144:08] help benchmark but we primarily use [144:10] cameras for this system [144:12] okay thanks yeah i i mean i think we [144:15] want to aim for the car to be positioned [144:17] uh accurately to the the sort of [144:20] centimeter level um [144:21] you know [144:22] something on that order obviously it [144:24] will depend on distance like close by [144:25] things can be much more accurate than [144:27] farther away things because and they [144:28] would matter less because the car [144:30] doesn't have to make decisions much [144:31] farther away and as it comes close it [144:33] will become more and more accurate [144:36] exactly [144:39] a lot of questions [144:40] thanks everybody my question has to do [144:43] with sort of ai and manufacturing it's [144:44] been a while since we've heard about the [144:46] alien dreadnought concept [144:48] is the humanoid that's behind you guys [144:50] is that kind of brought out of the [144:52] production health timeline and saying [144:53] that humans are underrated in that [144:55] process [144:57] um well sometimes like some [145:00] you know something that i say is uh [145:02] taken to too much of an extreme [145:04] there [145:08] there are parts of the tesla system that [145:09] are almost completely automated and then [145:11] there are some parts that are almost [145:12] completely manual [145:15] and [145:16] if you were to walk through the whole [145:17] production system you would see a very [145:19] wide range from yeah like i said fully [145:22] automatic to [145:23] almost completely manual [145:25] but the vast majority it's most of it is [145:28] is already uh automated [145:31] um [145:33] so and then [145:34] with the [145:35] some of the design architecture changes [145:37] like going to [145:39] large [145:40] aluminum [145:41] high pressure die cast [145:43] components we can take the entire rear [145:45] third of the car and cast it as a single [145:47] piece and now we're going to do that [145:50] the front third of the car is a single [145:52] piece [145:53] so the the body line [145:54] um [145:55] drops by like 60 to 70 percent in size [145:59] um [146:01] but yeah the the robot is not is not [146:04] prompted by [146:05] specifically by manufacturing [146:08] needs it's it's just that [146:11] we're just obviously making the pieces [146:13] that [146:14] are needed for a useful humanoid robot [146:19] so i guess we probably should make it [146:21] and if we don't someone else would well [146:23] and so i guess we should make it [146:26] and make sure it's safe [146:32] i should say like also manufacturing [146:34] volume manufacturing is extremely [146:36] difficult um and underrated and we've [146:38] gotten pretty good at that [146:40] it's also important for that humanoid [146:42] robot like how do you make the human [146:44] robot not be super expensive [146:46] and [146:49] hi [146:50] uh thank you for the present [146:53] presentation [146:54] and my question will be about skilling [146:57] of dojo [146:58] and uh in particular how do you scale [147:01] the compute nodes in terms of [147:04] thermal [147:06] thermals and power delivery because [147:08] there is only so much heat that you can [147:11] dispense [147:12] and only so much power that you can [147:14] bring to [147:16] like cluster rack [147:18] and how do you want to scale it and how [147:21] do you point to scale it in multiple [147:24] data centers [147:27] sure [147:33] hi [147:34] i'm bill i [147:35] one of the dojo engineers the um [147:38] so from a thermal standpoint and power [147:41] standpoint [147:42] we've designed it very modular so what [147:44] you saw on the compute tile that will [147:46] that will cool the entire tile so we we [147:50] once we hook it up to it is liquid [147:51] cooled on both the top and the bottom [147:53] side um it doesn't need anything else [147:57] and so when we talk about clicking these [148:00] together [148:01] once we click it to power and we once we [148:03] click it to um [148:05] cooling it will be fully powered and [148:07] fully cooled [148:08] and all of that is [148:10] less than a cubic foot [148:12] yeah [148:13] so tesla has a lot of expertise in power [148:15] electronics and in uh in cooling so we [148:18] took uh the power electronics expertise [148:21] from the vehicle powertrain [148:23] and the sort of the advanced cooling [148:25] that we developed [148:26] for the power electronics and for the [148:28] vehicle and applied that to the super [148:30] computer [148:31] because as you point out uh getting heat [148:34] out is [148:35] extremely important just really heat [148:36] limited [148:38] so [148:39] um [148:40] yeah so it's funny that like [148:42] at the compute level it's operating at [148:44] less than a volt [148:46] which is a [148:48] very low voltage there's a lot of amps [148:50] so therefore a lot of heat [148:52] i squared r is what really bites you on [148:54] the ass [148:58] um hi uh my question's also similarly a [149:01] question of scaling [149:02] um so it seems like a natural [149:04] consequence of using you know [149:06] significantly faster training hardware [149:07] is that you'd be either training models [149:09] over a lot more data [149:10] or you'd be training a lot more complex [149:12] models which would be [149:14] potentially significantly more expensive [149:16] to run at inference time on the cars [149:18] uh i guess i was wondering like if there [149:20] was a plan to like also [149:23] um [149:24] apply dojo as something that you'd be uh [149:27] using like on the self-driving cars and [149:30] if so like [149:31] you know do you foresee additional [149:32] challenges there [149:36] i can [149:36] so as you could see like android's [149:39] models are not just for cars like there [149:41] are auto labeling models there are other [149:43] models that are [149:45] like beyond car application but they [149:48] feed into the car stack so so dojo will [149:51] be used for all of those too [149:54] not just the car [149:56] inference part of the training [149:59] yeah i mean the dojo's [150:02] first application will be consuming [150:04] video data for training for that would [150:06] then be run in the inverse inference [150:08] engine on the car [150:09] but uh [150:11] and that i think is an important uh test [150:13] to see if it actually is good or but is [150:15] it actually better than gpu cluster or [150:17] not um so [150:19] but then beyond that it's basically a [150:21] general a generalized neural net [150:23] training computer [150:25] but it's very much optimized to be a [150:27] neural net so [150:29] you know cpus and gpus uh [150:33] they're they're they're not made to be [150:35] um [150:36] they're not they're not designed [150:37] specifically for training neural nets um [150:40] we've been able to make gpus especially [150:42] very efficient for [150:44] portraying neural nets but that's not [150:46] that was never their design intent so [150:48] it's it's bc gpus are still essentially [150:50] running it uh neural net training in [150:53] emulation mode so [150:55] um with with dojo we're saying like okay [150:57] let's just let's just asic the whole [150:58] thing [150:59] let's just [151:00] have this thing that's it's built for [151:02] one purpose and that is neural net [151:04] training [151:05] and just generally any system that is [151:07] designed for a specific [151:08] purpose will be better than one that is [151:10] designed for a general purpose [151:14] hey i had a question here [151:16] hi um [151:18] so you described two separate systems [151:20] one was for vision therefore planner and [151:22] control um does dojo love you train [151:25] networks that cross that boundary and [151:27] second thing is if you were able to [151:29] train such networks would you have the [151:30] onboard compute capability in the fst [151:32] system to be able to run that in in [151:34] under your tight latency constraints [151:36] thanks [151:39] yeah i think we should be able to train [151:40] uh planner networks on dojo or any gpus [151:44] it's really invariant to the platform um [151:47] and i think uh if anything once we make [151:49] this entire thing end to end it'll be [151:50] more efficient than decoding a lot of [151:52] these intermediate states so you should [151:53] be able to run faster if you make the [151:54] entire thing uh into in your neural [151:56] networks we can avoid a lot of decoding [151:58] of the intermediate states and only [151:59] decode essential things required for [152:01] driving the car [152:03] yep certainly and to endness and as the [152:05] guiding principle behind a lot of the [152:06] network developments and over time in [152:08] the stack neural networks have taken on [152:10] more and more functionality and so [152:13] we want everything to be trained [152:14] end-to-end because we see that that [152:15] works best but we are building it [152:16] incrementally so right now the interface [152:18] there is vector space and we are [152:20] consuming it in the planner but nothing [152:22] really fundamentally prevents you from [152:23] actually taking features and eventually [152:25] fine-tuning end-to-end uh so i think [152:27] that's definitely where this is headed [152:29] yeah and the discovery really is like [152:30] what are the right architectures that we [152:32] need to place in network blocks to make [152:34] it amenable to the task so like on a [152:36] describe we can place spatial rnns to [152:38] help with the uh perception problem uh [152:41] and now it's just neutral network so [152:42] similarly for planning we need to bake [152:43] in search and optimization into the [152:45] planning into the network architecture [152:46] and once we do that you should be able [152:48] to do planning very quickly uh similar [152:50] to c plus plus algorithms [152:58] okay [153:00] okay [153:01] i think i had a question very similar to [153:03] what he was asking about [153:05] it seems like a lot of neural nets [153:07] around computer vision [153:08] and kind of traditional planning you had [153:10] model predictive control in solving [153:12] convex optimization problems very [153:13] quickly [153:14] and [153:16] i'd wonder if there's a compute [153:17] architecture that's more suited for [153:19] convex optimization or the model [153:21] predictive control uh solutions very [153:23] quickly [153:24] yeah 100 if you want to bake in like i [153:26] said earlier if you want to bake in [153:27] these architectures that do say model [153:28] protein control but just like replace [153:30] some of the blocks with neural networks [153:32] or if we know the physics of it we can [153:34] also use physics-based models part of [153:36] the neural network's forward pass itself [153:37] so we are going to go towards a hybrid [153:40] system where [153:41] we will have neural network blocks [153:45] placed together with a physics-based [153:47] blocks and more networks later so it'll [153:49] be a hybrid stack and what we know to do [153:51] well we place with explicitly and what [153:53] the networks are created we'll use the [153:54] networks to optimize this uh so better [153:56] end-to-end stack with this architecture [153:58] baked in [154:00] i i mean i do think that [154:02] so as long as you've got like um [154:05] surround video [154:07] uh [154:08] neural nets for understanding what's [154:09] going on and can uh [154:12] convert [154:14] those surround video into vector space [154:16] then you basically have a video game [154:18] um and if [154:20] you know if you [154:21] it's like if you're in grand theft auto [154:23] whatever you can you can make the cars [154:24] drive around and pedestrians walk around [154:26] without crashing so [154:27] um you can do you don't have to have a [154:31] neural net for control and planning um [154:34] but it's probably [154:36] ultimately better um [154:38] so [154:39] but i think you can probably get to [154:41] in fact i'm sure you can get to much [154:42] safer than human with control and [154:45] planning [154:46] primarily in c plus plus with [154:49] perception vision in [154:51] neural nets [154:56] hi [154:57] my question is we've seen other [154:59] companies for example use reinforcement [155:02] learning and machine learning to [155:03] optimize [155:04] power consumption and data centers and [155:06] all kinds of other internal processes my [155:07] question is are is tesla using machine [155:10] learning within its manufacturing design [155:13] or other engineering processes [155:19] i [155:22] i discourage use of machine learning [155:26] because it's really difficult [155:27] unless you basically unless you [155:30] have to use machine learning don't do it [155:34] it's usually a red flag when somebody [155:35] says we want to use machine learning to [155:37] solve this test i'm like that sounds [155:39] like um [155:41] so [155:42] uh [155:43] 99.9 percent of time you do not need it [155:46] um [155:47] so [155:49] yeah [155:50] but so it's kind of like a [155:53] you you reach for machine learning when [155:55] you when you [155:56] need to not but it's i've not found it [155:58] to be a convenient easy thing to do [156:01] um it's a super hard thing to do [156:03] that may change if you've got a humanoid [156:04] robot that can [156:06] you know understand [156:08] normal instructions [156:10] um [156:11] but [156:12] yeah [156:13] generally minimize use of machine [156:15] learning in the [156:16] factory [156:20] hi um based on your videos from the [156:23] simulator uh it looked like a [156:25] combination of graphical and neural [156:27] approaches i'm curious what the set of [156:29] underlying techniques uh that are used [156:31] for your simulator and specifically for [156:34] neural rendering if you can share [156:38] yeah so we're doing uh [156:41] at the bottom of the stack it's just [156:42] traditional game techniques [156:44] uh just rasterization real time [156:47] uh you know very similar to what you'd [156:49] seen like gta [156:51] on top of that we're doing real-time ray [156:53] tracing and then those results were [156:56] really hot off the press um i mean we [156:58] had that little asterisk at the bottom [156:59] that that was from last night [157:01] we're going into the neural rendering [157:02] space we're trying out a bunch of [157:04] different things [157:05] we want to get to the point where the [157:06] neural rendering is the [157:08] the cherry on the top that pushes it to [157:10] the point where the models will never be [157:11] able to overfit on our simulator [157:15] currently we're doing things similar to [157:17] photorealism enhancement [157:19] there's a paper a recent paper photo [157:20] enhancing photo realism enhancement [157:23] but we can do a lot more than what they [157:24] could do in that paper because we have [157:26] way more labeled data way more compute [157:29] and also much we have a lot more control [157:31] over environments [157:33] and we also have a lot of people who can [157:34] help us make this run at real time [157:36] um but we're going to try whatever we [157:38] can do to get to the point where we can [157:40] train everything just with the simulator [157:43] uh if we had to but we will never have [157:45] to because we have so much real world [157:47] data that no one else has [157:49] it's just to fill in the little gaps in [157:51] the real world [157:53] yeah i mean [157:54] the simulator [157:55] is very helpful when there's like these [157:58] rare cases like like um [158:00] you know like [158:01] collision avoidance right before an [158:03] accident [158:04] um and then [158:05] ironically that the better our cars [158:07] become at avoiding accidents the fewer [158:09] accidents there are so then our training [158:10] set is small so then we have to make [158:12] them crash in the simulation [158:14] so it's like okay minimize potential [158:17] injury to [158:19] uh pedestrians and people in the car [158:22] you have five meters you're traveling at [158:26] you know 20 meters per second um [158:29] take what actions would minimize [158:32] probability of injury [158:34] we can run that in some [158:42] cars driving down the wrong side of the [158:43] highway that kind of thing [158:45] happens occasionally but not that often [158:48] um for your humanoid contacts i'm [158:50] wondering if you've decided on what use [158:52] cases you're gonna start with and what [158:54] the grand challenges [158:56] um [158:57] are in that context to make this viable [159:01] well i think for the humanoid for the [159:03] tesla bot um [159:06] optimus [159:07] it's basically going to start with [159:10] just dealing with [159:11] work that is [159:13] boring repetitive and dangerous [159:17] basically what is the work that people [159:18] would least like to do [159:27] um [159:28] hi um so quick question about [159:31] your simulations um obviously they're [159:34] not perfect right now so are you using [159:36] any sort of domain adaptation techniques [159:38] to basically bridge the gap between your [159:40] simulated data and your actual [159:42] real-world data because i imagine it's [159:44] kind of dangerous to [159:46] just deploy models which are solely [159:48] trained on simulated data so maybe some [159:50] sort of explicit domain adaptation or [159:52] something is that going on anywhere in [159:54] your pipeline [159:56] so currently uh i mean we're producing [159:59] the videos straight out of the simulator [160:01] uh the the full clips of kinematics and [160:03] everything and then we're just [160:04] immediately training on them but it's [160:06] not the entire data set it's just a [160:08] small targeted segment and we only are [160:10] evaluating based on real world video um [160:13] we're paying a lot of attention to make [160:14] sure we don't ever fit uh and if we have [160:16] to start doing fancier things we will [160:18] but currently it's we're not having an [160:21] issue with it overfitting on the [160:22] simulator we will as we scale up the [160:23] data um and that's what we're hoping to [160:25] use neural rendering to bridge that gap [160:28] to push that even further out um we've [160:29] already done things where we're using [160:31] like the same networks in the car but [160:33] retrain it to detect sim versus rail to [160:35] drive our art decisions [160:36] um and that's actually helped um prevent [160:39] some of these things as well [160:42] yeah just to emphasize that [160:43] overwhelmingly the data set is the real [160:46] video from the cars on the actual roads [160:48] uh nothing's weirder or uh or has more [160:50] corner cases than reality [160:52] um it's gets really strange out there [160:56] but but then if if we find say a few [160:59] examples of something very odd [161:02] and there's some very some for some very [161:04] odd pictures we've seen [161:06] then in order to train it effectively we [161:09] want to [161:10] create simulations [161:12] uh say a thousand simulations that are [161:14] that are variants of that [161:16] quirky thing that we saw [161:18] the foot to fill in the some important [161:20] gaps and and make the system better and [161:22] really all of this is about over time [161:24] just [161:26] reducing the probability of of a crash [161:28] or an injury and [161:31] it's called the march of nines like how [161:33] do you get to 99.999999 [161:36] uh safe you know and [161:40] it yeah [161:41] each nine is an order of magnitude [161:43] difficulty increase [161:47] uh thanks so much for the presentation i [161:50] was curious about [161:51] the tesla bot um [161:53] specifically i'm wondering [161:55] if there are any specific applications [161:57] that you think the humanoid form factor [161:59] lends itself to [162:00] and then secondary [162:02] um [162:04] because of its human form factor is [162:06] emotion or companionship at all thought [162:08] about [162:09] on the product roadmap at all [162:17] um [162:18] well we certainly hope this does not [162:20] feature in a dystopian [162:22] sci-fi movie [162:26] but uh you know like [162:29] really at this point [162:31] we're saying like maybe this robot can [162:33] just we're trying we're trying to be as [162:34] literal as possible can it do um boring [162:37] dangerous repetitive [162:39] jobs that people don't want to do [162:41] and [162:43] uh [162:44] you know once you can have it do that [162:46] then maybe you can do other things too [162:48] but [162:49] that's the that's the thing that we [162:51] really great to have [162:53] so [162:55] it could be your buddy too i mean buy [162:56] one and have it be or your friend and [162:58] whatever [163:02] i'm sure that people think of some very [163:03] creative uses [163:08] so [163:09] uh so [163:12] um so firstly thanks for the the really [163:15] incredible presentation um my question [163:17] is on the ai side um so one thing we've [163:20] been seeing is that with some of these [163:22] language modeling ais we've seen that [163:23] scaling has just had incredible impacts [163:27] in their capabilities and what they're [163:28] able to do [163:29] so i was wondering whether you're seeing [163:30] similar kinds of effects of scaling in [163:33] in your [163:34] neural networks in your applications [163:37] absolutely a bigger network typically we [163:39] see it performs better provided you have [163:41] the data to also train it with and this [163:43] is also what we see for ourselves [163:45] definitely in the car we have some [163:46] latency consideration to be mindful of [163:49] and so there we have to get creative to [163:50] actually deploy much much larger [163:51] networks but as we mentioned we don't [163:54] only train neural networks for what goes [163:56] in the car we have these um [163:58] auto labeling pipelines that can utilize [163:59] models of arbitrary size so in fact [164:01] we've traded a number of models that are [164:03] not deployable that are significantly [164:04] larger and work much better because we [164:06] want 100 like we want much higher [164:07] accuracy for the auto labeling and so [164:10] we've done a lot of that and there we [164:11] definitely see this trend [164:14] yeah the order labeling is uh [164:16] an extremely important important part of [164:18] this whole whole situation [164:21] without the order labeling i think we [164:23] would not be able to solve the [164:24] self-driving problem it's kind of a [164:26] funny form of distillation where you're [164:28] using these very massive models plus the [164:30] structure of the problem to do this [164:31] reconstruction and then you distill that [164:33] into neural networks that you deploy to [164:35] the car but we basically have a lot of [164:37] neural networks and a lot of tasks that [164:38] are never intended to go into the car [164:41] yeah and also as time goes on that you [164:43] get new frames of information so you [164:45] really want to make sure your computer [164:46] is distributed across all the [164:48] information as opposed to just taking a [164:49] single frame and hogging on it for say [164:51] 200 milliseconds you actually have newer [164:53] frames coming in so you want to like use [164:55] all of the information and not just use [164:56] that one frame [164:59] i think [165:00] one of the things we're seeing is that [165:01] the car's predictive ability is um is [165:04] quite is eerily good [165:06] um [165:07] it's really getting better than human in [165:09] terms of predicting like you say like [165:11] what [165:12] predict what this road will look like [165:15] out when it's out of sight like it's [165:17] around the bend and it predicts the road [165:19] with very high accuracy [165:22] and [165:23] you know [165:24] predict pedestrians or cyclists wherever [165:26] behind you know where to just see a [165:28] little corner of the bicycle and [165:30] a little bit through through the windows [165:31] of the bus and its ability to predict [165:34] things is going to be much better than [165:36] humans like really way way beyond right [165:39] yeah we see this often where we have [165:41] something that is not visible but the [165:42] neural network is making up stuff that [165:44] actually is very sensible sometimes it's [165:46] eerily good and you have to like you're [165:47] wondering this is in the training set [165:49] and actually [165:50] actually in the limit you can imagine [165:52] the neural net has enough parameters to [165:54] potentially remember earth so [165:56] in the limit it could actually give you [165:57] the correct answer and it's kind of like [165:58] an hd map back baked into the weights of [166:00] the neural net [166:08] i have a question about the design of [166:11] the tesla bot specifically in order uh [166:14] how important is it to maintain that [166:16] humanoid form [166:18] to build hands with five fingers [166:20] uh that also respects the weight limits [166:23] could be quite challenging you might [166:24] have to use cable driven and then that [166:26] also causes all kinds of issues [166:30] um i mean this is just going to be [166:32] bought version one and we'll see [166:35] so [166:36] the [166:37] it's [166:38] it needs to be able to do things that [166:40] that people do um and uh [166:43] you know be a generalized you know [166:44] humanoid robot [166:46] um [166:49] i mean you could make you potentially [166:50] have it give it like you know [166:53] two fingers and a thumb or something [166:54] like that [166:55] um [166:56] you know for now we'll we'll give it [166:58] five fingers and and see [167:00] see if that works out okay [167:01] it probably will it doesn't need to be [167:03] like uh you know have like [167:05] incredible grip strength [167:07] but it needs to be able to work with [167:09] tools [167:11] so and you know carry a bag that kind of [167:12] thing [167:15] all right thanks a lot for the [167:16] presentation [167:18] so an old professor of mine told me that [167:21] um the thing he disliked a lot about his [167:22] tesla was that the autopilot ux didn't [167:25] really inspire much confidence in the [167:26] system especially when like objects are [167:28] spinning classifications are flickering [167:31] i was wondering like even if you have a [167:34] good self-driving system how are you [167:35] working on convincing tesla owners other [167:38] road users or other road users and just [167:41] the general public that your system is [167:43] safe and reliable [167:45] well i think that's that's [167:47] the cars [167:48] a while back cars used to spend they [167:50] don't they don't spin anymore not in the [167:52] if you've seen the fsd [167:54] beta videos they [167:56] they're they're pretty solid um [167:58] and they will be getting more solid yeah [168:00] as him add more and more data and train [168:02] these multi-camera networks like these [168:03] are pretty decent actually just like few [168:04] months old and it's really improving [168:06] it's not a done product uh and that [168:08] we've never minds we can clearly see how [168:10] this is just going to be like perfect [168:11] but perfect for space because why not uh [168:14] all the information is that in the [168:15] videos it should produce a given lots of [168:16] data and good architectures [168:18] um [168:19] and this is just an intermediate point [168:21] in the timeline [168:23] i mean it's clearly headed to way better [168:26] than human without question [168:33] my turn oh [168:34] hi here um i was wondering if you could [168:37] talk a little bit about the short to [168:38] medium-term economics of the bot i guess [168:42] i understand the long-term vision of [168:43] replacing physical labor [168:46] but i also think repetitive dangerous [168:49] and boring tasks tend to not be [168:51] so highly compensated and so i just [168:53] don't see how to reproduce uh [168:56] you know start with a supercar and then [168:58] break into like the lower end of the [169:00] market how do you do that for a robot [169:02] humanoid [169:05] well i guess you'll just have to see [169:16] hello hi [169:17] um i was curious to know how the car ai [169:20] prioritizes [169:21] occupant safety versus pedestrian safety [169:24] and what thought process goes into like [169:27] deciding how to make this into the ai [169:32] well i mean we [169:34] the the thing to appreciate [169:36] is that from the computer standpoint [169:38] everything is moving slowly [169:41] so uh think you know to a human uh [169:44] things are moving fast to the computer [169:46] they are not moving fast so i think this [169:48] is in reality somewhat of a false [169:50] dichotomy not that it will never happen [169:52] but it will be very rare [169:54] um [169:56] you know if you think it was like [169:58] you know going the other direction [170:00] like rendering [170:02] you know with full ray tracing uh [170:04] neural net enhanced graphics on [170:07] something like cyberpunk or in any you [170:09] know advanced video game [170:11] you know doing 60 frames a second [170:14] uh [170:15] perfectly rendered [170:17] like how long would it take a person to [170:18] even render one frame [170:20] and without any mistakes [170:24] can be done i mean it would take like a [170:26] month [170:27] just to [170:28] just render one one frame out of 60 in a [170:31] second in a video game [170:35] it's uh [170:37] computers are fast and humans are slow [170:43] i mean for example [170:45] uh on on the rocket side the you you [170:48] cannot steer the rocket to orbit [170:51] um we actually hooked up a joystick to [170:53] see if anyone could steal the rocket [170:55] orbit but [170:56] you need uh [170:58] to react at [171:00] roughly [171:01] six seven hertz uh people can't do it [171:05] not even now that's pretty low you know [171:08] we're talking more like [171:09] even for like [171:10] 30 hertz type of thing [171:15] hi um with the over here uh with [171:19] hardware 3 there's been lots of [171:21] speculation that with larger nets it's [171:23] hitting the limits of what it can [171:24] provide [171:25] how much headroom has the extended [171:27] compute modes provided at what point [171:29] would hardware 4 be required if at all [171:34] well i'm confident that hardware 3 or [171:37] the full stop driving computer 1 will be [171:40] able to [171:42] achieve full self driving at a safety [171:44] level much greater than a human probably [171:46] i don't know [171:47] at least two or three hundred percent [171:48] better than a human um then obviously [171:51] there will be a future hardware for or [171:53] full self-driving computer too [171:56] which we'll probably introduce with the [171:57] cybertruck um so maybe in about a year [172:00] or so [172:02] uh [172:03] that is [172:04] probably well that'll be about four [172:05] times more capable roughly [172:08] um [172:09] but it's really just going to be [172:12] like can we take it from say for [172:14] argument's sake 300 safer than a person [172:17] to a thousand percent safer [172:19] um [172:20] you know just like there are people on [172:21] the road who with with varying driving [172:24] abilities but we still let people drive [172:26] it you don't have to be the world's best [172:27] driver to be on the road [172:30] so as we see [172:33] so [172:34] yeah [172:35] guess what's new [172:39] all [172:41] um right [172:42] are you worried at all since you don't [172:44] have any depth sensors on the car that [172:46] people might try like adversarial [172:48] attacks like [172:49] printed out photos or something to try [172:51] to trick the rgb [172:54] neural network [172:56] yeah like what pull some like wiley cody [172:58] stuff [172:59] you know like paint the tunnel on the on [173:01] the wall [173:04] it's like oops [173:06] um [173:08] we haven't really seen much of that um [173:10] i mean [173:13] for sure like like right now if you [173:16] pr most likely if you had like a [173:18] a t-shirt with a [173:19] t-shirt with like a stop sign on it [173:21] which i actually have a t-shirt with a [173:22] stop sign on it and and then you like [173:25] flash the car [173:28] it will it will stop [173:31] i i proved that [173:34] um but [173:35] we can obviously as we see these uh [173:37] adversarial attacks then we can we train [173:40] uh the cars to uh [173:42] you know notice that well it's actually [173:44] a person wearing a t-shirt the stop sign [173:46] on it so it's probably not a real stop [173:48] sign [173:54] hi uh my question is about um the [173:57] prediction and the planning i'm curious [173:59] how do you incorporate uncertainty into [174:02] your uh you know planning algorithms [174:05] do you just basically assume you know [174:08] you mentioned that you run the um [174:10] the autopilot for all the other cars on [174:12] the road do you assume that they're all [174:13] going to follow those rules or are you [174:15] accounting for the possibility that well [174:17] they might be bad drivers for example [174:20] yeah we do account for multi-modal [174:22] futures it's not that we just choose one [174:24] we account for this person can actually [174:26] do many things and [174:28] we use that actual [174:30] physics and kinematics to make sure that [174:31] they're not doing a thing that would [174:33] interfere with us before we act um so if [174:36] there's any uncertainty we are [174:37] conservative and then would yield to [174:39] them of course there's a limit to this [174:40] because if you're too conservative then [174:42] it's probably not practical so at some [174:44] point we have to assert [174:45] and we even then we make sure that the [174:47] other person can [174:48] yield to us and [174:50] act sensibly [174:54] i should say like [174:56] um [174:56] like before we introduce something into [174:59] the fleet we will run it in shadow mode [175:02] and so and we'll see what what would [175:05] this neural net for example have done in [175:08] this particular situation [175:10] um [175:11] because and then effectively the drivers [175:13] uh are training it training the net so [175:16] if the neural net would have [175:18] uh controlled and you know and say [175:20] veered right but the person actually [175:22] went left it's like oh there's a [175:24] difference why was there that difference [175:27] yeah and secondly [175:29] all the human drivers are essentially [175:31] training the neural net uh as to what is [175:34] the correct course of action [175:36] assuming it doesn't then end up in a [175:38] crash you know doesn't count in that [175:39] case [175:40] yeah and secondly we have various [175:41] estimates of uncertainty like flickr [175:44] and when we observe this we actually uh [175:46] say we are not able to see something we [175:48] actually slow down the car to be again [175:50] safe and get more information before [175:52] acting uh yeah we don't want to be [175:54] brazen and just go into something that [175:55] we don't know about we only go into [175:56] places where we know about [175:58] yeah but um yeah [176:01] it should be like [176:03] aspirationally that the car should be [176:04] the less it knows the sl you know the [176:06] slower it goes yeah [176:08] which is not true at some point but now [176:10] yeah yeah we've yeah [176:12] should we [176:13] speed proportionate to confidence [176:22] thanks for the presentation so i am [176:24] curious [176:26] appreciate the fact that the fsd is [176:29] improving but if you have the ability to [176:31] improve one component along the ai stack [176:34] to present it today whether it is [176:36] simulation data collections planning [176:38] control et cetera which one in your [176:41] opinion is going to have the biggest [176:42] impact for the performance of the full [176:44] self driving [176:46] system [176:51] it's really the area under the curve of [176:53] this like multiple points and if you [176:55] improve anything uh it should improve [176:56] the area [176:59] i mean in the short term it's arguably [177:01] we need all of the nets to be um [177:04] surround video uh and so we still have [177:06] some legacy this is a very short term [177:08] obviously we're fixing it fast but [177:10] there's [177:11] there's still some nets that are not [177:12] using surround video [177:14] um and i think ideally that all use [177:16] surround video [177:18] yeah very yeah i think a lot of puzzle [177:21] pieces are there for success we just [177:22] need more strong people to also just [177:24] help us [177:25] make it work yeah that is the actual box [177:27] so that is the actual bottleneck i would [177:28] say i'm really one of the reasons that [177:30] we are putting on this event exactly [177:31] what well said andre that um [177:34] there's just a tremendous amount of work [177:35] to do to make [177:37] make it work so that's why we need um [177:40] talented people to join in and [177:42] solve the problem [177:49] uh thank you for the great presentation [177:51] lots of my questions answered [177:53] but one thing is [177:55] uh when [177:57] imagine that now you have [177:59] a large amount of data even unnecessary [178:02] how do you consider that like there's a [178:04] forgetting problem in neural networks [178:07] like how are you considering those [178:09] aspects and also [178:12] another one are you considering online [178:14] learning or continuous learning [178:16] so [178:17] that maybe [178:18] each driver can have their version of [178:21] uh self-driving [178:23] software [178:25] i think i think i know the literature [178:27] that you're referring to that's not some [178:28] of the problems that we've seen and we [178:29] haven't done too much continuous [178:30] learning we train the system once we [178:32] find in a few times [178:34] that sort of goes into the car we need [178:36] something stable that we can evaluate [178:37] extensively and then we think that [178:39] that's good and that goes into cars so [178:41] we don't do too much learning on spot or [178:43] continuous learning and don't face the [178:44] forgetting problem uh but there will be [178:47] settings that you can say like if you do [178:49] do you want are you typically a [178:50] conservative driver or do you want to [178:52] drive fast or slow you know it's like [178:54] i'm late for my i'm late for the airport [178:56] uh could you go faster than you know [178:58] basically the kind of instructions you'd [178:59] give to your uber driver [179:01] it's like i'm late for the flight please [179:02] hurry um or take it easy or you know [179:05] whatever your style is [179:12] so let's take a few more questions here [179:14] so and then we'll call it a day all [179:17] right so as our models have become more [179:19] and more capable and i guess you're [179:21] deploying these models into the real [179:23] world um one thing i guess that's [179:25] possible is for ai to become more i [179:28] guess misaligned with what humans desire [179:31] so i guess is that something that you [179:32] guys are worried about as you guys [179:33] deploy more and more robots um or do you [179:36] guys like we'll solve that problem when [179:38] we get there [179:40] yeah i think that we should be worried [179:42] about ai um you know like [179:46] what we're trying to do here is i say [179:48] narrow ai uh [179:50] pretty narrow like just make the car [179:52] drive better than a human um [179:54] and then have the humanoid robot be able [179:57] to do basic stuff um [180:00] uh [180:01] you know so [180:03] um at the point at which you sort of [180:05] start get uh superhuman intelligence uh [180:08] yeah i don't know all bets are off [180:10] um [180:12] but [180:13] you know and that's that's that's you [180:14] know [180:16] that'll that'll probably happen but but [180:18] what we're trying to do here at tesla is [180:20] make useful [180:21] ai that [180:23] people love and and is [180:25] unequivocally good that's our [180:28] you know try to aim for that [180:31] okay maybe one more question [180:35] hi uh my question is about the camera [180:37] sensor in the beginning of the talk you [180:39] had mentioned about building a synthetic [180:41] animal [180:42] and if you think about it [180:44] a camera is a very poor approximation of [180:46] a human eye and a human eye does a lot [180:48] more than take a sequence of frames [180:51] have you looked into like there are like [180:53] these days are like cameras like event [180:54] cameras have you looked into them or are [180:57] you looking into a more flexible camera [180:59] design or building your own camera for [181:01] example [181:04] well with hardware four we will we will [181:06] have a next generation camera uh but i [181:09] have to say that the the current cameras [181:11] we have not reached the limit of the [181:13] current cameras uh so [181:15] um [181:17] and i'm confident we can achieve full [181:19] self-driving with much higher safety [181:20] than [181:21] humans with the current cameras and [181:23] current compute hardware [181:25] um [181:26] but [181:27] you know [181:29] are we good to be a thousand percent [181:30] better rather than 300 better [181:33] so [181:34] we'll see continued evolution on on our [181:37] levels [181:38] and pursue that goal and i think in the [181:40] future [181:41] people will look back and say [181:43] wow i can't believe we had to drive [181:44] these cars ourselves [181:45] you know it'll like self-driving cars [181:47] will just be just a normal like [181:49] self-driving elevators [181:51] you know uh elevators used to have [181:53] elevator operators [181:54] and uh there's someone there with like [181:56] you know big big relay switch [181:58] operating the elevator [182:00] and then every now and then they'd get [182:01] tired or you know some make a mistake [182:03] and share somebody in half so [182:06] um so now we uh you know we made [182:09] elevators automatic and you just go and [182:11] you press the button and you can be in a [182:12] 100 story skyscraper and [182:15] don't really worry about it just go and [182:16] press a button and [182:18] the elevator takes you where you want to [182:19] go [182:20] but it used to be that all elevators [182:22] were operated manually manually it'll be [182:24] the same thing like for cars [182:25] all cars will be automatic [182:28] and then um and electric obviously um [182:32] so there will still be some gasoline [182:33] cars and some manual cars just like [182:35] there are still some horses [182:39] so [182:40] um all right well thanks everyone for [182:41] coming and i hope you enjoyed a [182:43] presentation and thank you for the great [182:45] questions [182:46] [Applause] [182:56] [Music] [183:12] [Music] [183:19] you