Tesla Halts Radar Installs as Vision AI Outperforms Sensor Fusion

视觉AI表现优于传感器融合，特斯拉停止安装雷达硬件 特斯拉在其车队中取消了雷达硬件，因为发现多传感器融合会因幽灵刹车导致追踪性能下降，并证明了利用车队数据训练的纯摄像头神经网络能够可靠地解决3D深度和速度估算问题。 特斯拉人工智能高级总监Andrej Karpathy详细阐述了特斯拉决定弃用雷达、全面转向纯摄像头视觉栈背后的技术依据与工程架构。他解释了特斯拉如何利用离线自动标注、时序融合Transformer以及包含5760块NVIDIA A100 GPU的顶级超级计算机集群，在PB级车队数据上训练高容量深度神经网络。通过使用纯视觉模型取代脆弱的传感器融合启发式方法，该系统成功解决了自动驾驶领域长期存在的边缘案例，例如桥下幽灵刹车以及静止物体检测延迟等问题。 在CVPR 2021召开的三周前，特斯拉已将新生产的车辆完全转向纯视觉方案（“Tesla Vision”），并从传感器套件中彻底去除了毫米波雷达。 (6:00) 通过从人工标注转向利用重型网络和时序回溯的离线自动标注流水线，特斯拉构建了一个包含100万个高难度视频片段和60亿个深度与速度标注的1.5 PB数据集。 (17:10) 为了训练其合成视觉皮层网络，特斯拉部署了一个内部超级计算机集群，包含720个8卡NVIDIA A100 GPU节点（共计5760块GPU），并由可提供1.6 TB/s吞吐量的10 PB高速NVMe存储提供支持。 (20:31) 量产纯视觉版本的初步验证记录了170万英里的Autopilot自动驾驶里程且零事故发生，在自动紧急制动方面表现出比传统雷达融合栈更高的精确度和召回率。 (28:16)

At CVPR 2021, Andrej Karpathy argued that autonomous vehicles should rely exclusively on visible-light cameras rather than sensor fusion with radar or high-definition maps. Backed by Tesla's fleet-scale data engine, offline auto-labeling, and vertically integrated compute, Karpathy asserted that neural networks can reliably infer 3D depth and kinematics while eliminating the failure modes of multimodal fusion. Vision-Only Driving Versus HD Maps and Lidar (6:00) Karpathy contrasted Tesla’s deployment model with traditional autonomous driving systems that rely on roof-mounted lidar and high-definition (HD) maps. Pre-mapping strategies annotate lane topology, road geometry, and traffic signal relationships in advance, reducing onboard navigation to localizing within pre-recorded environments. Karpathy argued this framework cannot scale to millions of vehicles globally due to the prohibitive cost of maintaining dynamic, global map infrastructure. Instead, Tesla’s approach processes inputs from eight surround cameras at 36 frames per second, forcing neural networks to interpret intersections, lane connectivity, and road obstacles entirely online in real time. Eliminating Radar to Remove Sensor Fusion Debt (8:53) Addressing Tesla's decision to drop millimeter-wave radar, Karpathy noted that when visual perception becomes significantly more precise than radar, fusing the two degrades system performance. While radar reliably reports longitudinal velocity under ideal conditions, it frequently introduces unpredictable anomalies. Lacking vertical resolution, radar often misidentifies overhead bridges or manhole covers as stationary obstructions in the vehicle's path. When transient noise in legacy vision models associated with these phantom radar readings, the fusion software triggered unexpected phantom braking. Radar also struggled with track drops during aggressive deceleration, losing lead-vehicle locks up to six times in track tests. Attempting to reconcile these disparate sensor failure modes generated brittle heuristic fusion code packed with conditional statements. Pure-vision tracking maintained smooth continuous trajectories without dropped tracks, leading Tesla to decommission its separate radar and fusion pipelines to consolidate engineering efforts into a single vision team. Offline Auto-Labeling and the Fleet Data Engine (12:04) To train neural networks to estimate 3D bounding boxes, metric depth, and velocity without radar ground truth, Tesla treated range finding as a supervised learning challenge. Real-time vehicle computers must operate within strict 20-millisecond latency budgets, but offline auto-labeling leverages unconstrained compute. The offline pipeline applies heavy non-deployable neural networks, multimodal signals, and bidirectional temporal trajectory smoothing across past and future frames to reconstruct clean 3D training data through visual occlusions like dust clouds, snowstorms, and falling debris. Sourcing corner cases relied on an iterative data engine running across customer vehicles in passive shadow mode. Tesla engineered 221 software triggers over four months to monitor telemetry anomalies, driver interventions, and sensor disagreements. Across seven shadow mode cycles, Tesla curated an edge-case dataset of 1,000,000 video clips, 6,000,000,000 object labels annotated for depth and velocity, and 1.5 petabytes of storage. Karpathy emphasized that semantic triggers, such as tunnel entry and exit lighting swings, required human engineering intuition rather than fully automated mining. Supercomputing Infrastructure and Fleet Validation (20:31) Training Tesla's multi-camera temporal networks requires massive on-premises computational capacity. The model processes eight video streams through ResNet-style convolutional backbones, a transformer-based spatial fusion module, and temporal sequence layers before splitting into hierarchical branches, heads, and task terminals. To support this architecture, Tesla built an on-premises compute cluster featuring 720 nodes, 5,760 NVIDIA A100 GPUs (80 GB), and 10 petabytes of NVMe storage yielding 1.6 TB/s throughput. In the car, compiled models run on dual redundant System-on-Chips with in-house Neural Processing Units delivering 36 TOPS each. Prior to fleet release, the pure-vision system underwent evaluation against 6,000 categorized edge-case clips, track testing, and simulation. In shadow-mode trials, vision-only Automatic Emergency Braking exhibited superior precision and recall over the legacy radar-fusion stack. Following deployment, the pure-vision build logged 1.7 million autonomous Autopilot miles across 15 million driving miles with zero crashes, giving the engineering team confidence in the safety profile of camera-only autonomy. Karpathy reaffirmed that visible-light cameras provide necessary and sufficient information for autonomous driving, pointing to human drivers as biological proof. While acknowledging the theoretical interest of alternative sensors such as thermal imaging or radar bounce tracking, he concluded that commercial autonomy hinges on solving vision through fleet-scale data and compute rather than adding peripheral hardware.

在CVPR 2021自动驾驶研讨会上，时任特斯拉AI高级总监Andrej Karpathy阐述了特斯拉全面放弃毫米波雷达与高精地图、转向纯视觉架构的技术路线。他指出，依托大规模量产车队数据引擎、离线自动标注与高性能训练集群，合成神经网络在深度与运动学估计上已超越异构传感器融合，纯视觉不仅在技术上完全可行，更是全球规模化落地的核心路径。 放弃雷达与高精地图的根本考量 (6:00) 传统自动驾驶行业普遍依赖激光雷达与预制高精地图（HD Maps），将在线驾驶任务降维为在预先扫描构建的环境中进行定位。Karpathy认为，全球道路基础设施处于动态变化中，维护高精地图的成本极为高昂，无法支持数百万辆汽车的全球化落地。相比之下，特斯拉要求车辆依靠环绕车身的8颗摄像头（运行帧率为36 fps，每秒产生约800万比特的环境约束信息），完全在线实时推断车道拓扑、交通信号关联与障碍物几何关系。 在传感器配置上，特斯拉在演讲三周前已正式从出厂新车中彻底剔除毫米波雷达。Karpathy援引埃隆·马斯克的观点指出，当视觉精度已达到雷达百倍以上时，强行融合会适得其反。车载雷达虽然提供直接的距离与纵向速度测量，但缺乏垂直分辨率，容易将下水道井盖、天桥或金属接缝误判为前方静止障碍物，并且在横穿车辆与对向车流的追踪上表现极差。为了消除雷达噪声引发的幽灵刹车，并终结维护异构传感器融合分支带来的庞大工程负债，特斯拉选择裁撤独立雷达与融合团队，将工程资源全力押注在统一的纯视觉管线上。 影子模式与离线自动标注引擎 (12:04) 为使神经网络在米制深度、速度与加速度预测上全面超越雷达，特斯拉将测距问题转化为大规模端到端有监督学习任务。针对这一目标，特斯拉构建了一套无需满足实时性限制的离线自动标注系统。与车载端20毫秒严苛延迟预算的部署模型不同，离线管道运行在云端集群，采用容量巨大、无法部署在车端的重型网络，并利用前向与后向时序的“后见之明”进行轨迹双向优化，从整段行驶片段中生成高精度的3D标注与动态轨迹。 为捕获长尾极端案例，特斯拉在量产车队中以影子模式运行了221个定制软件触发器。这套机制在不介入车辆操控的前提下，持续监控两套视觉网络的不一致性、主副摄像头偏差以及“人类驾驶员踩油门但模型误判减速”等异常事件，自动上传短视频片段。通过7轮影子模式迭代，特斯拉累积了包含100万个高难度多样化视频片段（通常为10秒长、36 fps）、60亿个带有精确深度和速度的有监督目标标注，总数据量达1.5 PB。训练出的网络不仅能在严重暴风雪中稳定保持目标时空持久性，还能在沙尘暴完全遮蔽前车或前方车辆抛洒大面积杂物时实现平滑无缝的轨迹插值。 多任务神经皮质与端到端软硬件垂直整合 (18:25) 在模型结构设计上，特斯拉构建了模拟“合成视觉皮质”的多任务神经网络。8路视频流首先输入ResNet风格的卷积骨干网络提取图像特征，随后通过Transformer模块完成跨相机的多视角空间融合；接着利用Transformer、循环神经网络（RNN）或3D卷积进行时序融合，以捕捉多帧时序上下文。时序融合层下游连接着由表头、主干和末端构成的分层分支网络。这种共享骨干网的架构不仅有效摊销了车载推理的前向传播计算成本，还允许团队内约20名全职算法工程师独立调优特定终端预测任务（如特定障碍物速度），而无需频繁扰动整体网络结构；全团队会定期对共享骨干进行端到端反向传播整合。 该网络架构依托全栈垂直整合支撑：在云端训练侧，特斯拉搭建了包含720个节点、共计5760张80GB NVIDIA A100 GPU的超算集群，配备10 PB且吞吐量达1.6 TB/s的热NVMe存储网络，Karpathy估计其算力在当时位列全球第五左右；在车端执行侧，模型经过图优化、层融合、整型量化与阈值校准后，部署至自研FSD芯片的双冗余SoC上运行，单颗NPU算力约为36 TOPS。这种从自研芯片、超算基础设施到模型架构的软硬件协同设计，消除了对第三方软硬件供应链的依赖。 实测对比与主动安全量产验证 (23:48) 在封闭试验场的极端刹车对比测试中，老款雷达与视觉融合架构在测试车全力急刹时，因雷达回波在车身俯仰和几何突变下严重散射，导致追踪器在单次制动中丢失目标重置多达6次，使得下游规划层读取到的目标出现剧烈速度跳变；而纯视觉模型输出的原始未经平滑的数据保持了全程连续稳定跟踪。在过天桥场景下，由于雷达无法区分高空桥面与车道内停放的汽车，老款融合架构在视觉模型出现几帧速度微弱负值的噪声波动时，即触发虚假关联导致幽灵刹车，纯视觉架构则凭借相机的垂直分辨率彻底消除了该失效模式；在接近完全静止障碍物测试中，纯视觉网络亦比以往犹豫的融合系统更早识别障碍物并执行减速。 在发布交付前，该纯视觉系统通过了包含约70个类别、6000个精选严苛视频片段的基准测试集持续评估，覆盖急刹、强行插队、罕见车型等场景，并在仿真平台、试验场物理测试及量产客户车队的影子模式中完成验证。在影子模式下，其自动紧急制动（AEB）的精准度与召回率均超过了运行中的雷达融合版本。全面推向车队后，纯视觉版本在累计行驶约1500万英里、其中Autopilot接管运行170万英里的历程中保持了零碰撞记录，为纯视觉在主动安全与辅助驾驶中的全面替换提供了实证数据支撑。 Karpathy总结强调，可见光摄像头在物理世界中已包含自动驾驶所需的全部充分信息，且具备无可替代的大规模部署成本优势；然而这一纯视觉路线高度依赖大规模量产车队以获取极端长尾工况。问答环节指出，目前如进出隧道时光照突变等高级语义边缘案例的触发器构建仍需工程师大量人工直觉介入，数据闭环的全自动化生成仍是尚未完全解决的工程挑战。

[12:04] Andrej Karpathy: When you have a large clean diverse data set and you train a large enough neural network on it, what I've seen in practice is, in the words of Ilya Sutskever, success is guaranteed. [8:53] Andrej Karpathy: when radar and vision disagree, which one do you believe? Vision has much more precision, so better to double down vision than do sensor fusion. [0:00] Andrej Karpathy: we have these metallic objects traveling incredibly quickly and with very high kinetic energy, and we are putting meat in the control loop into the system.

[0:00] i can see andre is the screen sharing [0:03] working [0:05] i hope so let's try [0:09] i'm embarrassed to say that i barely use [0:11] zoom i know this is shocking [0:14] yes [0:17] [Music] [0:26] okay yeah seems still to work [0:31] even with limited experience okay so [0:34] uh it's a pleasure to introduce andre [0:37] uh i have to say i think he doesn't [0:39] really need a lot of introduction [0:41] uh but yeah um he is the senior director [0:44] of ai at tesla [0:45] and he basically supervises uh and leads [0:48] the team [0:49] responsible for all the neural networks [0:52] uh in the autopilot so [0:57] with this uh it's a pleasure to have you [0:59] here [1:00] and i'm really looking forward to your [1:02] talk [1:03] uh yeah great thank you for the [1:04] introduction and i'm very happy that i [1:06] was able to display some [1:08] impressive zero shot learning on zoom [1:10] okay [1:12] uh yes i'm very excited to be here thank [1:13] you for the introduction um [1:15] i would like to talk a bit more about [1:16] what tesla has been up for the last few [1:18] months [1:18] um so first to start uh [1:22] i guess we are here at the cbpr uh [1:24] workshop on autonomous driving so i [1:25] think i'm preaching to the core a little [1:26] bit [1:27] i'd like to start with some slides on [1:28] like why we're doing all of this and i [1:30] think [1:30] unfortunately we're kind of like in a [1:32] bad shape when it comes to [1:33] transportation in society [1:34] and uh basically the issue is that we [1:36] have these metallic objects traveling [1:38] incredibly quickly [1:40] and with very high kinetic energy and we [1:42] are putting meat in the control loop [1:44] into the system and so this is quite [1:46] undesirable and really fundamentally [1:47] what it comes down to is [1:49] people are not very good at driving they [1:50] get into a lot of trouble uh in [1:52] accidents [1:52] uh they don't want to drive and also [1:56] in terms of economics we are involving [1:58] people in transportation and of course [1:59] we'd like to automate transportation and [2:01] really [2:01] reap the benefits of that automation in [2:03] society so we all [2:05] i think are on the same page that it [2:06] should be possible to replace the meat [2:08] computer with a silicone computer [2:09] and get a lot of benefits out of it in [2:11] terms of [2:13] safety and convenience and economics so [2:15] in particular [2:16] silicon computers have significantly low [2:18] latencies they [2:19] have 360 degree situational awareness [2:22] they are fully attentive and never check [2:23] their instagram [2:25] and uh alleviate all of the issues that [2:27] are presented and so i think [2:29] uh this is not super shocking and a lot [2:31] of people have been kind of [2:32] really looking forward to this kind of [2:33] feature this is a frame from irobot [2:36] a really good movie if you haven't seen [2:38] it and here will smith is about to drive [2:40] the car manually and uh this other [2:42] person here is shocked that this is [2:43] going to happen this is ridiculous like [2:45] why would you actually have a huge [2:46] driving cars [2:47] um and so i think this is not very far [2:50] from [2:50] truth and this movie is taking place in [2:52] 2035 uh so i think by then [2:54] um actually i think this is a pretty uh [2:56] present prediction [2:58] uh so what i think is kind of unique [2:59] about tesla and uh [3:01] in our approach to autonomy is that we [3:03] take a very incremental approach towards [3:05] this problem [3:06] so in particular we already have [3:07] customers with the autopilot package and [3:09] we have millions of cars [3:10] and uh the the autopilot software is [3:12] always running and providing active [3:14] safety features and of course the [3:15] autopilot functionality [3:17] so we provide additional safety and [3:19] convenience to our customers today [3:20] and also the team is working on full [3:22] self-driving ability [3:24] uh so i would really like to very [3:26] quickly speak to [3:27] uh the value day that we provide just to [3:29] give you some examples of what the team [3:30] is up to [3:31] over uh sort of doing today uh so here's [3:33] an automatic emergency braking scenario [3:35] uh so the driver here is proceeding [3:37] through an intersection and you can see [3:39] trouble coming here [3:41] so the person kind of crept up from here [3:43] and the car sees the pedestrian [3:45] object detection kicks in and we slam on [3:47] the brakes and we avert the coalition [3:49] here's an example of a traffic control [3:51] warning so this person is probably not [3:53] paying attention potentially on the [3:54] phone [3:55] is not breaking for the traffic lights [3:56] ahead but we see that these are relevant [3:58] traffic lights they are red [4:00] and we beat and the person immediately [4:01] starts slowing and probably [4:04] you know does not actually enter the [4:05] intersection uh these are examples from [4:07] uh pedal misapplication mitigation pmm [4:11] here a person is unparking from their [4:13] driving spot and [4:14] they are trying to turn and then they [4:16] mess up and they accidentally floor it [4:19] so they floor it right there and the [4:21] system kicks in it sees pedestrians [4:22] ahead of us and it [4:23] slams on the brakes and converts what [4:26] would be a very gnarly situation in this [4:27] case [4:28] um here's another last scenario i can [4:31] show briefly [4:31] so this person is trying to park and [4:34] they turn to the right here [4:35] and they intend to press the brake but [4:37] they actually floor it and [4:39] the system again sees that there's no [4:41] driving path forward and actually slams [4:42] and breaks and inverts [4:44] you know in this case this person is not [4:45] going to be swimming in this river [4:47] imminently uh so these are some examples [4:49] of the value that we are providing today [4:50] in this [4:51] incremental autonomy fashion but of [4:53] course the team is actually also tonight [4:57] sorry about that uh the team is um [4:59] working primarily on [5:00] the fsd functionality which is our [5:03] wholesale dragon suite [5:04] so here we have about fs we have the ssd [5:07] beta product and it's in the hands of [5:08] about 2000 customers [5:10] and this is an example from one of the [5:11] customers driving around san francisco [5:13] and uh these people are posting videos [5:15] on youtube so you can check out a lot of [5:17] videos but we're showing here on the [5:18] instrument cluster [5:19] is we're showing all of our predictions [5:21] so you're seeing some road edges [5:22] some lines some objects and the car of [5:25] course is navigating around autonomously [5:27] here in san francisco environment [5:30] now we of course drive this extensively [5:32] as engineers [5:34] and so it's actually fairly routine for [5:36] us to have zero intervention drives i [5:37] would say in like sparsely populated [5:38] areas like palo alto and so on i would [5:40] say we definitely struggle a lot more in [5:42] uh very adversarial environments like [5:44] san francisco a lot of people [5:46] working in autonomy of course know all [5:47] about that as well [5:49] so this drive ends up being a fairly [5:51] long drive and it's zero interventions [5:53] in this case now one thing i'd like to [5:57] point out always whenever i showed the [5:58] video of tesla driving around [5:59] autonomously [6:00] in these city environments is you've [6:03] seen this before and you've seen it for [6:04] a decade [6:05] so or more so here's a waymo taking a [6:07] left at an intersection [6:09] and uh this is actually a pretty old [6:10] video i believe and so you've been [6:12] seeing stuff like this [6:13] for a very long time so what is the big [6:15] deal why is this impressive and so on [6:17] and i think the important thing to [6:18] realize and that i like to always stress [6:20] is that even though these two scenarios [6:21] look the same so there's a car taking a [6:23] left at an intersection [6:24] under the hood and in terms of the [6:26] scalability of the system things are [6:28] incredibly different [6:29] so in particular a lot of the competing [6:31] approaches in the in the industry [6:33] uh take this lidar plus hd map approach [6:35] and so the idea is that you take an [6:37] expensive sensor a lidar [6:38] on the top of the car and um it [6:41] basically gives you rangefinding [6:42] around around the vehicle in 360 and [6:44] gives you a point cloud and what you [6:45] have to do is you have to pre-map the [6:47] environment [6:48] with the lidar sensor and you have to [6:49] create a high definition map [6:51] and then you have to insert all of the [6:52] lanes and how they connect and all the [6:54] traffic lights and where they are and [6:55] you basically have to create a high [6:56] definition map [6:57] and then a test time you are simply [6:58] localizing to that map to drive around [7:01] and so the approach we take is vision [7:03] based primarily [7:05] so everything that happens uh happens [7:07] for the first time there in the car [7:08] based on the videos [7:09] from the eight cameras that surround the [7:11] car and so we come to an intersection [7:13] for the very first time and we have to [7:14] figure out where are the lanes how do [7:16] they connect what are the traffic lights [7:17] which ones are relevant [7:18] what what traffic lights control what [7:20] lanes everything is happening at that [7:21] time [7:22] um on that car and we don't have too [7:24] much high definition sort of information [7:25] at all [7:26] and this is actually a significantly [7:28] more scalable approach because [7:30] if our product is on a scale of millions [7:32] of customers [7:33] on earth and so it's actually quite [7:35] unscalable to [7:36] uh to actually collect build and [7:39] maintain [7:39] these high definition lidar amounts it [7:42] would be incredibly expensive to keep [7:43] this infrastructure up to date [7:45] and so we take the vision-based approach [7:46] which of course is much more difficult [7:48] because you actually have to get to [7:49] get neural networks that uh function [7:51] incredibly well based on the [7:53] videos uh but once you actually get that [7:55] to work it's a general vision system [7:57] and can in principle be deployed [7:59] anywhere on earth [8:00] so that's really the the problem that we [8:02] are solving now [8:03] in terms of the cartoon diagram of the [8:05] typical sensing suite of an autonomous [8:07] vehicle [8:09] these are kind of like the typical [8:10] sensors you would see and as i mentioned [8:11] we do not use [8:12] high-definition maps and we do not use [8:14] lidar we only use uh video cameras [8:17] and in fact um the um vision system that [8:20] we have been building over the last few [8:22] years [8:22] has been getting so incredibly good that [8:24] it's kind of leaving a lot of the other [8:26] sensors in the dust [8:27] and so actually um the [8:30] cameras are doing most of the heavy [8:32] lifting in terms of the perception that [8:33] you've seen in the car [8:34] and actually it's gotten to the point [8:36] that we are able to start removing some [8:37] of the other sensors because they are [8:39] just becoming these crushes that you [8:40] start to not really need at all [8:42] so actually three weeks ago we started [8:44] to ship cars that have no radar at all [8:46] so we deleted the radar and we are [8:48] driving on vision alone [8:50] in these cars and the reason we are [8:52] doing this i think is [8:53] well expressed by elon and suite he's [8:55] saying uh when radar and vision disagree [8:57] which one do you believe [8:58] vision has much more precision so better [9:00] to double down vision than do sensor [9:01] fusion [9:02] and what he's referring to is basically [9:04] like vision is getting to the point [9:05] where [9:06] the sensor is like 100x better than say [9:08] radar [9:09] then if you have a sensor that is [9:11] dominating the other sensor and so much [9:13] better then the other sensor is actually [9:14] starting to like really contribute it's [9:16] actually holding back [9:17] and it's really starting to attribute [9:18] noise uh to the former system [9:20] and so we are really doubling down uh on [9:22] the vision only approach [9:24] and um actually in this talk what i [9:26] would like to primarily talk about is [9:27] how we have achieved [9:28] vision only control without radar and [9:30] how we have released this [9:32] quite successfully so far to defeat and [9:34] what it has sort of um [9:36] taken for us to for us to do that so [9:39] here i wanted to show a video um [9:42] basically showing what the input of the [9:43] system is so we have eight cameras [9:44] around [9:45] and these are fairly high definition [9:47] cameras and we have 36 frames a second [9:49] and you can see that basically you can [9:51] actually understand a lot about this [9:52] environment from this input [9:53] it's extremely information rich you're [9:56] getting a huge amount of constraints [9:57] eight million bits roughly of [9:58] constraints per second on the state of [10:00] the surroundings [10:02] and so it's incredibly information rich [10:03] compared to any other sensor you may [10:05] want to have in the car [10:06] and so this is the one that we're [10:07] doubling down on because this is [10:08] ultimately where all the [10:09] difficult um sort of scene [10:11] interpretation challenges lie [10:13] so we prefer to just focus all of our [10:15] infrastructure and development on this [10:17] and we're not wasting people working on [10:18] the radar stack and the sensor fusion [10:20] stack [10:20] we only have one team division team [10:24] now typically when we talk about vision [10:25] but the reason people are squeamish [10:27] about using vision [10:28] is and the sort of challenge that almost [10:30] always is pointed out is people are not [10:32] certain if um neural networks are able [10:34] to [10:35] do range finding or depth estimation of [10:37] these objects [10:39] so obviously humans drive around with [10:40] vision so obviously where our neural net [10:42] is able to process visual input to [10:44] understand the depth and velocity of all [10:45] the objects around us [10:46] but the big question is can our [10:48] synthetic neural networks also do the [10:49] same [10:50] and i think the answer to us internally [10:52] over the last few months that we've [10:53] worked on this is an unequivocal yes [10:55] and i'd like to sort of talk about that [10:57] so in particular the radar api [10:59] or like what the radar really gives you [11:01] is it gives you a quite accurate [11:03] depth and velocity measurement it's a [11:04] pretty direct measurement of the car [11:06] ahead of you [11:06] in very nominal situations so here i'm [11:09] showing [11:09] a random image on the left and you are [11:11] seeing the depth velocity and [11:12] acceleration recorded by the radar stack [11:14] for this car [11:15] and you can see that it's a bit wiggly [11:16] on the velocity so this is some kind of [11:17] a stop and go scenario [11:19] now when radar has a good lock on the [11:22] car in front of you it's given [11:23] incredibly good depth and velocity but [11:25] the problem with radar is like [11:26] once in a while at random it will give [11:28] you a dumb measurement [11:30] and you will not know when that is and [11:32] so it's incredibly hard to fuse [11:34] with vision in our experience so it [11:36] might suddenly see a spurious [11:38] stationary object because of the manhole [11:40] or because of the bridge [11:41] or crossing objects are not very well [11:43] tracked or oncoming objects are not very [11:45] well tracked [11:45] and it ends up contributing noise i'll [11:48] go into [11:48] more detail in a bit but for our [11:50] purposes what we want to do right now [11:52] is we want to match the quality of these [11:54] predictions using vision alone [11:56] so the question is how can you get a [11:57] neural net to predict depth velocity [11:58] acceleration [11:59] directly and do it with a very high [12:01] fidelity matching that of radar [12:04] so the approach we're going to take of [12:05] course is we're going to treat this as a [12:06] surprise learning problem [12:07] and we need a massive data set of depth [12:09] velocity acceleration on a lot of cars [12:11] and we're going to train a large enough [12:12] neural network and do a very good job at [12:14] that that's the [12:15] standard vanilla approach and that [12:16] actually works really well if you do it [12:18] properly [12:19] so here's what it took um [12:22] so the first component that you need is [12:23] an incredibly good data set [12:25] so when i talk about a good data set i [12:27] believe it has to have three properties [12:28] that are critical [12:29] it has to be large and millions of [12:31] examples [12:32] it needs to be clean so in this case we [12:34] have to have a very clean uh source of [12:36] depth velocity and acceleration for all [12:37] these cars [12:38] and it must be diverse so we're not just [12:40] talking about a single [12:42] you know driving club of going forward [12:44] on a highway we really have to get into [12:46] the edge cases and mine all the [12:47] difficult scenarios and i'm going to [12:48] show you some examples [12:50] and when you have a large clean diverse [12:52] data set [12:53] um and you train a large enough neural [12:55] network on it what i've seen [12:56] in practice uh is in the words of ilya [12:59] saskar saskar [13:01] success is guaranteed and so let me show [13:03] you some examples of that [13:04] so how are we going to so first of all [13:06] how are we going to achieve this data [13:07] set [13:08] so of course we need to collect training [13:10] data [13:11] the typical approach might be to use [13:12] humans to annotate [13:14] cars around us in three dimensions um [13:17] what we found actually works really well [13:18] is an auto labeling approach so it's not [13:21] pure humans just like annotating cars [13:23] it's it's an offline tracker as we call [13:25] it and it's an auto labeling process [13:27] for uh collecting data at the scale that [13:30] is necessary [13:31] uh so we need again millions of hard [13:32] examples so this is where the scale [13:34] comes from is that it's not labeled [13:35] purely by humans although humans are [13:37] involved [13:38] it's labeled automatically so here's an [13:40] example of some automatic labels we were [13:42] able to derive [13:42] four cars on the highway and the way you [13:45] do this [13:46] is because you're offline and you're [13:48] trying to just annotate a clip [13:49] you have a large number of benefits that [13:52] you don't typically have if you're at [13:53] test time [13:54] under strict latency requirements in the [13:56] car [13:57] so you can take your time to fully [13:59] figure out exactly all the objects and [14:00] where they are [14:01] you can use neural networks that are [14:02] extremely heavy they are not deployable [14:04] for various reasons you can use benefit [14:06] of hindsight because you know the future [14:08] not just the past [14:09] you can use all kinds of expensive [14:10] offline optimization and tracking [14:11] techniques [14:12] you can use extra sensors in this case [14:14] for example actually radar was one of [14:16] the sensors that we used for the auto [14:17] labeling [14:18] but there's actually a massive [14:19] difference between using radar at test [14:20] time and using it [14:22] in the offline tracker and that is that [14:24] at test time a radar can just report [14:26] a stationary track immediately in front [14:27] of you and you have 20 milliseconds to [14:29] decide if you're gonna break or not [14:31] if you're offline you have the benefit [14:32] of hindsight and so you can do a much [14:34] better job of [14:35] calmly fusing if you really want to and [14:38] so there is a massive difference there [14:39] and in addition you can of course [14:40] involve humans and they can do cleaning [14:42] verification editing and so on [14:45] um and so we basically found that this [14:47] was a massive lever to allow us to reach [14:49] the [14:49] scale and the label and quality and then [14:52] to get the diversity [14:53] uh that was also a massive fight so [14:55] here's some examples of [14:57] really tricky scenarios so here there's [14:59] some [15:00] i don't actually know exactly what this [15:01] is but basically this car um [15:04] drops a bunch of debris on us and we [15:06] maintain a consistent track [15:08] for the label and of course if you have [15:10] millions of labels like this [15:12] the neural net if it's a powerful enough [15:14] neural net uh will actually end up [15:15] learning to persist these tracks in [15:17] these kinds of scenarios [15:18] here's another example there's a car in [15:20] front of us [15:22] and i actually am not 100 sure what [15:23] happens in this case but [15:25] as you'll see there's some kind of a [15:26] dust cloud that develops here and [15:28] briefly occludes the car [15:30] but in the auto labeling tool we are [15:32] able to persist this track because we [15:33] saw it before [15:34] and we saw it after so we can actually [15:35] stitch it up and use it as a training [15:37] set for the neural net [15:39] here's one more example of an [15:40] adversarial scenario from heavy snow [15:43] again we can auto label this fine and [15:46] create a large collection of samples now [15:49] as we were working on this over a [15:51] duration of i want to say roughly four [15:52] months of really just focusing a lot of [15:54] the team on this problem of achieving [15:56] really good depth velocity and [15:57] acceleration [15:58] we've ended up developing 221 triggers [16:01] manually [16:02] that we were using to source data from [16:05] our customer fleet [16:06] and this is just an example um some [16:09] examples of the 221 triggers that were [16:12] used to collect all of these uh [16:13] diverse scenarios so for example we have [16:17] um shadow mode where we deploy a neural [16:20] network that is pretty good at [16:21] predicting depth and velocity and what [16:23] we do is we run it silently in the cars [16:25] of our customers but it's not actually [16:27] connected to control [16:29] what's driving is the legacy stack but [16:31] we're [16:32] we're basically running the new [16:33] measurements of depth and velocity and [16:34] we're for example looking at whether or [16:35] not they agree or disagree with the [16:37] legacy stack or with the radar [16:38] we're looking for other sources of gen [16:40] like for example if there's bombing box [16:42] jitter [16:42] detection jitter the main and the narrow [16:44] camera disagree [16:46] we predict that there's a harshly [16:47] decelerating object but the person seems [16:49] to not mind [16:50] um all kinds of disagreements between [16:52] different neural network [16:53] signals and you know there's a there's a [16:56] long list here [16:58] it took us a while to actually perfect [16:59] these triggers and all of them are [17:01] iteration and you're looking at what's [17:02] coming back [17:03] you're tuning your trigger and uh you're [17:05] sourcing data from all these scenarios [17:08] so basically over the last four months [17:10] we've done quite extensive data engine [17:12] we've ended up doing seven shadow modes [17:14] and seven loops [17:15] around this data engine here where on [17:17] the top right is where you begin you [17:18] have some seed [17:19] data set you train your neural network [17:21] on your data set [17:22] and you deploy the neural network in the [17:24] customer cars in shadow mode [17:25] and the network is silently making [17:27] predictions and then you have to have [17:28] some mechanisms for sourcing [17:30] inaccuracies of the neural net so you're [17:31] just looking at its predictions [17:33] um and then you're using one of these [17:35] triggers like i described you're getting [17:37] these scenarios where the network is [17:38] probably misbehaving [17:40] some of those clips end up going to unit [17:41] tests to make sure that we [17:43] even if we're failing right now we make [17:44] sure we pass later and in addition those [17:47] examples are being auto labeled [17:48] and incorporated into a training set and [17:50] then as a asynchronous process we're [17:52] also always data cleaning the current [17:53] training set [17:55] and so we spin this loop over and over [17:56] again until the network basically [17:58] becomes incredibly good [18:00] so in total we've done seven rounds of [18:02] shadow mode for this release [18:04] um we've accumulated one million [18:06] extremely hard diverse clips [18:08] and these are videos so these are you [18:10] should roughly think about say 10 second [18:12] clips [18:12] 36 fps something like that in total we [18:15] have about six billion [18:16] objects labeled cleanly for depth and [18:18] velocity and this takes up roughly 1.5 [18:20] megabytes [18:21] of storage uh so that gives us a really [18:24] good data set [18:25] of course that by itself does not [18:26] suffice so we have an incredibly good [18:29] ai team of uh that is designing the [18:32] neural network architecture [18:33] there's basically the layout of the [18:34] synthetic visual cortex in order to [18:36] efficiently process this information so [18:39] our architecture roughly looks like this [18:41] we have these images coming from [18:42] multiple cameras on the top all of them [18:44] are processed by an image extractor like [18:45] a backbone like [18:46] think resnet kind of style then there's [18:50] a multicam fusion [18:51] that fuses the information from all the [18:52] eight views um [18:54] and this is a kind of a transformer that [18:57] we use uh to fuse this information [18:59] and then we fuse information first [19:00] across all the cameras and then across [19:03] all of time and that is also done either [19:05] by transformer [19:06] by the current neural network or just by [19:08] three-dimensional convolutions we've [19:09] experimented with a lot of [19:11] kind of fusion strategies here to get [19:13] this to work really well [19:14] and then what we have afterwards after [19:16] the fusion is done is we have this [19:18] branching structure that doesn't just [19:19] consist of heads [19:20] but actually we've expanded this over [19:22] the last few last year or so [19:24] where you now have heads that branch [19:26] into trunks that branch into terminals [19:28] so there's a lot of branching structure [19:30] and the reason you want this branching [19:32] structure [19:32] is because there's a huge amount of [19:33] outputs that you're interested in and [19:35] you can't afford to have a single neural [19:36] network for every one of the individual [19:38] outputs you have to of course amortize [19:40] the forward pass [19:41] for efficient inference at this time and [19:43] so there's a lot of [19:45] feature sharing here the other nice [19:46] benefit of the branching structure is [19:48] that it decouples [19:49] at the terminals it decouples all these [19:51] signals so if i am someone working on [19:53] velocity for a particular object type or [19:55] something like that [19:56] i have a small piece of neural network [19:57] that i can actually fine-tune without [19:59] touching any of the other signals [20:01] and so i can work in isolation to some [20:03] extent and actually get something to [20:04] work pretty well [20:05] and then once in a while so basically [20:07] the iteration scheme is that [20:08] a lot of people are fine-tuning and once [20:10] in a while we do an upper [20:11] of all of the backbone end-to-end and so [20:14] it's a very interesting [20:15] uh sort of mechanism because we have uh [20:18] you know a team of roughly 20 people who [20:19] are [20:20] i would say training neural works full [20:21] time and they're all cooperating on the [20:23] single neural net [20:24] and so what if the workflow by which you [20:25] do that is pretty fascinating [20:27] and continues to be a challenge to [20:29] actually design efficiently [20:31] so we have a neural network architecture [20:32] we have a data set now training these [20:34] neural networks like i mentioned this is [20:35] a 1.5 petabyte data set [20:37] requires a huge amount of compute so i [20:39] wanted to briefly give a plug to [20:41] this insane supercomputer that we are [20:43] building and using now [20:44] um and uh you know for us computer [20:47] vision is the brand and part of what we [20:49] do and what enables the autopilot [20:50] and uh for that to work really well you [20:52] need a massive data set we get that from [20:54] the fleet but you also need to train [20:55] massive neural nets and experiment a lot [20:57] so we've invested a lot into the compute [20:59] in this case we have here a data center [21:01] we have a cluster that we're just [21:02] building [21:03] that is 720 nodes of atx a100 [21:06] of the 80 gigabyte version so this is a [21:08] massive [21:09] supercomputer i actually believe that in [21:12] terms of flops this is roughly [21:14] uh number five supercomputer uh in the [21:16] world [21:17] so there's actually a fairly significant [21:18] uh computer here um [21:20] we have 10 petabytes of hot or nvme [21:22] storage and [21:23] uh it's also an incredibly fast uh [21:26] storage so [21:26] i believe 1.6 terabytes per second this [21:29] is one of the world's [21:30] fastest file systems and we have a 60 we [21:33] have also a very uh efficient [21:34] fabric um that connects all of this [21:37] because of course if you're doing [21:38] distributed training across your nodes [21:39] you need your gradients to to be [21:41] synchronized very efficiently and of [21:42] course we are reading all of these [21:43] videos from [21:44] the file system and that requires uh [21:46] really fat bite as well [21:48] so uh this is a pretty incredible [21:49] supercomputer um [21:51] and uh so this is a gpu cluster uh next [21:54] up we're really hoping that [21:56] we're currently working on project dojo [21:58] which will take this to next level [22:00] uh but i'm not ready to sort of reveal [22:01] any more details about that at this [22:02] point [22:03] what i would like to do is i would like [22:04] to just briefly plug the supercomputing [22:06] team [22:06] uh they've been uh growing a lot um and [22:09] so [22:09] uh if if high performance computing for [22:11] this application and for training these [22:13] uh [22:13] crazy neural networks excites you then [22:15] we definitely appreciate more help [22:17] so please contact the super computing [22:18] team at supercomputing and tesla [22:20] to help us build these clusters on the [22:22] left here we have the compute nodes and [22:24] on the right this is a [22:25] network switch actually so the wires [22:28] here [22:28] is kind of like the white matter of this [22:30] uh synthetic cortex i guess [22:33] uh the other thing i wanted to briefly [22:34] talk about is um [22:36] i wanted to also mention briefly that um [22:40] this effort basically is incredibly [22:42] vertically integrated [22:43] uh in the ai team so as i showed you we [22:46] own [22:47] the vehicle in the sensing and we source [22:49] our own data and we annotate our own [22:50] data [22:51] and we train our on-prem cluster and [22:53] then we deploy all of the neural [22:54] networks that we train on our in-house [22:56] developed chip [22:57] so we have the fsd computer uh here that [23:00] has two socs fse chips here and they [23:03] have our own custom npu neural [23:05] processing unit here [23:06] at roughly 36 tops each and uh so [23:09] these chips are specifically designed [23:11] for the neural networks that [23:13] we want to run for fsd applications and [23:16] so [23:16] everything is very vertically integrated [23:17] in the team and i think that's pretty [23:19] incredible because [23:20] you get to really co-design an engineer [23:23] at all the [23:24] layers of that stack and uh there's no [23:26] third party that is holding you back [23:28] you're [23:28] fully in charge of your own destiny [23:30] which i think is incredibly unique and [23:31] very exciting [23:32] and then we have a deployment a pipeline [23:34] here where we take these neural networks [23:35] and we do a lot of [23:36] typical graph optimization fusion [23:38] quantization threshold calibration and [23:40] so on and deploy it on our chip [23:42] and uh we and this is in the customer's [23:44] cars and [23:46] these neural networks are under so now [23:48] what i'd like to show is [23:49] i first would like to show some [23:51] qualitative examples of some of our [23:52] results [23:53] in terms of the depth and velocity [23:54] predictions that we're able to achieve [23:56] by putting all these pieces together and [23:57] training these networks at scale [23:59] so the first example here i have a video [24:01] where this is on track testing so this [24:03] is an engineering car [24:04] and we asked it to slam on the brakes as [24:06] hard as it possibly can [24:07] so this is a very harsh braking here in [24:09] front of us even though it doesn't look [24:11] like that [24:11] in the videos this is very heartbreaking [24:13] um so what you can see on the right here [24:15] is you can see the outputs from the [24:17] legacy stack which had [24:18] radar vision fusion and from the new [24:20] stack which is vision alone [24:22] in blue so in the orange legacy stack [24:25] you can actually see these uh [24:27] track drops here when the car was [24:29] breaking really harshly [24:30] and basically the issue is that the [24:31] braking was so harsh that the radar [24:33] stack that we have [24:35] actually ended up uh not associating the [24:37] car and dropping the track and then [24:39] reinitializing it all the time [24:40] and so it's as if the vehicle [24:42] disappeared and reappeared like [24:44] six times during the period of this [24:46] breaking and so this created a bunch of [24:48] artifacts here [24:48] but we see that the new stack in blue is [24:50] actually not subject to this [24:52] behavior at all it just gives a clean [24:54] signal in fact here there's no smoothing [24:56] i believe on the blue [24:57] signal here this is the raw depth and [24:59] velocity that comes out from the neural [25:00] net [25:01] the final neuron uh that we released [25:03] with about three weeks ago and you can [25:05] see that it's fairly smooth here [25:06] and of course you could go into the [25:08] radar stack and you could [25:10] um you know adjust the height parameters [25:12] of the tracker like why is it dropping [25:13] tracks and so on [25:15] but then you are spending engineering [25:16] efforts and focus on a stack [25:18] that is like not really barking up the [25:20] right tree [25:21] and so it's better to again just focus [25:23] on the vision and make it work really [25:24] well [25:24] and we see that it is much more robust [25:26] when you train it at scale [25:28] then something like this here's another [25:30] example [25:31] um fairly infamous example of slowdowns [25:34] when there are cars [25:36] going below a bridge the issue here [25:38] again is that the radar does not have [25:39] too much vertical resolution [25:41] so radar reports a stationary object in [25:44] front of you [25:44] it's as it like the radar doesn't know [25:46] that there's if it's there's like a [25:48] stationary car in front of you or if [25:49] it's the bridge but [25:50] it cannot differentiate those two so [25:51] radar thinks that there might be [25:52] something stationary in front [25:54] and it's just like looking for something [25:55] in vision to tell it that it might be [25:57] correct and then we create a stationary [25:58] target and break [25:59] and so in this case in the legacy vision [26:01] uh predictions [26:02] which were already producing depth and [26:04] velocity but [26:06] because we were using radar the vision [26:08] inaccuracies were being [26:10] masked because um your bar [26:13] is only at radar association your bar is [26:15] not at [26:16] actual driving and so the depth and [26:17] velocity were not held up to high enough [26:19] bar [26:20] and so basically what happens is vision [26:22] reports a slightly [26:23] for a few frames reports a slightly too [26:25] negative velocity for the car [26:27] and then it associates to the stationary [26:28] object and the stack is like oh that [26:30] must be the stationary thing and then [26:31] you break [26:32] and so um this of course is much cleaner [26:36] and you see that the new stack [26:37] does not see this at all and there's no [26:39] track there's no uh slowdowns in this [26:41] case [26:42] um because we just get the correct depth [26:44] and velocity and vision obviously has [26:45] the vertical resolution to differentiate [26:47] a bridge from a car and whether or not [26:48] the car is slowing or not [26:50] so again you could go into the vision [26:52] step the radar stack or the sensor [26:54] fusion stack [26:55] and if you have an improved depth and [26:56] velocity you could change the fusion [26:57] strategy [26:58] uh but again you're just kind of like [26:59] doing dead work like this signal is so [27:01] good by itself [27:02] why would you why would you do that um [27:05] so [27:06] in this in this setting now we've we've [27:08] improved the situation quite a lot [27:10] here's another last example um we have a [27:12] stationary approach [27:13] again this is in track testing [27:15] environment uh this is an example of a [27:16] test that we would run [27:18] and we are just approaching this vehicle [27:19] i'm hoping to stop what you see in [27:21] orange in the legacy stack is that it [27:23] actually takes us [27:24] quite a bit of time for us to start [27:25] slowing and basically what's happening [27:27] here is that [27:28] the radar is very trigger happy and it [27:29] sees all these false stationary objects [27:31] everywhere [27:32] like everything that like sticks out is [27:33] a is a stationary target [27:35] and radar by itself doesn't know what [27:36] actually is a stationary car and what [27:38] isn't so it's waiting for vision to [27:40] associate with it and vision if it's not [27:42] held up to a high enough bar [27:43] is noisy and contributes um sort of [27:46] error [27:46] and the sensor fusion stack just kind of [27:48] like picks it up too late and so again [27:50] you could fix all that [27:51] even though it's a very gross system [27:52] with a lot of if statements and so on [27:54] uh because the sensor fusion is [27:55] complicated because the error modes for [27:57] vision and radar [27:58] are slightly are quite different but [28:00] here when we just work with vision alone [28:02] and we take out the radar uh vision [28:04] recognizes this object very early gives [28:06] the correct depth and velocity and [28:07] there's no issues [28:08] so we actually get an initial slow down [28:09] much earlier and we've really [28:11] like simplified the stack a lot and as [28:14] well [28:16] so just speaking very briefly to the [28:17] release and validation um [28:20] we've extensively validated this before [28:22] we of course uh ship this to customers [28:24] uh just some example uh numbers for the [28:27] validation itself [28:28] uh we've hand picked six thousand clips [28:29] in about 70 categories of scenarios like [28:31] harsh breaking crossing vehicles [28:33] uh different vehicle types environments [28:35] and so on and we run these uh [28:37] tests on all the commits of the build [28:39] and we run them also every day [28:41] periodically as we were trying to drive [28:42] up the performance on these clips [28:44] we also use simulation extensively for [28:46] validation we've also used simulation [28:48] for training although i haven't gotten [28:49] into that [28:50] but we've had some successes there but [28:51] we used primarily for validation at this [28:53] stage [28:53] and that's again like diversity of [28:55] scenarios trying to make sure that the [28:56] stack is performing correctly [28:59] we've done a lot of track testing and [29:00] i've shown you some examples of that [29:02] we've driven this extensively in the qa [29:03] fleet and we've also deployed this in [29:06] shadow modes [29:06] and seen that this stack performs fairly [29:09] well so in particular for example for [29:11] the [29:11] automatic emergency braking we actually [29:13] see a higher precision and recall [29:15] compared to the legacy stack that is a [29:16] fusion stack and so having seen all this [29:19] we've actually released this we've [29:20] accumulated about 15 million miles [29:22] so far and 1.7 million [29:25] of those have been on autopilot and so [29:26] far there have been no crashes now of [29:28] course we're running at [29:29] a massive scale here and so we do expect [29:32] some crashes at some point [29:33] uh the legacy stack uh has a crash [29:35] roughly every five million miles or so i [29:38] believe [29:38] and so we do expect some some ascendance [29:40] uh um [29:42] at some point but this uh the [29:44] improvements for the vision stack are [29:45] not sort of stopping so i think we're [29:47] very confident that [29:48] uh we're barking up the right tree here [29:50] and that we can actually get this work [29:51] really incredibly well [29:54] um i also want to briefly give a shout [29:56] out to other labeling as an incredibly [29:58] powerful [29:58] method for sourcing training data so [30:01] we're applying auto labeling to all of [30:03] the fsd tasks not just the depth and [30:04] velocity of the car in front of us [30:06] here's an auto labeler for pedestrians [30:09] and you can see that the tracks are very [30:11] smooth and this is again a completely [30:13] offline process [30:14] that we are applying not just to objects [30:16] but also to [30:18] a lot of static environments so this is [30:21] some example of a [30:22] 3d reconstruction so here we see a clip [30:24] and this is a vision only [30:26] estimate of the depths of all the points [30:29] and of course you don't actually want to [30:31] really deploy [30:32] these measurements directly into a car [30:34] because they are too raw [30:36] so what you actually want to do is you [30:37] want to again auto label and you want to [30:39] really think about what is it what is [30:40] the information i really need at test [30:41] time in the car [30:43] and what is like as close as possible to [30:45] action right [30:46] so you want to do all this work at [30:47] training time figure out exactly what [30:49] happened in the club [30:49] reconstruct the entire environment and [30:51] then you want to actually like [30:52] post-process the curb [30:53] at training time so you want to do as [30:55] much a training time and labeling time [30:57] as possible [30:58] and then you want to figure out what are [30:59] the tasks i really need at test time [31:01] and that's the thing that happens in the [31:04] car [31:05] you know subject to very strict latency [31:06] requirements you want to do as little [31:08] processing there as possible [31:10] so we see that this is a very powerful [31:13] lever on occurring [31:14] dss so in summary what i try to argue [31:17] and give you a sense of is [31:18] vision alone is actually in our is our [31:20] finding that this is perfectly capable [31:22] of death sensing [31:23] um it is an incredibly rich sensor in [31:26] terms of bandwidth [31:27] of information and doing this and [31:30] matching radar performance and depth and [31:31] velocity is incredibly hard [31:33] i believe it requires the fleet because [31:35] the data set that we were able to [31:36] achieve was critical in all these [31:38] performance [31:38] uh improvements and if you do not have [31:40] the fleet i'm not 100 sure how you can [31:42] source [31:43] all the difficult and diversity [31:44] scenarios that we did source [31:46] um because i believe that was critical [31:47] to getting this to work so [31:49] it's hard requires massive networks a [31:51] super computer and [31:52] a data engine and the fleet but all [31:56] these components are coming together [31:57] in um in a vertical integrated fashion [32:00] at [32:01] tesla ai and i believe that this makes [32:03] us uh [32:04] sort of uniquely positioned in the [32:05] industry where um we are [32:07] barking up the right tree and we have [32:09] all the puzzle pieces to make [32:10] to make this work and so if you are [32:13] excited about the [32:13] leapers that we're taking and uh the [32:16] networks and the work that we're doing [32:17] then i would encourage you to please [32:18] apply uh and join the team [32:20] and help us make this a reality so you [32:22] can go to tesla.com autopilot ai [32:24] it's a very simple process uh you upload [32:26] a resume and you [32:27] you have a blurb about some of the [32:29] things that you've done that are [32:30] impressive [32:31] and then that comes directly to us and [32:33] uh we'd love to work with you to make [32:34] this reality [32:35] thank you [32:39] thanks for this very very interesting [32:42] talk [32:43] uh we actually got quite some questions [32:46] uh since we are already over time [32:49] i will limit it to a few we are at the [32:52] you know the kind of nice point at the [32:55] end of the workshop so [32:57] you are a bit of a rough spot that you [33:00] have to answer more questions than maybe [33:02] other [33:03] speakers um and kind of i would like to [33:06] go [33:07] as a first one into the you know [33:10] kind of the auto lay maybe bit to auto [33:12] labeling but actually [33:13] the thing which really is necessary for [33:15] the auto labeling and this is [33:17] these um you know this definition of [33:20] these triggers [33:22] uh did you ever try to investigate if [33:25] you can [33:25] automatically automatically uh generate [33:28] those triggers [33:30] um that's an interesting question [33:33] um so the triggers are kind of like [33:37] they're designed based on what we're [33:39] seeing and what's coming back in the [33:40] telemetry from the cars [33:42] so we are seeing clips where maybe we we [33:44] didn't break [33:45] what the person did or vice versa [33:47] there's all kinds of disagreements with [33:48] like human driving and the legacy stack [33:50] and we're looking at this and typically [33:52] we try to have signals of [33:53] various um generality so for example [33:56] like a vision [33:57] radar uh disagreement is a very general [34:00] trigger it can source [34:01] all the disagreements um and then there [34:03] are very specific ones like maybe we [34:04] struggled actually for a while when we [34:06] were entering tunnels and exiting [34:07] tunnels because there was a lot of [34:08] brightness variation [34:10] so we struggled a lot there as an [34:11] example and we had to [34:13] specifically design some triggers for [34:14] that because we were not catching it [34:16] uh with a high enough frequency um [34:19] for any of these situations you actually [34:21] need to make sure they're represented [34:22] well in the training step [34:23] so for us this was a fairly manual [34:25] process but we do have a team [34:27] basically dedicated to doing this [34:29] full-time and they did this for four [34:30] months [34:33] so i don't like how do you automate this [34:36] i think is a very tricky scenario [34:37] because [34:39] again you can have these general [34:40] triggers but i think they will not [34:42] correctly represent uh the error modes [34:45] and i think it would be really hard to [34:46] for example automatically somehow have a [34:48] trigger [34:48] um that triggers for entering and [34:51] exiting tunnels that's something [34:52] semantic that [34:53] you as a person have to like intuit that [34:55] this is a [34:56] challenge for whatever reason and [34:58] actually go after it specifically [35:00] so um yeah it's not it's not clear how [35:03] that would work but i think it's a [35:04] really interesting idea [35:06] yeah uh so i thought it sounded the very [35:10] interest the question sounded very [35:11] interesting and [35:12] not easy to answer yeah uh [35:15] maybe another one which actually in [35:18] somehow let's say [35:18] two version uh showed up a bit was [35:23] kind of in your vision only approach [35:27] um kind of [35:30] could there be other sensor like you [35:32] know for example [35:33] thermal uh thermal imaging you know [35:36] kind of thermal cameras uh which could [35:40] help you in low light situation or [35:42] actually even the radar which could [35:44] kind of give you some very specific [35:46] information i think there was an example [35:48] of [35:49] radar signals bouncing through basically [35:52] the car before you and then you still [35:55] get somehow a signal of the [35:56] car you know one what a head [36:00] things like that you think really vision [36:03] vision is enough or [36:06] kind of you know maybe some special [36:08] sensors which could give you a certain [36:10] edge [36:11] yeah i think it's an interesting idea i [36:13] would say [36:15] um like clearly people people use vision [36:17] and visual spectrum [36:18] and they make it work and so there's an [36:21] infinity sensors [36:22] of varying economic costs actually [36:24] vision and uh in the visible [36:26] light spectrum is incredibly cheap [36:28] sensor right so the economics of this [36:30] are very [36:30] uh appealing and you can actually uh [36:32] manufacture and include this at scale [36:34] so it's a very cheap and uh nice sensor [36:37] and [36:38] i would say that basically we have proof [36:40] from humans that it is [36:41] um sufficient and i would say also that [36:44] um [36:45] vision has all the information uh for [36:47] driving so i would say that [36:48] basically vision is necessary and [36:51] sufficient in my mind [36:52] and um so we're primarily focusing on [36:55] that and you could go crazy with a lot [36:56] of sensors but [36:58] uh i think we're definitely currently [36:59] just doubling down on that alone [37:03] yeah thanks a lot um i think [37:07] we are now already more than five [37:08] minutes overtime [37:10] uh so [37:13] i let you go and don't ask other evil [37:16] lidar versus camera questions [37:20] and um with this i would actually like [37:23] to give some [37:24] uh kind of concluding remarks on the [37:27] workshop uh or better [37:29] uh you know move over to the last part [37:33] so uh before [37:38] yeah i would really like to thank the [37:41] whole team [37:42] without them this would have never been [37:45] possible