Meta's Hoose Sees AI CapEx Hitting $1 Trillion by 2027

Meta的Hoose预计2027年AI资本支出将达1万亿美元 AI代码量激增虽降低了单次变更的错误率，却推高了整体生产事故发生率，迫使Meta协同设计软硬件以防大规模集群停机。 Peter Hoose强调了当前AI基础设施建设惊人的资金与运营规模，将其数千亿美元的发展轨迹与阿波罗太空计划相提并论。他概述了Meta为满足这些需求所采取的物理与技术策略，包括快速部署数据中心“帐篷”、混合云扩展，以及通过全栈软硬件协同设计来减少集群故障。Hoose还探讨了AI生成代码带来的运营悖论——单次变更虽更安全，却被庞大的代码量和调试负担所抵消——同时他也提醒，未来的增长将在电力、硬件和物理场地等各方面面临严峻限制。 2025年全球AI基础设施支出（4500亿美元）超过了香港全年的GDP，也超越了长达十年的阿波罗太空计划总预算（3380亿美元）。 (0:01) 预计AI基础设施资本支出将在2026年达到7650亿美元，并在2027年接近1万亿美元，与美国全年的国防预算总额相当。 (0:01) 2025年AI开发者工具从简单的行内自动补全演进为全自主的Agent工作流，开发者参与度达到70%–80%。 (2:03) Meta将其快速部署的数据中心“帐篷”发展为数百兆瓦的在线容量，并推出了首个多吉瓦级训练集群Prometheus。 (3:24) 随着GPU集群规模突破10万台，未经优化的组件故障率大幅复合增加，可能导致多达30%的算力陷入闲置。 (8:32)

Meta's Head of Production Engineering, Peter Hoose, argues that the global AI infrastructure buildout has entered an unprecedented capital cycle—a 'Sputnik moment' rather than a moon landing—that demands fundamental architectural changes across hardware reliability, software operations, and multi-resource management. Capital Expenditure and the Sputnik Analogy (0:01) Global spending on AI infrastructure reached $450 billion in 2025, exceeding the annual GDP of Hong Kong and eclipsing the inflation-adjusted $338 billion decade-long cost of the 1960s Apollo space program. According to Goldman Sachs estimates cited by Hoose, this single-year spend is projected to climb to $765 billion in 2026, with industry projections approaching $1 trillion by 2027—a sum roughly equal to the entire annual United States military budget. Hoose contextualizes this trajectory not as the culmination of the AI buildout, but as its starting phase. Drawing a parallel to the launch of Sputnik rather than the lunar landing, he asserts that the industry is only at the beginning of an era of exponential infrastructure scaling and capital expenditure. Rapid Physical Scaling and Multi-Gigawatt Clusters (3:24) To meet physical capacity demands, Meta transitioned rapid-deployment data center 'tents' from confidential prototypes into an operational standard delivering hundreds of megawatts within months rather than years. These facilities undergo fault-injection and extreme-weather testing to bring power, land, and network capacity online rapidly. Meta supplemented this footprint by committing tens of billions of dollars to major cloud providers, bringing new external capacity online in days, and launching 'Prometheus,' its first multi-gigawatt cluster for frontier training. By early 2026, Meta’s compute footprint exceeded 1.3 million NVIDIA H100-equivalent GPUs, managed under unified software abstraction layers spanning NVIDIA, AMD, and in-house MTIA silicon alongside diverse CPU architectures. Cluster Reliability and Full-Stack Failure Mitigation (8:32) Unlike traditional distributed architectures where component faults remain isolated, massive AI training tightly couples tens of thousands of GPUs, meaning a single hardware flaw or broken cable can halt an entire training run. Citing an Alibaba research paper, Hoose noted that unoptimized failure frequencies compound from once every eight hours on 1,000-GPU clusters (GPT-3) to once every eight minutes on 100,000-GPU setups, which would idle 40,000 GPUs and waste approximately $250 million in capital expenditure. Meta addressed this failure cascade by integrating its full operational stack across silicon, firmware, diagnostics, and training frameworks. Rather than immediately evicting nodes with minor flags, Meta's control plane evaluates whether a fault actively impacts training before triggering an interruption. Through co-design with vendors like NVIDIA across successive GPU generations, Hoose reported Meta achieved an estimated 95% reduction in job-interrupting failures. The Operational Paradox of AI-Generated Software (14:09) Following the release of agentic tools such as Claude Code and Opus 4.5 in 2025, developer workflows shifted toward autonomous code generation, driving an estimated 8x increase in developer output. Concurrently, code quality per individual change improved, registering a greater than 50% drop in incidents per diff by early 2026. However, this surge in velocity produced an operational paradox: because total code volume increased much faster than per-diff defect rates declined, the absolute volume of production incidents rose. Concurrently, the human engineering toil required to diagnose and resolve each incident escalated by more than 120% year-over-year, leading platforms like GitHub to expand capacity planning from a 10x projection to 30x. The Everything Constraint and Operational Bottlenecks (20:34) Looking beyond current compute limits, Hoose warned of an emerging 'everything constraint.' As capital investment expands toward the trillion-dollar mark, bottlenecks will widen from GPUs alone to encompass widespread shortages in CPUs, system memory, disk storage, electrical power, and physical real estate. These physical deficits will force operators to prolong the lifespan of aging servers, compounding fleet heterogeneity and component failure rates. While autonomous agents have largely automated code generation, Hoose emphasized that critical operational responsibilities—including production deployments, legacy migrations, and end-to-end incident mitigations—remain unsolved frontiers for automated systems. Hoose concluded that navigating these scaling limits requires tight vertical control from silicon to consumer applications—such as multimodal low-latency systems powering Ray-Ban Meta glasses for visually impaired veterans. However, whether autonomous agents can evolve to handle complex operational remediation and relieve surging human engineering toil remains an open industry challenge.

Meta生产工程负责人Peter Hoose指出，全球AI基础设施正经历史上规模最大的建设浪潮，其资金与硬件投入已远超阿波罗登月计划；但在这一扩张背后，代码爆炸、超大规模集群的连锁硬件故障以及全方位物理资源瓶颈，正引发严峻的新型运维挑战。 基建投资规模与斯普特尼克时刻 (0:01) Hoose通过多组数据对比了当前AI基础设施的投资规模：20世纪60年代耗时十年且动员超50万人的阿波罗登月计划，按通胀调整后总支出约为3380亿美元；而在2025年单年，全球AI基础设施建设支出就达到了4500亿美元，已超过拥有750万人口的中国香港的年度GDP。依据高盛（Goldman Sachs）的估算，2026年单年基建支出预计攀升至7650亿美元，超过阿波罗与双子座计划总成本的两倍以上；到2027年，全球AI基建资本开支更预计接近1万亿美元，相当于美国军方的年度总预算。Hoose认为，尽管资金投入空前，但当前行业尚未迎来“登月时刻”，而是正处于太空竞赛初期的“斯普特尼克时刻”，基建投资与规模即将迎来更剧烈的指数级增长。 敏捷数据中心与混合云全栈布局 (3:24) 为了支撑指数级增长的算力需求，Meta推行了从底层硬件到应用服务的全栈式基础设施扩张。在物理设施层面，Meta将其原本处于保密阶段的快速部署型数据中心“帐篷”（tents）投入实战，在完成故障注入测试与极端气候验证后，已承载数百兆瓦的在线生产算力，将土地、电力和网络从规划到上线的周期由数年压缩至数月。在云端扩展上，Meta在一年之内将混合云合作从几千张GPU拓展到数百亿美元规模，使云服务商的新算力能够在数天内接入生产流量；2025年7月，Meta还上线了首个用于前沿模型训练的多吉瓦级集群Prometheus。截至2026年初，Meta全网部署的算力已超过130万张NVIDIA H100等效GPU，并通过统一的软件抽象层，屏蔽了NVIDIA、AMD、Meta自研MTIA芯片以及不同CPU架构之间的底层硬件差异。 超大集群硬件可靠性与协同减损 (8:32) 以配备72张GPU和72颗CPU并辅以液冷机架的GB200 Catalina系统为例，海量NVLink铜缆互连使系统复杂性急剧增加。传统分布式系统依赖机器独立故障的容错假定，但前沿AI训练要求成千上万节点紧密耦合，单一线缆损坏或信号丢失都可能导致整个训练作业停滞。Hoose引用阿里巴巴“Train Mover”论文的数据指出集群规模扩大带来的故障连锁效应：GPT-3训练（约1000张GPU）约每8小时发生一次故障，Llama 3训练（约1.6万张GPU）故障间隔缩短至2.5小时，而在未经优化的10万卡前沿模型集群中，故障频率将达到每8分钟一次，这会导致整个集群因作业中断与重启耗费30%以上的时间，相当于约4万张GPU空转，造成近2.5亿美元的资本浪费。为此，Meta通过贯通芯片、固件到训练框架的纵向协同设计，不再对轻微报错的服务器执行激进下线，而是由控制平面动态评估其实际影响，据Hoose称，此举将导致作业中断的故障率降低了约95%。 AI代码生成激增与运维悖论 (14:09) 随着行业在2025年从行内补全转向以Claude Code和Opus 4.5为代表的自主智能体（Agentic）开发，现代生产系统中的大部分代码已转由AI生成，开发者产出提升约8倍，长远预期达到10倍至100倍。Hoose指出，AI辅助生成的单个代码变更在安全性上有所提升，2025年底至2026年初单次变更引发的故障率下降了超过50%。然而，这却催生了显著的“运维悖论”：代码产出与交付总量的爆炸式增长，远超单次提交事故率下降的速度，致使生产环境中绝对故障数量总体攀升。为此，GitHub不得不将扩容预期从10倍调高至30倍以支撑代码吞吐量；与此同时，由于系统复杂性加剧，人类工程师在单次事故排查与缓解上花费的时间同比增加了120%以上，过去需1小时的任务现在需要1小时20分甚至3小时才能解决。 全方位资源瓶颈与自动化运维挑战 (20:34) Hoose警示称，当前从“斯普特尼克迈向登月”的进程面临着严重的基础设施瓶颈。随着资本开支逼近万亿美元大关，行业瓶颈正从单一的“GPU短缺”转变为涵盖CPU、内存（RAM）、磁盘存储、电力与土地机房的“全方位资源瓶颈”（everything constraint）。这种短缺迫使企业不得不延长旧硬件的在网服役寿命，从而加剧了异构硬件管理难度与底层设备故障率。另一方面，虽然AI智能体在代码编写环节已趋于成熟，但在端到端软件运维领域——包括自动化部署、跨多年的老旧系统迁移、故障应急缓解以及系统自愈——仍存在未被攻克的技术壁垒，仍高度依赖工程师团队与新兴智能体防护机制的配合。 Hoose总结指出，在经历近30年技术周期后，当前基础设施工程正处于历史上最大规模的建设中心，其稳定运转依赖于保持技术全栈自主、消除组织繁文缛节与顶尖工程人才的协同；但如何在全方位物理资源受限、代码爆炸与超大集群故障频发的现实下实现可靠的自动化运维，仍是整个计算产业悬而未决的核心命题。

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 7 月 2 日发布，记录了 Meta 生产工程（Production Engineering, PE）负责人 Peter Hoose 在 @Scale 大会上的闭幕主题演讲。视频简介介绍了 Peter Hoose 自 2010 年加入 Meta 以来长期统领基础设施工程，致力于保障从数据中心底层硬件、服务器与网络直到前端软件系统的高可用性、高性能与弹性扩展。 ## TL;DR Peter Hoose 深入剖析了全球 AI 基础设施建设在万亿级资本开支浪潮下的演进历程与工程极限挑战。Meta 通过快速部署的“帐篷”数据中心、多云异构硬件抽象层以及端到端全栈协同，将超大集群任务故障率削减了约 95%，成功筑牢了庞大算力底座。同时，他揭示了 AI 辅助编程普及带来的代码吞吐量激增与线上事故处理负担等新瓶颈，并强调当前技术节点远未达到“登月时刻”，而仅仅是全方位资源受限、长达数十年变革的“人造卫星时刻”。 ## Atmosphere & style 整场演讲充满史诗感与硬核工程的务实精神，主讲人以阿波罗登月等巨额投资对比开场，语气既宏大幽默又极具感染力。台下高频互动让氛围迅速升温，随后的技术拆解则在严谨数据与现场打趣间切换自如，既有万卡训练频发故障的危机紧迫感，又有成功压降 95% 故障率后放烟花式的自豪与释然。整场演讲呈现出一种在“物理学定律是唯一边界”的狂飙时代中，顶尖工程师亲手重塑技术历史的澎湃热情与敬畏之心。 ## Bulletin - [00:01](https://www.youtube.com/watch?v=lCgEybKXE34&t=1s) 全球 AI 基础设施投资迎来史诗级扩张，高盛预计 2026 年基建资本支出将达 7650 亿美元，到 2027 年全行业 CapEx 更有望逼近 1 万亿美元，规模远超当年阿波罗登月计划。 - [03:24](https://www.youtube.com/watch?v=lCgEybKXE34&t=204s) Meta 将快速部署型“帐篷”数据中心从理论探索推向生产级主流，把基建周期从数年压缩至数月，并依靠全栈软硬件抽象层整合了异构公有云与多吉瓦级 Prometheus 集群。 - [11:53](https://www.youtube.com/watch?v=lCgEybKXE34&t=713s) 面对超大规模训练中“每 8 分钟发生一次故障”的指数级级联挑战，Meta 依托端到端全栈协同与“机会性维修”策略，成功将硬件故障与训练中断率压降约 95%。 - [14:02](https://www.youtube.com/watch?v=lCgEybKXE34&t=842s) AI 辅助编码展现出鲜明的效能悖论：单次代码变更事故率降低了超 50%，但因变更总量激增且排障复杂度上升，导致事故绝对量剧增、排障耗时暴涨 120% 以上。 - [17:57](https://www.youtube.com/watch?v=lCgEybKXE34&t=1077s) 当前 AI 革命仅处于太空竞赛初期的“人造卫星时刻”而非真正的“登月完成”，算力瓶颈已从单纯的 GPU 紧缺泛化为 CPU、内存、存储全维度的全方位物理约束。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=lCgEybKXE34&t=1s) 2025 年全球 AI 基建支出达到 4500 亿美元，预计 2026 年将跃升至 7650 亿美元，单年规模即达到耗资 3380 亿美元阿波罗与双子座登月工程总和的两倍以上。 - [02:03](https://www.youtube.com/watch?v=lCgEybKXE34&t=123s) 开发范式在 12 个月内实现剧烈跃迁，从初级的行内代码自动补全快速转向以 Claude Code、Opus 4.5 为代表的智能体驱动开发（Agentic Development）。 - [03:24](https://www.youtube.com/watch?v=lCgEybKXE34&t=204s) Meta 生产工程团队将快速部署型“帐篷”数据中心推向规模化落地，盘活现有电力与土地储备，将建设周期由传统数年压缩至短短数月，并稳定输出数百兆瓦算力。 - [06:13](https://www.youtube.com/watch?v=lCgEybKXE34&t=373s) 为应对公有云网络抖动与硬件固件异构挑战，Meta 构建了全栈硬件抽象层，无缝屏蔽底层 GPU、CPU 及自研 MTIA 差异，统一向上层暴露纯粹算力。 - [08:37](https://www.youtube.com/watch?v=lCgEybKXE34&t=517s) AI 超大集群打破了传统分布式系统“单机故障易隔离”的容错设计哲学；跨数千机架的紧耦合训练使得任何细微硬件或线缆异常都可能导致全集群训练崩盘。 - [10:06](https://www.youtube.com/watch?v=lCgEybKXE34&t=606s) 在 10 万卡集群规模下故障频率可飙升至每 8 分钟一次；若缺乏极致的恢复优化，集群闲置算力将超 30%（约 4 万张 GPU），造成近 2.5 亿美元的闲置 CAPEX 损失。 - [11:53](https://www.youtube.com/watch?v=lCgEybKXE34&t=713s) Meta 凭借自研芯片到训练软件的全栈端到端控制能力，摒弃激进的即时维修，推行“带病运转与机会性修复”机制，将硬件故障与训练中断率显著降低 95%。 - [14:02](https://www.youtube.com/watch?v=lCgEybKXE34&t=842s) AI 辅助写代码使单次代码变更的事故率下降超过 50%，打破了代码质量下滑的刻板印象，但开发者提交的代码变更总量爆发式增长，冲垮了安全性提升红利。 - [15:20](https://www.youtube.com/watch?v=lCgEybKXE34&t=920s) 由于智能体生成逻辑复杂度高，人类开发者排查单一线上事故的时间同比暴增超 120%，亟需构建智能体安全护栏与自主运维工具以分担研发琐事（toil）。 - [16:49](https://www.youtube.com/watch?v=lCgEybKXE34&t=1009s) 端到端软硬件掌控在消费级产品落地中释放价值，Ray-Ban Meta 智能眼镜通过端云协同的多模态推理，成功为全美视障退伍军人提供实时视觉辅助。 - [17:57](https://www.youtube.com/watch?v=lCgEybKXE34&t=1077s) 行业当前的进展更接近太空竞赛早期的“人造卫星（Sputnik）时刻”而非最终的登月时刻，预计到 2027 年行业资本开支将达到 1 万亿美元，开启长达数十年的技术变局。 - [20:34](https://www.youtube.com/watch?v=lCgEybKXE34&t=1234s) 硬件瓶颈正在从单一的 GPU 告急扩散为 CPU、内存、存储与电力等“全方位约束”，迫使数据中心超期服役旧硬件并面对系统异构复杂度的指数级提升。 - [22:22](https://www.youtube.com/watch?v=lCgEybKXE34&t=1342s) 在大规模基础设施工程的极限扩张中，团队打破繁文缛节与条条框框，坚信“唯一允许争论的边界只有物理学定律”。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=lCgEybKXE34&t=1s) 登月工程与 AI 基建的史诗级资金竞赛 演讲在一场充满活力的现场互动中拉开帷幕。作为大会的压轴闭幕演讲，主讲人抛出了一个引人深思的数字：“3380 亿美元”。台下观众纷纷猜测这是某个国家的 GDP 或是大型数据中心的建设成本，而主讲人揭晓了这个深藏情怀的答案：这正是 20 世纪 60 年代，全美凝聚了超过 50 万人的心血、历时整整十年将人类送上月球并安全返回的阿波罗与双子座计划的总投入折现估值。这场登月史诗证明了一件事——只要拥有宏大的资金投入、卓越的技术突破与顶尖的工程能力，人类就能创造出不可思议的奇迹。 随后，他又给出了第二个数字——“4500 亿美元”。台下甚至有人猜测这是否是曼哈顿工程的成本，然而现实更加震撼：这笔金额不仅比阿波罗与双子座两项航天工程的总和高出约 50%，更是全球在 2025 年仅一年内投入到 AI 基础设施建设中的真实资金。作为参照，4500 亿美元大体相当于坐拥 750 万人口的整个中国香港的年度 GDP。然而，这仅仅是序幕。据高盛（Goldman Sachs）预测，2026 年全年全球在基础设施层面的投入将飙升至惊人的 7650 亿美元——这意味着仅用短短一年时间，整个科技行业就砸下了两倍于全套登月计划的惊人资本。 - **登月工程投资对比**：1960 年代阿波罗与双子座计划历时十年、动用 50 万人，总耗资约 3380 亿美元。 - **2025 年全球 AI 基建支出**：4500 亿美元，与香港全年的经济总量相当。 - **2026 年基建资本开支预测**：高盛预计将达到 7650 亿美元，单年规模即达到登月项目的两倍以上。 --- ### [02:03](https://www.youtube.com/watch?v=lCgEybKXE34&t=123s) 从代码补全到智能体驱动：开发工具的范式跃迁 巨额的资本注入在工程实践中带来了肉眼可见的裂变式飞跃。当台上询问最近几周有谁使用过 Claude Code 或 Codex 这类工具时，台下举手的比例迅速攀升到了七八成。就在一年前的 @Scale 大会上，当时的业界现状还显得尤为初级：AI 代码助手在很大程度上只是一种高阶的“自动补全（Inline Autocomplete）”，开发者键入几个字符，模型反馈几个字符，充其量也就是偶尔免费送上几行样板代码。 转折点在短短 12 个月内接踵而至。2025 年 5 月底，第一版 Claude Code 正式向公众发布；而真正的结构性震颤发生在当年的 10 月——伴随着 Opus 4.5 模型的推出，行业迎来了从行内代码补全向“智能体驱动开发（Agentic Driven Development）”的剧烈范式转移。如今，AI 智能体已经不再是实验室玩具，而是彻底融入了日常生产力工作流：从规划休假路线、学术研究、技术写作，到自动化生成生产代码，甚至全面接管大型基础设施的构建。 ```mermaid flowchart LR accTitle: AI 辅助编程发展演进 accDescr: 展示从传统的行内自动补全到智能体驱动开发的范式转变历程 subgraph Phase1["2025 年上半年"] A["行内代码补全<br/>(Inline Autocomplete)"] --> B["Claude Code 早期版本<br/>(2025 年 5 月初次公测)"] end subgraph Phase2["2025 年秋季及以后"] B --> C["Opus 4.5 模型发布<br/>(2025 年 10 月)"] C --> D["智能体驱动开发<br/>(Agentic Development)"] end ``` --- ### [03:24](https://www.youtube.com/watch?v=lCgEybKXE34&t=204s) 理论落地现实：Meta“帐篷”数据中心从机密走向主流 Meta 生产工程（Production Engineering, PE）负责人 Peter Hoose 随后登台阐释背后的硬核工程体系。PE 团队肩负着数以百万计的服务器集群的性能与可靠性，守护着服务全球数十亿用户的庞大基础设施生态。Peter 指出，上一届大会曾探讨如何从理论走向真实商业价值，而今天他要揭秘的是如何从架构理论一步步跨越到真实的物理世界。 ```mermaid flowchart TD accTitle: 数据中心帐篷可靠性工程迭代 accDescr: 展现从环境压力测试、故障注入到冗余调优的闭环可靠性提升流程 subgraph StressInput["环境与外部扰动"] E1["恶劣气候事件<br/>(风雨/极端温差)"] E2["物理磨损与施工意外"] E3["主动故障注入<br/>(DR 容灾测试)"] end subgraph CoreLoop["工程诊断与闭环优化"] A["高频/高破坏半径组件识别"] --> B["电气与机械架构优化"] B --> C["控制面与训练系统重构"] C --> D["软硬件冗余度精确配置"] end StressInput --> A D -.->|"持续迭代投产"| A ``` 在数据中心物理层面，Peter 分享了一段颇具戏剧性的幕后故事：在去年 @Scale 演讲时，他曾剧透 Meta 正在研发的快速部署型“帐篷（Intents）数据中心”方案，但由于彼时技术过于超前、尚处于第一代模型部署的纯理论探索期，相关内容甚至被法务和公关团队从最终公开的视频录像中紧急剪切。而在过去的一年里，这一原本被视为激进的理论方案已经转变为 Meta 核心战略的关键支柱： - **建设周期断崖式压缩**：利用“帐篷”模式，Meta 能够盘活现有的土地、电力和网络资源储备，将传统数据中心数年之久的漫长建设周期一举压缩至短短几个月。 - **可靠性反直觉跃升**：在经历整整一年的真实物理环境考验后，其抗风险能力远超预期。这并非运气，而是电气、机械、软件系统深度协同的结果。 - **面向失败的分布式工程设计**：团队通过灾难恢复（DR）演练、主动故障注入，并结合实际施工意外及极端天气数据，精准掌握了破坏半径最大、故障率最高的组件，进而对控制平面、冗余备份与容灾调度进行了有针对性的极限优化。 - **规模化生产落地**：一年之内，这项曾被称为天方夜谭的理论构想已发展为每天稳定输出数百兆瓦（Hundreds of Megawatts）生产级算力的坚实底座。 --- ### [06:13](https://www.youtube.com/watch?v=lCgEybKXE34&t=373s) 异构多云挑战、Prometheus 集群与跨硬件统一抽象 为了满足 AI 浪潮对算力的极度渴望，仅依靠自建帐篷数据中心仍然不够，Meta 毅然进军公有云，构筑起了全球规模最大的混合云架构之一。在短短一年时间内，Meta 在云端的规模从最初仅有数千张 GPU 的试验性小集群，暴增为价值数百亿美元的战略级合作伙伴部署；一旦公有云容量交付，Meta 的自动化工程栈能在短短几天内将其直接纳入生产集群接入真实流量，而非耗费数周或数月。 但这套庞大的多云混合体系同样带来了剧烈的架构冲击。过去，Meta 的内部软件长期运行在高度同构的基础设施上，享有近乎无限的跨数据中心对切带宽（Cross-sectional Bandwidth）；而公有云各供应商之间、供应商与互联网之间的带宽参差不齐且波动剧烈，迫使 Meta 彻底重构了针对大带宽优化的原有微服务网络架构。此外，每个云服务商都拥有独特的硬件构型、专属固件与软件栈，这进一步放大了底层复杂性。 面对复杂性，Meta 祭出了统一抽象与集群突破的核心打法： | 维度 | 传统内部同构环境 | 现代混合云与巨型集群环境 | |:--- |:--- |:--- | | **网络带宽特征** | 跨数据中心对切带宽近乎无限且高度一致 | 云厂商间及出网带宽差异悬殊，依赖自适应网络调度 | | **硬件与固件生态** | 统一标准化的内部自研白盒硬件与统一固件 | 各云厂商异构硬件、定制驱动、差异化固件深度共存 | | **算力交付响应** | 传统自建需数月到数年的基建周期 | 帐篷模式压缩至数月；云端交付数天内即可承接生产流量 | | **超大规模训练** | 早期中小型分布式计算集群 | 2025 年 7 月上线 Prometheus 多吉瓦级（Multi-Gigawatt）训练集群 | 为了彻底抚平这些异构裂痕，Meta 的核心武器是“全方位硬件抽象层”。无论是部署在物理帐篷、公有云提供商，还是在 7 月刚刚投产训练世界顶级大模型的首个多吉瓦（Multi-gigawatt）级超大集群“Prometheus”中；无论是搭载来自 NVIDIA、AMD 的主力 GPU，亦或是 Meta 自研的 MTIA 芯片，抑或是采用 AMD、ARM、Intel 的 CPU，底层的所有物理差异均被彻底抽象隐藏。对于上层业务、模型训练与推理系统而言，底层暴露出来的只有一个纯粹的概念——那就是无穷无尽的“算力（Compute）”。 正是凭借这套软硬协同的综合战略，Meta 在 2026 年初已坐拥超过 130 万张 H100 等效 GPU 算力集群，且整体算力规模正以日益加剧的加速度持续狂飙。 ### [08:37](https://www.youtube.com/watch?v=lCgEybKXE34&t=517s) 算力架构与可靠性模型的范式转变 Peter Hoose 将话题转向计算基础设施，展示了 Meta 的 GB200 机架架构——内部代号为“Catalina”。该架构包含两个计算机架，合计搭载 72 颗 GPU 和 72 颗 CPU，机架四周环绕着四个空气辅助液冷（air-assisted liquid cooling）机架。这套系统的工程复杂度极高，融入了大量前沿技术，例如此前 NVIDIA 团队分享的 NVLink 技术，仅在机架和背板内部就布设了数千条定制铜互连线缆（copper interconnects）。这些极限工程设计彻底打破了以往的系统设计假设，迫使工程团队必须重塑对系统“可靠性”（Reliability）的思考模型。 在过去二十年左右的传统分布式系统架构中，工程师遵循的是“面向故障设计”（design for failure）的哲学：集群容忍单点硬件损坏，若某个独立组件发生故障，运维系统并不焦虑，只需隔离损坏的机器，将受影响的任务迁移到别处，整个大集群依然照常运转，故障影响范围被严格限制在单机或局部。然而，超大规模 AI 模型训练（large-scale training）的出现从根本上颠覆了这套逻辑。训练任务并不是在单一服务器上跑，而是由横跨数千个机架、通过数万条线缆紧密交织在一起的庞大机器集群协同作业。任何单点部件的闪失或线缆异常，若系统缺乏针对性优化，都会直接导致整个大规模训练任务崩盘停摆。因此，Meta 必须彻底重构软件栈，不再把系统视作由独立服务器拼凑的集合，而是将其当成一个高度耦合、拓扑结构更接近复杂网络拓扑的统一集群来管理。 ```mermaid flowchart TD accTitle: 分布式架构与超大规模AI训练可靠性模型对比 accDescr: 对比传统分布式系统单机容错与超大规模AI训练系统全链路紧耦合故障模式 subgraph Traditional["传统分布式系统 (容错设计)"] direction TB T1["单节点发生故障"] --> T2["仅隔离故障机器并迁移任务"] T2 --> T3["全局集群继续运行<br/>影响严格局部化"] end subgraph ModernAI["超大规模 AI 训练 (强相互依赖)"] direction TB M1["单点铜缆 / GPU / 内存异常"] --> M2["跨节点分布式训练中断"] M2 --> M3["整个训练任务崩溃<br/>集群进入闲置等待"] end ``` ### [10:06](https://www.youtube.com/watch?v=lCgEybKXE34&t=606s) 故障复合叠加与闲置算力的巨额成本 为了说明维持训练任务持续运转的严峻性，Peter 引用了阿里巴巴此前发布的一篇关于其 TrainMover 解决方案的研究论文。该论文揭示了一个核心规律：随着训练任务规模从数百卡、数千卡扩展到数万张 GPU，微小的单点故障会发生级联复合（compound），导致任务中断率呈指数级飙升。从规模演变来看： - 早期 GPT-3 大致在 1,000 张 GPU 上运行了约一个月； - Meta 的 Llama 3 扩展到约 16,000 张 GPU 上连续运行了一个半月以上； - 如今的前沿模型（frontier models）则在接近 100,000 张 GPU 的超级集群上全力运转。 在千卡规模下，集群平均大约每 8 小时发生一次故障；扩展到 1.6 万卡时，平均无故障时间急剧缩短至约 2.5 小时；而一旦迈入 10 万卡门槛，集群预计每 8 分钟就会遭遇一次故障。Peter 直言，如果一个任务每 8 分钟就崩一次且无法瞬间恢复，那么系统基本上就处于“彻底瘫痪”（dead in the water）的状态。规模扩大带来的直接后果是系统有效训练速率的断崖式下跌，大量 GPU 将陷入被迫等待的闲置状态。若没有底层优化与极致的重启加速机制，在 10 万卡集群中，随时可能有多达 30% 以上的算力处于闲置，相当于约 40,000 张 GPU 在“白白烧电却毫无产出”。粗略估算，这意味着将近 2.5 亿美元的资本支出（CAPEX）处于空转损耗状态。在分秒必争的前沿模型军备竞赛中，每一次任务中断与重启不仅意味着巨额金钱浪费，更是无法挽回的战略时间损失。 - **千卡集群（~1,000 GPUs）**：平均约 8 小时发生一次硬件/系统故障。 - **万卡集群（~16,000 GPUs）**：故障间隔压缩至约 2.5 小时。 - **十万卡集群（~100,000 GPUs）**：平均每 8 分钟就会产生一次故障。 - **闲置与成本风险**：缺乏架构优化时，十万卡集群闲置算力可超 30%（约 4 万张卡），对应闲置 CAPEX 高达近 2.5 亿美元。 ### [11:53](https://www.youtube.com/watch?v=lCgEybKXE34&t=713s) 全栈协同与机会性维修：实现 95% 故障率压降 面对如此严峻的挑战，Meta 选择回归物理与软件底层，重新审视从内存、CPU、GPU、控制器到各类 I/O 接口的每一个细微故障源。目标非常明确：既要压低故障发生的原始频率，又要大幅压缩任务崩溃后的恢复耗时。Meta 与硬件伙伴深度协同，对软件、固件以及系统中的每一个元器件进行极限压力测试，并在此过程中重塑诊断测试工具。在反思中他们甚至发现，过去工程师信奉的传统运维维修法则本身也在制造中断——以往一旦检测到硬件有瑕疵，自动化监控便会立刻将其从集群中剥离替换，这种激进的维修行为本身就会硬生生打断正在运行的训练大任务。 Peter 强调，Meta 在应对该难题时拥有极具壁垒的“掌控自身命运”的独特优势：具备从自研芯片、定制服务器、底层模型到分布式训练软件的全栈（full-stack）端到端控制能力。这种端到端的视野带来了业界少有的全局协同能力：运维系统既能看清最底层硬件的亚健康与降级征兆，又能实时感知上层训练任务的运行状态。借此，Meta 实现了全新的诊断调度逻辑： 1. **任务受损即刻介入**：当底层硬件出现隐患且已经实质性影响到训练任务时，系统才会果断介入处理； 2. **带病运转与机会性修复**：当单台服务器出现轻微异常但上层训练任务并未受阻时，系统会选择“静默观察”，允许其继续留在集群内工作，直到集群因其他自然原因（如偶发的外部线缆故障）整体短暂停机重置时，才顺道统一进场，排查并替换损坏设备。 通过结合软件与固件的协同优化、与 NVIDIA 深度联合改进的诊断机制、跨世代硬件平台（从 A100、H100 到 GB200、GB300）经验知识的持续复利累积，Meta 最终成功将硬件故障与训练任务中断次数降低了大约 95%。Peter 风趣地指着屏幕上的烟花动效打趣道：“是的，屏幕上在放烟花，因为马上就要到独立日 7 月 4 号了，但这确实是一项令人惊叹的工程成就。” ```mermaid flowchart LR accTitle: Meta基于全栈可见性的机会性修复工作流 accDescr: 展现硬件状态与训练任务状态结合的决策机制，区分立即修复与机会性修复 subgraph Telemetry["全栈遥测感知"] H["底层硬件遥测<br/>(芯片 / 固件 / 服务器)"] J["上层任务监控<br/>(分布式训练作业)"] end subgraph Decision["智能决策联动"] D{"底层异常是否<br/>影响训练任务？"} end subgraph Action["维护与恢复策略"] A1["立即调度干预<br/>快速重组集群"] A2["继续保留在线运行<br/>(避免主动打断任务)"] A3["等待集群自然中断<br/>(机会性批量换修)"] end H & J --> D D -- "是 (任务受阻)" --> A1 D -- "否 (任务正常)" --> A2 A2 --> A3 ``` ### [14:02](https://www.youtube.com/watch?v=lCgEybKXE34&t=842s) AI 编程时代的矛盾：单次变更更安全，总运维负担却剧增 从底层算力硬件向上延伸至软件系统，业界正经历着从“人类为人类写代码”向“AI 智能体为 AI 智能体写代码”的根本性转变。Meta 内部系统中的绝大多数代码如今已由 AI 编写，开发者的生产力和交付吞吐量获得了爆发式提升。先前的演讲中曾提及开发效率实现了 8 倍提升，并且未来有望走向 10 倍乃至 100 倍。 针对这种技术演进，外界普遍担心引入 AI 写代码会导致代码质量下滑、事故频发。但 Peter 给出了一组打破大众刻板印象的真实数据：AI 生成的代码在单次变更的安全性上实际上更高，而非更危险。回顾 Meta 在 2025 年的运行数据，因代码变更引起的单次变更故障率原本长年处于平稳状态；但从 10 月、11 月以及年初开始，随着代码生成中 AI 渗透率的激增，单次代码变更的事故率呈现急剧下降趋势，在短短一年内，单次变更引发故障的概率下降了超过 50%。 然而，现实情况存在一个巨大的转折（caveat）：尽管单次代码变更的质量更加安全，但由于 AI 工具让写代码变得极其低门槛，开发者提交的代码差异与变更总量（diffs and changes）出现了爆炸式增长。提交频率的增长斜率远远冲垮了单次变更安全性提升带来的红利，最终导致事故的**绝对总量**反而显著上升。这并非 Meta 独有的困境，GitHub 约一个月前也曾发文透露，他们原先规划了 10 倍的系统容量来应对 AI 辅助编程，但实际面对的代码量洪峰却迫使他们必须扩容 30 倍才能勉强维持运转。 更棘手的挑战在于单次事故处理复杂度的提升。根据 Meta 对研发人员“琐事”（toil，指处理线上事故、根因排查、后续追查及编写修复代码等运营负担）的时间追踪，随着 AI 生成代码的激增，人类开发者在诊断、解决和缓解单一事故时耗费的时间同比暴增了 120% 以上。以前花 1 小时就能定位排查的事故，现在往往需要人类耗费 1 小时 20 分钟、2 小时甚至 3 小时去艰难理清由复杂智能体生成的代码逻辑。这正是当前行业必须大力攻坚智能体安全护栏（Agent Guardrails）、智能体自主运维（Agentic Operations）以及微软等团队在推进的高级调试系统（Advanced Debugging）的关键原因——业界必须利用智能体运维工具来分担人类沉重的排错成本，避免研发效能被泛滥的运维负担所吞噬。 ### [16:49](https://www.youtube.com/watch?v=lCgEybKXE34&t=1009s) 全栈掌控与 AI 赋能：Ray-Ban Meta 助力视障老兵 技术在内部开发环境落地之外，其现实成果已全面渗透至 Meta 的全系产品线。从 Threads、WhatsApp 到 Instagram，到处都能看到 Agent 行为与全新 AI 能力的身影。其中最令人自豪的落地实践，当属最新推出的 Ray-Ban Meta 智能眼镜公益项目——为全美视障退伍军人免费提供 Ray-Ban Meta 眼镜。 ```mermaid flowchart LR accTitle: Ray-Ban Meta 端到端视觉辅助架构 accDescr: 展示从眼镜传感器采集到云端推理模型再到语音反馈给用户的闭环流程 subgraph EdgeDevice["边缘设备 (Ray-Ban Meta)"] Sensors["摄像头与麦克风<br/>采集图像与音视频"] Output["语音交互播报<br/>还原周围世界"] end subgraph MetaInfra["Meta 云端与基础设施"] Infra["推理系统与服务器集群"] Models["多模态大模型"] end Sensors -->|"实时数据回传"| Infra Infra --> Models Models -->|"生成场景描述"| Infra Infra -->|"实时反馈"| Output ``` 这款眼镜充分利用内置摄像头和麦克风感知外部世界，捕捉周围的图像、音频与视频流，并将数据实时传输至后端推理系统与服务器上的大模型进行解析。随后，系统会向视障用户精准描述眼前的一切与交互细节，“从某种意义上说，这赋予了他们重新看见世界的能力”。这种体验只有在掌握全链条自主权（从底层芯片到终端产品乃至其间所有环节）的前提下才得以实现。这不仅是一项令人兴奋的技术突破，更为现实世界带来了真正深远的意义。 --- ### [17:57](https://www.youtube.com/watch?v=lCgEybKXE34&t=1077s) 登月还是人造卫星？重新校准 AI 演进的时间线 回顾一年前的对比，Meta 曾在演讲中用阿波罗登月计划打比方，彼时 2026 年在基础设施上的开销已达阿波罗计划的近两倍。过去一年见证了跨越式的狂飙：基础设施从单一的原型设计扩展到数百兆瓦，外部投入达到数百亿美元，全球布局从 100 兆瓦容量激增到数吉瓦（multi-gigawatt）规模的数据中心；计算硬件从 H100、A100 演进到 AMD 方案以及 NVIDIA GB200、GB300，任务可靠性全方位提升；开发方式也从基础行内代码补全跃迁至全流程的 Cloud Code 与 Codex。 然而，如果重新审视整体 AI 时间线，去年的判断可能“过于乐观了”。尽管过去一年的进展速度远超所有人预期，但当前的行业节点其实远未到达真正的“登月时刻”，而更接近人类太空竞赛伊始的“人造卫星（Sputnik）时刻”——这仅仅是一场长达二十年技术巨变的真正起点。 - **距离对比**：从地面到人造卫星轨道的高度，远远短于从人造卫星飞向月球的距离；当前的成就距离终极目标仍有漫长的征途。 - **资本支出预测**：预计到 2027 年，全行业资本支出（CapEx）将逼近 1 万亿美元（1 后面跟着 12 个零）。 - **万亿体量参照**：1 万亿美元不仅相当于当今全球最富有人士的全部身家总和（引得全场会心大笑），更相当于全美军所有军种及 100 万现役军人的全部年度预算总额。 --- ### [20:34](https://www.youtube.com/watch?v=lCgEybKXE34&t=1234s) 从“GPU 瓶颈”到“全方位约束”的工程现实 尽管会议上展示了诸多在故障排查、自动修复和智能运维（Agentic Operations）上的成果，但真正的系统瓶颈依然严峻。在模型训练中，硬件故障仍会无情打断作业流，训练算法的容错性以及检测修复系统必须持续进化以适配不断迭代的 GPU 架构。代码编写问题虽已初步攻克，但编码仅占实际工作的一小部分，部署上线、生产运维、故障消除、故障修复以及漫长的长尾技术迁移，依旧等待着 AI Agent 去探索。 更严峻的挑战在于硬件资源。那即将涌入的 1 万亿美元投资非但不会缓解紧缺，反而会把整个行业推向“全方位约束（everything constraint）”： - **瓶颈泛化**：从最初单一的 GPU 告急，蔓延为 CPU、内存、存储磁盘的全链条短缺。 - **机架寿命延长**：现有硬件被迫在数据中心超期服役，设备故障率随之显著攀升。 - **异构性剧增**：必须从一切可能渠道挖掘算力，机房异构度与系统复杂度呈指数级上升。 --- ### [22:22](https://www.youtube.com/watch?v=lCgEybKXE34&t=1342s) 物理法则之下的三十年未有之大变局 在跨越近 30 年的从业生涯中（其中近一半时间奉献给了 Meta），从早期的互联网与 Dot-com 泡沫，到移动计算的兴起，再到短视频与长视频的爆发，整个行业从未像当下这样以如此惊人的速度狂奔。不可思议的是，整个系统居然顶住了压力、稳定运转至今。这种奇迹背后的支撑要素与当年的阿波罗登月如出一辙： 1. **掌握自身命运**：端到端掌控核心系统栈。 2. **极致提速与破除繁文缛节**：打破条条框框，赋予团队绝对自主权；正如核心理念所贯彻的那样——“唯一允许争论的边界，只有物理学定律（The only thing that is up for debate is something that is not bounded by the laws of physics）”。 3. **团队与人才聚合**：齐聚现场的工程师将各自的经验与才华注入其中，这才是推动行业前行的原动力。 正因如此，眼下正是投身技术行业的绝佳历史节点；而致力于系统架构与可靠性工程的建设者，正亲手锚定人类历史上最大规模基础设施建设的最核心战场。 ## Notable quotes > [00:01](https://www.youtube.com/watch?v=lCgEybKXE34&t=1s) 3380 亿美元 — Peter Hoose > [00:01](https://www.youtube.com/watch?v=lCgEybKXE34&t=1s) 4500 亿美元 — Peter Hoose > [08:37](https://www.youtube.com/watch?v=lCgEybKXE34&t=517s) 面向故障设计 — Peter Hoose > [10:06](https://www.youtube.com/watch?v=lCgEybKXE34&t=606s) 彻底瘫痪 — Peter Hoose > [10:06](https://www.youtube.com/watch?v=lCgEybKXE34&t=606s) 白白烧电却毫无产出 — Peter Hoose > [11:53](https://www.youtube.com/watch?v=lCgEybKXE34&t=713s) 掌控自身命运 — Peter Hoose > [11:53](https://www.youtube.com/watch?v=lCgEybKXE34&t=713s) 是的，屏幕上在放烟花，因为马上就要到独立日 7 月 4 号了，但这确实是一项令人惊叹的工程成就。 — Peter Hoose > [16:49](https://www.youtube.com/watch?v=lCgEybKXE34&t=1009s) 从某种意义上说，这赋予了他们重新看见世界的能力 — Peter Hoose > [17:57](https://www.youtube.com/watch?v=lCgEybKXE34&t=1077s) 人造卫星（Sputnik）时刻 — Peter Hoose > [20:34](https://www.youtube.com/watch?v=lCgEybKXE34&t=1234s) 全方位约束 — Peter Hoose > [22:22](https://www.youtube.com/watch?v=lCgEybKXE34&t=1342s) 唯一允许争论的边界，只有物理学定律（The only thing that is up for debate is something that is not bounded by the laws of physics） — Peter Hoose ## Mentioned Peter Hoose, Meta, 高盛 (Goldman Sachs), NVIDIA, AMD, Intel, ARM, 阿里巴巴 (Alibaba), GitHub, 微软 (Microsoft), Claude Code, Codex, Opus 4.5, MTIA, H100, A100, GB200, GB300, Catalina, NVLink, TrainMover, Llama 3, GPT-3, Prometheus 集群, Ray-Ban Meta, Threads, WhatsApp, Instagram, 阿波罗计划 (Apollo program), 双子座计划 (Gemini program), @Scale 大会 ## Video description Peter Hoose is the head of Production Engineering at Meta. PE is a unique role that ensures Meta’s Infrastructure is not only reliable and efficient, but also scalable and high-performing from the lowest levels of the hardware in Data Centers, Servers and Networking, up to the front end software systems and products. Peter has been with Meta since 2010, prior to leading PE, Peter previously supported the AI, Data Infra, Core Systems and Network PE teams. Prior to Meta, Peter designed, built and automated large scale networks at NTT for customers including Twitter, Coca-Cola, Sony and more. Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=lCgEybKXE34)

[0:01] Peter Hoose: 3380 亿美元 [0:01] Peter Hoose: 4500 亿美元 [8:37] Peter Hoose: 面向故障设计 [10:06] Peter Hoose: 彻底瘫痪 [10:06] Peter Hoose: 白白烧电却毫无产出 [11:53] Peter Hoose: 掌控自身命运 [11:53] Peter Hoose: 是的，屏幕上在放烟花，因为马上就要到独立日 7 月 4 号了，但这确实是一项令人惊叹的工程成就。 [16:49] Peter Hoose: 从某种意义上说，这赋予了他们重新看见世界的能力 [17:57] Peter Hoose: 人造卫星（Sputnik）时刻 [20:34] Peter Hoose: 全方位约束 [22:22] Peter Hoose: 唯一允许争论的边界，只有物理学定律（The only thing that is up for debate is something that is not bounded by the laws of physics）

[13:52] Peter Hoose: We've been able to reduce the amount of failures and job interruptions by something on the order of like, I don't know, 95%.

[0:01] last talk, so we're going to finish [0:02] strong here. [0:04] I got a quick question to start this [0:05] off. So, [0:07] the first question is [0:09] actually just a number. And that's $338 [0:11] billion. [0:12] Does anyone know what this represents? [0:15] All at once, come on. Big energy, we've [0:16] got to finish last. Any guesses? [0:19] GDP of something, that's true. Any other [0:22] guesses? [0:24] What was that? [0:26] Data center buildout, pretty close. [0:28] Okay. This is actually something that's [0:30] near and dear to my heart and something [0:31] that I find actually very, very [0:32] exciting. And that's the amount of money [0:34] that we estimate that it took over the [0:36] course of a decade to put a person on [0:38] the moon. In the 1960s, over 500,000 [0:41] people came together and we spent about [0:43] $338 billion to put a person on the moon [0:46] and bring them back safely. And what [0:48] that tells us is with a large investment [0:50] and with a great technology and great [0:51] engineering, we can do some incredible [0:53] things. We can put people on the moon. [0:56] I've got another number for you. [0:58] $450 billion. Any guesses what that is? [1:03] What was that? [1:05] Manhattan Project. Ooh, that's an [1:07] interesting one. Okay, no. [1:09] This, despite being about 50% more than [1:12] the cost of the Apollo missions and the [1:15] Gemini missions combined, is the amount [1:17] of money that we've spent in 2025 on the [1:19] AI buildout. That's pretty massive. And [1:22] interestingly, somebody said, you know, [1:23] the GDP of a country previously. Well, [1:25] just for some perspective, $450 billion [1:28] is about the GDP of Hong Kong, home to 7 [1:31] and 1/2 million people. And that's how [1:33] much money went into the AI buildout in [1:35] 2025. [1:37] But that was last year. In 2026, Goldman [1:40] Sachs estimates that we'll spend $765 [1:42] billion on infrastructure throughout the [1:45] course of the year. Not in a decade, in [1:47] 1 year. Twice the cost of the Apollo and [1:50] Gemini missions inside of 1 year. [1:52] Now, I want to spend some time going [1:54] through what these investments have [1:55] gotten us over the course of the last [1:56] year, as well as what we're seeing in [1:58] the industry and in Meta and what we've [1:59] heard from many of our colleagues and [2:01] peers throughout today. But one last [2:03] show of hands, is anyone in the last [2:05] couple weeks here used Claude Code or [2:07] Codex or anything like that? [2:09] All right, I'm going to guess cuz I [2:10] can't see anything based on the lights [2:12] that we're at like 70-80%. [2:14] And that's actually a pretty big [2:15] difference because if you remember just [2:17] over a year ago, the last time I was on [2:19] stage at @Scale, things were quite a bit [2:21] different. You know, the code Claude and [2:22] Codex are sort of, you know, these [2:24] things we almost take for granted today. [2:25] But a year ago, the state of the art was [2:27] quite different. In fact, it kind of [2:29] looked something more like, I don't [2:30] know, auto complete. You typed a few [2:32] characters, you get, you know, a couple [2:33] more characters after that, maybe get a [2:35] few lines of code for free. [2:37] And things changed pretty dramatically [2:38] over the course of those 12 months. In [2:40] fact, it wasn't until the end of that [2:42] month in May of 2025 that the first [2:44] version of Claude Code was released to [2:45] the public. [2:47] But things changed really dramatically [2:48] around October. And I think that's when [2:50] all of us felt that seismic shift as [2:52] Opus 4.5 came out and we really shifted [2:55] away from something that looked like [2:56] in-line code completion to agentic [2:58] driven development. So in the last 12 [3:00] months, we've gone from something pretty [3:02] basic to a powerful tool that is now in [3:04] daily use for many of you in the room. [3:06] And it's doing everything from planning [3:08] my vacation that comes up after this to [3:10] research, to writing, to code generation [3:13] and more. It's building our [3:14] infrastructure. [3:15] But that's just scratching the surface. [3:17] And I want to dig into what takes what [3:19] it takes to make this all possible, the [3:20] infrastructure, the engineering, and the [3:22] products. [3:24] Hello everyone. I'm Peter Hues. I lead [3:26] production engineering at Meta and we're [3:28] responsible for the performance and [3:30] reliability of a fleet of millions of [3:31] servers serving some of the largest [3:33] products on planet Earth to billions of [3:35] people every single day. [3:36] Uh Srujan started us off earlier today [3:38] talking about going from theory to real [3:40] value. I'm going to close this out on [3:42] going from theory to reality and all the [3:44] things along the way that make that [3:46] happen. [3:47] Now let's do this by taking a look [3:49] behind the scenes at that infrastructure [3:50] development and build-out. I'm going to [3:52] start from the ground up with our data [3:53] centers. I'm going to follow this by [3:55] talking about the compute that's [3:56] involved in this, and lastly, I'll [3:58] connect this all together with the [3:59] systems and servers involved in that. [4:01] Now, let's start with data centers. Last [4:04] year I was up on this stage, I started [4:06] talking about some of the cool new [4:07] things that we were doing to be [4:08] innovative, to move faster, to bring us [4:10] optionality. [4:11] I might have been a little bit early in [4:12] that. So much early that in fact we had [4:15] to remove that from the recorded version [4:16] of the talk. True story, I was talking [4:18] about our investment in intents. [4:21] Intents at the time, we had just [4:22] deployed our first model of this into [4:24] production. We had a few servers in [4:26] this. We weren't even actually using [4:27] these live yet. It was that early along [4:28] the way. It was purely a theory. [4:31] Now, if you fast forward to today over [4:33] the course of the last year, this has [4:34] become a critical part of our overall [4:36] strategy. [4:37] Well, not only being fast in the [4:38] absolute sense, it allows us to take [4:40] advantage of our investments in land and [4:42] power and network and use those much [4:45] more quickly [4:46] and on the order of months instead of [4:47] the order of years. [4:50] Now, over the course of last year we [4:51] actually have a whole lot of real life [4:52] information as well. Now, we have [4:54] weather data, we have wear and tear [4:56] data, [4:57] we have all the sort of operational [4:59] experience that you gain over a year of [5:01] running these things. And one of the [5:02] things that really surprised us is the [5:04] reliability is actually far better than [5:06] we expected. [5:07] And that wasn't by accident. That was [5:09] the result of real hardcore engineering. [5:11] Electrical, mechanical, software, [5:14] systems, everything coming together to [5:16] optimize that system and make it better. [5:18] We used our DR planning and fault [5:20] injection and systems to disrupt it and [5:22] try these things out to try to create [5:23] failures. We experienced weather events. [5:26] We also experienced construction and [5:27] accidental events. All of these things [5:29] gave us data on how to understand which [5:31] components would fail the most [5:32] frequently [5:33] and with the largest blast radius. [5:35] That meant we could tune and optimize [5:37] the things that we repaired, that we [5:38] increased redundancy on, as well as our [5:41] systems that improve this is on top of [5:43] that, whether there's a control plane, [5:45] our training systems, and so on and so [5:46] forth. [5:48] And all of this is actually built on top [5:49] of the history that we've all done in [5:51] building large-scale distributed [5:52] systems. You know, whether that's [5:53] component failure or system failure or [5:55] services failure, we've designed these [5:56] things to fail and built around that. [5:58] With these tents, we assume that they'll [6:00] fail, we test it, and we iterate [6:02] forward. [6:03] Now all of this engineering and [6:05] investment means that now, 1 year later, [6:07] that theory [6:08] is reality and serving hundreds of [6:10] megawatts of production capacity every [6:11] single day. [6:13] But to really move fast, tents weren't [6:14] enough for us. We needed to build into [6:16] the cloud. We had to build one of the [6:17] largest hybrid clouds in the world, and [6:19] you heard from Sarupa and Dinesh and [6:21] Sargun about our efforts there. And [6:23] inside of 1 year, we went from just a [6:25] small cluster with a few thousand GPUs [6:28] to tens of billions of dollars invested [6:29] with our partners in this. It's massive. [6:32] And the exciting part about this is when [6:34] that capacity is handed to us, we can [6:36] bring it online and start serving real [6:37] production traffic not in months or [6:39] weeks or years, but just in a few days. [6:42] But it wasn't all easy, it wasn't all [6:44] for free. In fact, one of the things [6:45] that we [6:46] found that was quite unexpected is for [6:48] many years we've actually taken a lot of [6:50] the things in our homogeneous [6:52] environment for granted. One of these is [6:54] the cross-sectional bandwidth we have [6:55] between our data centers. In effect, [6:57] it's essentially unlimited. But as we go [6:59] to the cloud, we find that bandwidth to [7:01] any given provider, between those two [7:03] given providers or and the to those [7:04] providers and the internet, obviously [7:06] varies quite a bit. So all these systems [7:08] and services that have optimized for [7:09] that large cross-sectional bandwidth [7:11] that's consistent region by region, data [7:13] center by data center, had to be changed [7:15] and optimized. [7:16] We also went from a single homogeneous [7:18] hardware platform to something that's [7:20] unique per provider. Each provider has [7:22] their own systems, their own software, [7:24] their own components, their own [7:27] firmware, and all of these things add up [7:28] to increased complexity that we have to [7:30] modify and iterate and change our stack [7:32] to accommodate. [7:34] But we didn't stop there, cuz that [7:35] wouldn't be enough. In July, Prometheus, [7:37] our first large-scale multi-gigawatt [7:39] cluster, came online and is now training [7:41] some of the largest models in the world. [7:44] And all this together actually gives us [7:45] a really unique strategy. [7:47] And key to this strategy is [7:49] abstractions. Cuz whether that capacity [7:51] is in a tent or in a cloud or in a [7:54] multi-gigawatt facility, we abstract [7:57] that away from the end user. [7:58] Regardless of that capacity is using [8:00] GPUs from AMD or Nvidia or own custom [8:03] silicon like MTIA or it's using CPUs [8:06] from AMD or ARM or Intel, all of these [8:09] things [8:11] fade into the background because for the [8:12] products and services that use it, it [8:14] just looks like compute. [8:16] It's pretty mind-blowing. [8:17] Now, across the fleet, all this has [8:19] added up in this all-in-one approach has [8:21] given us over 1.3 million H100 [8:24] equivalent GPUs at the start of 2026 and [8:27] that is increasing at an increasing rate [8:28] every single day. [8:30] Massive stuff. [8:32] Speaking of GPUs and servers, let's dig [8:34] into that a little bit as well and see [8:35] what we've seen over the last year. [8:37] Starting with compute, this is our GB200 [8:39] rack. It's called Catalina. There's two [8:41] compute racks in here and then combined [8:43] they have 72 GPUs and 72 CPUs and [8:46] they're surrounded by four air-assisted [8:48] liquid cooling racks. And I think if you [8:49] heard from the folks at Nvidia earlier [8:51] today, these things are quite complex, [8:53] but they also have a lot of new novel [8:55] technologies in them. NVLink, one of the [8:57] things they talked about in just that [8:58] specific example, has thousands of [9:00] custom copper interconnects inside of [9:02] these racks and backplanes. And many of [9:04] this has new feats of engineering that [9:06] really means it means that we have to [9:07] change the model of how we think about [9:09] reliability in a system like this. [9:12] And if you go back for the last 20 years [9:13] or so of how we thought about systems [9:15] and reliability, we've designed these [9:16] things to fail. If any given single [9:18] component fails, we don't really care. [9:20] The large cluster continues to operate [9:23] and things move forward. [9:25] Only the impacted machine and jobs on it [9:27] are failed, everything else keeps [9:28] running and those jobs move somewhere [9:30] else. The impact is generally pretty [9:31] isolated. [9:33] But with large-scale training that [9:34] changed pretty fundamentally. [9:36] Because it's not just a single [9:37] In fact, it's many systems that are [9:39] interconnected. All of those things are [9:41] interdependent on each other. All of [9:42] those thousands of cables across [9:44] thousands of racks, any failure at any [9:46] of those, if you don't optimize your [9:47] system, can take that entire training [9:49] job down. [9:50] Now, to accommodate this, we had to [9:52] redesign our software and think of these [9:53] systems not as individual servers, but [9:55] as a large cluster and a large [9:57] collection of machines with complex [9:59] interdependencies. [10:01] The model for this starts to look a lot [10:02] more like a network topology instead of [10:04] a typical distributed systems topology. [10:06] But why does that really matter? Why is [10:08] it so important that that job keeps [10:09] running? Well, fortunately, Alibaba just [10:12] released a paper about a month ago on [10:13] their train mover solution for this, and [10:16] it had some really excellent data points [10:17] in it. In fact, I'd encourage you all to [10:18] go read this. [10:20] And what they found is as a job scales [10:22] from hundreds to thousands to even tens [10:24] of thousands of GPUs, those individual [10:26] failures compound and increase the [10:28] interruption rate. For some perspective [10:30] on this, GPT-3 ran on about 1,000 GPUs [10:33] for about a month. [10:34] Llama 3 at Meta ran on about 16,000 GPUs [10:37] for a little over a month and a half. As [10:39] each new model comes along, it generally [10:41] requires more GPUs and longer run times. [10:44] Today's frontier models are running on [10:46] something like 100,000 GPUs, and that's [10:48] where these compounding really effects [10:50] really add up. Because at 1,000 GPUs, [10:52] you expect to see a failure once every [10:54] eight or so hours. At about 16,000 GPUs, [10:57] that drops down to about 2 and 1/2 [10:59] hours. As you start to approach hundreds [11:01] of thousands of GPUs, you would expect [11:02] to see a failure every eight minutes. If [11:05] that job fails every eight minutes and [11:06] doesn't restart immediately, you're dead [11:08] in the water. You're not doing anything. [11:10] And their data shows this pretty [11:11] clearly. In fact, as that job scales up, [11:13] you start to lose the effective training [11:15] rate of the system. [11:17] And you can in effect end up with idle [11:19] GPUs on the order of hundreds to [11:21] thousands to even tens of thousands of [11:22] GPUs. [11:24] As you approach 100,000 GPUs without [11:26] optimizations, without [11:28] changes to how you restart, without [11:29] making this better, you can expect to [11:30] have over 30% of your capacity idle at [11:33] any given time. [11:34] In a neighborhood of 40,000 GPUs sitting [11:37] and doing nothing. [11:38] If you just do the rough math on the [11:39] assumed CAPEX of that, it's like close [11:41] to a quarter billion dollars doing [11:42] nothing. [11:44] If the dollars weren't enough, I told [11:46] you in the beginning this is a race. [11:47] That means that we're falling behind. [11:49] Every time that job starts is time lost. [11:51] We had to improve this. We had to get a [11:53] lot better at it. So, we went back and [11:55] we started thinking about this from the [11:56] very basic level. [11:58] We had to reduce the frequency of [11:59] failures. We had to reduce the time it [12:00] takes to recover from a failed job. We [12:02] had to rethink everything and all those [12:04] component level failures from memory to [12:06] CPU to GPU to control to IO and so on. [12:10] And we work closely with our partners to [12:11] stress test the software and firmware [12:14] and all the components involved in this. [12:15] And we work closely with them to improve [12:17] their diagnostics along the way. [12:19] But we found that even our basic things [12:21] like our own repair methodology was [12:22] interrupting jobs as well. Because that [12:24] same mindset of hey, if we detect a [12:26] failure, we need to get it out of the [12:27] fleet as soon as possible. Well, that in [12:29] and of in of itself would actually [12:31] interrupt a job. [12:32] So, what do you do? How do you reduce [12:34] failures when they're hardware and real [12:35] and causing problems? [12:37] Well, and I think this is an area where [12:39] we have a unique advantage. And that's [12:41] that we control our own destiny from the [12:42] silicon to the server to the model to [12:45] the training software and everything in [12:47] between. And what this means is that we [12:49] can do something that few others can. [12:51] Because we can see the issues that occur [12:53] at the lowest level of the system and we [12:55] can see when it's degraded or when it's [12:56] impacted. We can also see from the job [12:58] level when it's been interrupted and [13:00] it's having problems. And we can marry [13:01] these things together so that we can [13:02] understand that when a system is having [13:05] issues and if that affects the job, then [13:07] we want to do something about it. But if [13:09] the server is having an issue but it is [13:10] not affecting the job, we can just leave [13:12] it in the fleet and let it continue to [13:13] run until something else naturally [13:15] occurs like a cable failure and takes [13:17] out the cluster. At that point, we can [13:19] go in, we can identify those broken [13:21] systems, we can swap them out with new [13:23] hardware, and the job can keep going. Or [13:25] I can repair them. Whatever we need to [13:26] do. [13:27] Now, the collective result of all these [13:28] optimizations from the software and the [13:30] firmware [13:31] through improved diagnostic in [13:33] partnership with an NVIDIA to optimizing [13:36] our operational systems to doing this [13:38] across many generations and building on [13:39] that compounding knowledge from A100 to [13:42] H100 to GB200 to GB300 and everything in [13:45] between, it all adds up. In fact, what [13:48] we're finding is that we've been able to [13:49] reduce the amount of failures and job [13:50] interruptions by something on the order [13:53] of like, I don't know, 95%. And yes, [13:56] there's fireworks because it's almost [13:57] July 4th, so there you go. [14:00] This is pretty awesome stuff. [14:02] Okay. Now, as we move up the stack from [14:04] the compute and into the software, you [14:06] heard from a lot of folks today. You [14:07] heard from Serupa and Gaurav and others [14:09] who mentioned that we're moving from a [14:10] world where software is written by [14:12] humans and for humans to one that's [14:14] written by agents for agents. [14:16] And most of the code in our systems [14:17] today is actually being written by AI. [14:19] And I think all of us have experienced [14:21] this massive increase in developer [14:22] productivity and output as a result. I [14:24] think Joe earlier showed that graph [14:26] that, you know, was telling us that [14:26] there's about an 8x improvement in [14:28] developer output. That's something that [14:30] looks like it can get to us 10x and [14:31] maybe in a 100x over time. It's pretty [14:33] amazing. [14:34] But interestingly, [14:35] and I think actually counter to the [14:37] popular narrative on this, we're [14:38] actually finding that each of those [14:41] changes inside of software is actually [14:44] safer as a result of AI. [14:46] Not more dangerous and not causing more [14:47] incidents. [14:49] In fact, if we look at this over time, [14:50] in the course of 2025, the amount of [14:52] incidents caused by changes in our code [14:55] and software was relatively flat. [14:57] Meaning for any given, you know, change [14:58] that we made, the amount of incidents [15:00] that we had was pretty constant over [15:01] time. But that started to change [15:03] dramatically, especially in October and [15:04] November and the beginning of the year [15:06] as we introduced more and more AI. And [15:08] what we found is that every given change [15:09] became dramatically safer. Over the [15:11] course of just 1 year, we're seeing an [15:13] over 50% reduction in the amount of [15:14] incidents per any given change. [15:17] There's a caveat here, and there always [15:18] is. [15:19] While this is on the order of any given [15:21] change, the amount of diffs and changes [15:23] has gone up dramatically. So, even [15:25] though safety is increasing, the [15:26] frequency has increased at a far faster [15:28] rate, which means incidents are up [15:30] overall. [15:31] And this is not just a meta-specific [15:33] issue. In fact, GitHub had a post about [15:35] a month ago about how they had planned [15:37] for a 10x increase in capacity as a [15:39] result of AI-driven development, and [15:41] they are finding that they need 30x [15:43] increases in capacity to keep up because [15:45] the volume of code and the volume of [15:47] changes is that great. [15:49] But, it's not just the volume of changes [15:50] making this more difficult. It's the [15:52] time that it takes to deal with any one [15:53] of these given incidents. And we can [15:55] track the time our developers spend on [15:57] what we would call toil. And that's [15:59] dealing with the operational results, [16:00] whether that's incidents themselves or [16:02] the follow-ups from this or the code [16:04] that's necessary to fix it. And what [16:06] we're finding is that over the course of [16:07] the time that every one of these [16:09] incidents is actually taking humans more [16:11] time to diagnose, to resolve, and [16:13] mitigate. In fact, year over year, it's [16:14] increased over 120%. [16:17] So, you're spending an hour before, [16:19] spending an hour and 20 minutes, 2 [16:20] hours, 3 hours now at this point on all [16:22] the same incidents. And that's why the [16:24] work you heard of them from David in [16:26] terms of agent guardrails, the work from [16:28] Gaurav on agentic operations, the work [16:31] from the folks at Microsoft and [16:33] increasing their debugging is actually [16:34] so important because we need to drive [16:36] that down and reduce the cost of this. [16:39] All right. So, with that, I've given you [16:40] a pretty good summary, hopefully, of [16:42] data centers, of compute, of the [16:44] software systems involved, and where [16:46] those investments are going across our [16:47] industry. I started this talk also [16:49] talking about Codex and [16:52] cloud code and the kinds of things that [16:53] we're seeing in reality. And at Meta, [16:55] you can actually see that reality in our [16:56] products as well, whether that's Threads [16:59] or WhatsApp or Instagram, you're [17:00] starting to see all that agentic [17:01] behavior and all those new AI [17:03] capabilities. What I'm particularly [17:05] proud of is Ray-Ban Metas, and a new [17:08] program that we just launched that [17:09] Sheryl mentioned earlier before, and [17:11] that's to give Ray-Ban Metas for free to [17:13] all the vision-impaired vets in the the [17:15] States. [17:16] Now, these glasses can take advantage of [17:18] the cameras and the microphones and they [17:20] can effectively see the world around [17:22] them. They can capture images and audio [17:24] and video of the what these people are [17:26] looking at. And they can send that data [17:28] back to our inference systems, which [17:30] goes back to our servers, which [17:31] eventually goes back to our models and [17:33] it can process that data and then it can [17:35] give those people a clear description of [17:37] what they're seeing and what they're [17:38] interacting with and in a sense it gives [17:41] them an ability to see again. [17:43] That is pretty awesome and that is [17:44] something is only possible when you have [17:46] that control of your destiny from [17:47] silicon [17:48] to the products and everything in [17:50] between. [17:51] And that's not just technically exciting [17:52] to me. I think that's really meaningful [17:54] and important to people in the world. [17:57] Now, before I close this out, I wanted [17:59] to kind of come back to the start of [18:00] this. I wanted to come back to that [18:01] picture of Neil Armstrong on the moon [18:04] because I used, you know, an example in [18:05] comparing this to the Apollo mission and [18:07] I explained that we're spending, you [18:09] know, almost twice the amount of money [18:10] that we spent on the Apollo mission in [18:12] 2026 and I talked through all the things [18:14] that we're getting from this, you know, [18:15] from one single prototype of a 10 to [18:17] hundreds of megawatts in this. [18:19] From a single hybrid cloud provider to [18:21] tens of billions of dollars invested in [18:23] this. From 100 megawatts of capacity to [18:27] multi-gigawatt facilities all around the [18:28] world. From primarily H100 and A100 GPUs [18:32] to AMD and GPU or sorry, GB200 and GB300 [18:35] and significant improvements in job [18:37] reliability and things across the board. [18:39] From in basic inline code completion to [18:41] cloud code and code X and really [18:43] identified [18:44] software development life cycles and [18:46] ultimately to products like Ray-Ban Meta [18:48] that gives people the power to see [18:51] again. [18:52] Now, all told last year was pretty [18:53] incredible and the reality, the results, [18:55] the impact is clearly coming into [18:57] picture. [18:58] And going back a bit further, last year [19:00] I ended my at scale talk just asking [19:02] where we were on that sort of overall AI [19:05] timeline. And I said, you know, what? [19:07] We're not at the middle of this curve or [19:09] the end of this curve. We're just at the [19:10] beginning. It's the next 20 years where [19:13] we're going to see this really massive [19:14] change in our industry, in our [19:16] technology, and in the lives of all the [19:18] people that we interact with. And you [19:20] would think that given how much we've [19:21] progressed in the last year, and how [19:23] much faster that was than I think all of [19:24] us in this room thought, that we would [19:26] have moved a much further on that line. [19:29] But I think the reality is, if anything, [19:31] I was probably being a little bit too [19:32] optimistic. In fact, we are not at our [19:34] moon moment. We're far from it. I think [19:37] the best analogy, in fact, is we're at [19:38] the beginning of the space race. We're [19:39] at our Sputnik moment. [19:41] All right, this is just the very start. [19:43] And what that means [19:45] is the coming weeks and months and [19:46] years, is we're going to see that [19:48] investment really take off. And we're [19:49] going to see the results of that [19:51] increase dramatically over time. In [19:53] 2027, the estimate for CapEx is close to [19:56] $1 trillion. That's That's one followed [20:00] by 12 zeros. [20:02] Massive, massive amounts of money. [20:04] Almost impossible to think about. [20:07] In fact, just for one more perspective [20:08] on this, and just to give you some [20:10] context to the size and scale of what a [20:11] trillion dollars is, [20:13] that's the combined net worth of exactly [20:16] one person. [20:20] Jokes aside, I couldn't handle it. [20:22] >> [laughter] [20:23] >> But in reality, it's the size of the [20:25] entire budget of the entire United [20:27] States military, all branches and all 1 [20:30] million service members. That's the [20:31] amount of money that we're about talking [20:33] about here. [20:34] And well, throughout the course of [20:35] today, you heard about the progress in, [20:37] you know, fault detection and [20:38] remediation. You heard about agentic [20:40] operations. You heard about building [20:42] infrastructure for agents. [20:44] You heard about all the real-world [20:45] experiences and some of the challenges [20:47] that people have dealt with with outages [20:49] and incidents and problems that have [20:50] been caused. [20:51] And then you think about that and the [20:52] progress we've made and where we need to [20:54] go, and I think back to Sputnik. And the [20:56] distance between Sputnik and the ground [20:59] is far less than Sputnik into the moon. [21:01] And that's the work that we still have [21:02] in front of us. Because when I think [21:03] about training and fault tolerance, yes, [21:06] we've done a ton of work, and yes, we've [21:07] done a ton of progress, but guess what? [21:09] Those failures still interrupt those [21:10] jobs. We have to get much better at [21:12] fault tolerance in our training [21:13] algorithms. We have to get much better [21:15] at our detection and remediation [21:16] systems. We have to get much better at [21:18] our gigantic operations, as well. [21:20] And all of that changes with every new [21:22] GPU platform that comes out. [21:24] We In many ways, we've cracked the [21:25] software coding problem, but as we all [21:27] know, code is only part of our job. [21:29] There's deployments, there's operation, [21:31] there's mitigation, there's remediation, [21:33] there's long-tail migrations. I was just [21:35] talking to somebody about at lunch. [21:36] These are all parts of our job that [21:38] agents have still yet to solve and still [21:40] have a long way to go to help us. [21:42] I spoke about the challenges of hardware [21:44] constraints, and I think many of you who [21:45] have been involved in this for the last [21:47] couple years can talk about how GPUs [21:48] have been holding us back and how we [21:50] haven't been able to find, you know, all [21:51] the hardware that we need. [21:53] And far from helping us, that $1 [21:54] trillion investment is actually going to [21:56] make this even worse. We're going to [21:57] move from a place where we have just GPU [21:59] constraints to the everything [22:01] constraint, from CPU to memory to disk, [22:03] and we're already seeing that today, and [22:05] it's only going to increase as we go [22:06] further. And to add to that, that means [22:08] we're going to have to keep hardware in [22:09] our fleet longer, which means we're [22:10] going to have increased failure rates. [22:12] We're going to have to take capacity [22:13] from wherever we can get it. We're going [22:14] to have increased heterogeneity, which [22:16] means more complexity and more problems [22:17] to solve. [22:19] Pretty interesting stuff, and I'm super [22:21] excited about it. [22:22] Now, in closing, and on a bit more of a [22:23] more personal note, I've been in this [22:25] industry for something like almost 30 [22:27] years at this point. About half of that [22:29] time has been at Meta. [22:30] From the early dot-com boom to early [22:32] internet to the advent of mobile and [22:35] streaming video, both on your phone and [22:37] on your desktop and long-form and [22:39] short-form, a lot of things have changed [22:41] dramatically, but never, never in that [22:43] 30 years have we moved more faster than [22:46] we are right now. [22:47] And the crazy part is we're actually [22:48] doing a pretty darn good job of this. [22:50] Things are actually generally to to [22:51] work. And I was trying to understand [22:53] what is it that makes that happen. And [22:55] this is where it comes back to those [22:56] moon missions, because I think [22:58] there's a lot in common. [22:59] The first is we have control of our [23:01] destiny. The second is we allow people [23:03] to move very fast and reduce the amount [23:05] of red tape that gets in the way. And [23:07] the third one is the people themselves. [23:09] And to go in a bit more detail of what I [23:11] mean by that, on the first topic, it's [23:13] about giving people the autonomy and the [23:15] speed and reducing the red tape to allow [23:16] them to move really fast. On the second [23:18] topic, it's the ability to give them [23:21] the the products and capabilities to [23:24] really move things quickly and go back [23:26] to the point where you say, "Look, the [23:28] only thing that is up for debate [23:30] is something that is not bounded by the [23:32] laws of physics." And if you can get to [23:33] that point, we can move very quickly. [23:36] And finally and most importantly, like I [23:37] said, it's the people. It's all the [23:39] people in this room. It's all of you [23:41] collectively coming together and [23:42] bringing your experiences and your [23:43] talent together to move the needle [23:45] forward. It's these ingredients and [23:47] these investments and the technology [23:49] that's going to come together [23:50] to allow us to make more meaningful and [23:52] valuable products in the world, to [23:54] increase the capabilities of people [23:55] around the world. And that's why it's [23:57] such an exciting time to be right where [23:59] we are today in this industry. And [24:01] that's why it's so exciting to work on [24:02] systems and reliability. Cuz this is [24:05] right at the heart of some of the [24:06] hardest problems in the largest [24:08] infrastructure build-out in human [24:10] history. [24:11] With that, [24:12] thank you.