Meta Adapts Infrastructure and Cloud Stack to Scale AI Agents

Meta调整基础设施与云技术栈以扩展AI Agent规模 Surupa Biswas指出，Agentic AI需要底层基础设施实现指数级增长并采用混合云架构，以处理多步骤工作负载。 Surupa Biswas概述了Meta为满足Agentic AI巨大的算力需求而做出的战略转变：从单纯依赖自建专有数据中心，转向利用公有云、新型云（neo-clouds）以及全球硬件“Rings”。她详细介绍了Meta如何调整其企业级技术栈——引入Meta Kubernetes Service (MKS)、DSQLite、Nest和Feather VM等工具——以支持开源标准和开发人员的快速实验。此外，她强调了安全控制平面与智能体工作流的集成，并指出随着AI从简单的编程助手演进为多步骤执行者，指数级增长的AI需求将持续驱动重大的基础设施创新。 Agentic AI工具正在从理论潜力转化为实际价值，凭借跨会话的持久记忆与多步骤执行能力，推动持久用例的发展，超越了简单的单次会话搜索交互。 (0:01) 在Meta核心基础设施团队每日提交的代码变更中，无论是全新系统还是已有数十年历史的旧系统，目前已有超过90%包含AI编写的代码。 (0:01) 为突破GPU、DRAM、晶圆代工和电力供应等物理供应链的严重瓶颈，Meta从纯自建定制数据中心转向公有云和新型云提供商，迅速将规模扩展至数百兆瓦。 (4:40) Meta引入了名为“Rings”的地理抽象概念，通过高带宽网络将临近的数据中心区域互联，解决了在分散的云节点之间重复运行微服务带来的运维开销。 (10:12) 为确保智能体能够安全地调用内部基础设施，Meta推出了用于工具发现的Meta CLI和Meta IAC，并由持续的健康检查及自动化回滚机制提供安全保障。 (12:04)

Surupa Biswas, Vice President of Infrastructure at Meta, argues that the transition to agentic AI has transformed infrastructure into a physical supply race, compelling operators to merge heterogeneous clouds, open-source standards, and automated safety systems to handle compounding compute demand. Overcoming Physical Bottlenecks Through Hybrid Cloud Supply (4:40) Biswas characterizes the modern AI competition as fundamentally tied to physical supply chain limits across data center power, fiber, fabrication capacity, memory, and storage. After relying exclusively on proprietary data centers since 2011, Meta pivoted to public clouds and neo-clouds, scaling to hundreds of megawatts of production capacity within a single year. To sustain longer-term energy requirements, Meta announced plans to secure up to 6.6 gigawatts of nuclear power in the United States by 2035, while industry peers explore concepts like orbital data centers. Meta's compute orchestrator, Twine, now operates across a heterogeneous hardware footprint including NVIDIA GPUs (H100, B200, GB200, GB300), AMD's MI355X, Meta's proprietary MTIA silicon, and incoming Google TPUs. To manage data movement across distributed locations, Meta deployed a "Storage Hub" that enables external compute to query on-premises data directly, paired with a lightweight "Storage Light" stack to run workloads ranging from batch inference to large-scale reinforcement learning. Regional Networking Abstractions and Container Standardization (10:12) Onboarding capacity across fragmented cloud regions introduced administrative bottlenecks, as duplicating backend microservice quorums threatened to inflate operational overhead. To resolve this, Meta introduced "Rings," a geographic networking abstraction that links proximate data centers—whether proprietary, public cloud, or neo-cloud—over ultra-high-bandwidth connections. This design lets front-end systems communicate with backend dependencies across separate facilities without performance penalties, with two global Rings operational and more planned. To prevent internal infrastructure from slowing down experimentation, Meta aligned with open-source tooling by building the Meta Kubernetes Service (MKS) in under a quarter. Rather than deploying an isolated environment, MKS was built directly atop Twine, inheriting its machine allocation, capacity management, and job orchestration. This provides a CNCF-compliant environment that lets developers run standard containerized workflows natively while shortening the transition from prototyping to production from months to weeks. Deploying Directory Aggregation and Agent Safety Guardrails (12:04) With over 90% of daily code changes in Meta's core infrastructure team now containing AI-authored code, internal systems have been re-engineered for multi-step agent execution. To help autonomous agents navigate internal tooling without damaging infrastructure, Meta introduced the Meta CLI as a unified directory service, which saw over 500 internal systems integrate within three months. Meta Infrastructure as Code (IAC) was also adopted to standardize control-plane interactions. Safety is maintained by adapting existing reliability and security frameworks rather than isolating AI operations. Autonomous agents deploy small canary releases and continuously evaluate hardened health checks, with automated rollbacks triggering if anomalies emerge. When agents fail, execution trajectories are preserved, prompting automated ticket creation, code diff generation, and integration into continuous evaluation loops to improve autonomous reliability over time. Modernizing Data Architectures and Execution Environments (19:21) Meta updated its data tier by disaggregating storage and compute for Postgres, enabling autonomous agents to provision databases in seconds. The platform introduced speculative execution for databases, allowing engineers to test modifications and merge or discard them similarly to source code control. In parallel, Meta engineered DSQLite, a lightweight database providing full SQLite semantics backed by Meta's distributed key-value store, ZippyDB. For internal application delivery, Meta created Nest, a Vercel-like Function-as-a-Service platform running on DSQLite that powers over 22,000 internal applications, including the MetaMate assistant. Meta AI's service layer transitioned away from static C++ linking toward gRPC and HTTP endpoints via Project Hub. To secure autonomous workflows requiring persistent state and privacy, the team implemented Feather VM, a lightweight virtual machine architecture built to deliver isolated execution environments. Biswas frames this infrastructure transformation as an "electrification moment," noting that announced capital expenditures among the top five hyperscalers already surpass $700 billion. As AI advances from task-specific coding assistants to autonomous, persistent agents, Meta's strategy focuses on balancing open-source CNCF compatibility with specialized internal abstractions to keep foundational infrastructure ahead of compounding demand.

Meta基础架构副总裁Surupa Biswas在@Scale大会上指出，AI行业正从理论探索迈入产生实际价值的阶段，Meta正在通过打破传统自建边界、重构软件抽象层以及融入开源标准，推动底层系统实现全面变革。 算力供给瓶颈与混合云扩展 (4:40) Biswas强调，当前的AI竞争本质上是一场物理供应链竞赛。全球范围内，电力、光纤网络、晶圆厂产能、GPU、CPU、DRAM、高带宽内存（HBM）以及机械硬盘等关键资源均面临严重短缺。为应对长远能源缺口，Meta公布了到2035年在美国获取最高达6.6吉瓦（GW）核电供应的计划，并指出同业正在探讨轨道数据中心等极端设计。 面对算力制约，Meta做出了关键运营调整：结束自2011年以来完全依赖自建自营数据中心的模式，开始大举引入公有云和“新云（neo-clouds）”服务商。Meta在一年内将这一模式从初期试验扩展到在全球承载数百兆瓦（MW）的核心生产负载。Meta原生的集群编排平台Twine目前已支持包括NVIDIA（H100、B200、GB200、GB300）、AMD MI355X、Meta自研MTIA芯片以及即将引入的Google TPU等异构硬件。通过开发直接跨区查询链下数据的“Storage Hub”与轻量化“Storage Light”技术，混合云部署的工作负载已由早期的合成数据生成和离线批处理推理，拓展至大规模强化学习和实时模型推理。 Rings抽象化解跨区域运维开销 (10:12) 在快速引入外部多云区域的过程中，数据中心地域扩展进入了Meta Superintelligence Lab与产品团队获取算力的关键路径。然而，在碎片化的多云节点中重复部署后端微服务法定副本（quorums），带来了显著的运维开销与延迟隐患。 为此，Meta工程团队提出了名为“Rings”的地理抽象架构。该架构利用超高带宽网络将地理位置邻近的Meta自建数据中心、公有云及新云区域连接起来，使得某一区域的前端层可以无损耗地访问同一Ring内另一云区域的后端依赖。目前Meta已在全球范围上线两个Rings架构，并持续推进其他区域的网络部署，以消除多云环境下的跨区域调用壁垒。 智能体工程化：目录聚合与安全护栏 (12:04) 在企业内部构建AI应用时，自主智能体在执行多步骤工作流过程中常面临工具链繁琐和操作风险。Meta并未另起炉灶建立平行的AI安全技术栈，而是将现有的安全、隐私与可靠性框架改造为智能体的控制中枢。在工具发现层面，Meta推出了统一命令行目录服务Meta CLI，在3个月内接入了超过500个内部系统；在资源编排层面，通过Meta IAC（基础架构即代码）规范了智能体对控制平面的配置操作。 为防范系统性故障，Meta实施了严格的安全防御策略：智能体仅能开展最小化金丝雀发布，并持续监测服务健康指标，一旦异常便触发系统自动回滚，关键核心系统被严格禁止执行破坏性指令。同时，Meta对智能体的执行轨迹展开记录，若任务执行失败，系统会自动提交故障工单、生成代码补丁差异（diff），并将执行记录反馈至持续评估闭环中，以自动化手段提升智能体工程的可靠性。 拥抱开源标准与数据系统现代化 (17:38) 为消除大型底层架构对原型快速实验的阻碍，Meta转向深度拥抱开源与容器化标准。工程团队在单季度内依托成熟的Twine集群管理底座，快速构建出完全符合CNCF标准的Meta Kubernetes Service（MKS），实现原生运行开源容器镜像，将开发成果从早期原型推向生产规模的耗时从数月缩短至数天或数周。 在数据架构层面，Meta实现了Postgres的存算分离，允许智能体在数秒内拉起新数据库，并引入了类似代码版本管理的“投机执行”机制，便于开发者分支合并或废弃数据操作。针对轻量场景，Meta构建了兼容SQLite语法的DSQLite，底层直接由ZippyDB分布式键值存储提供支撑。此外，Meta还推出了基于函数即服务（FaaS）的类Vercel应用平台Nest，目前已承载包括MetaMate在内的2.2万个内部应用；通过Project Hub将底层C++服务转化为HTTP和gRPC接口；并依托轻量虚拟化技术Feather VM为智能体提供安全隔离的执行空间与跨会话持久上下文。 Biswas将当前AI与基础设施的发展轨迹比作历史上的“铁路时代”或“电气化时代”，强调AI带来的指数级价值创造必然拉动底层基础架构工程的指数级增长。然而，面对超过7000亿美元的行业资本开支、极度紧缩的供应链以及多云异构并存的现状，如何在保持极速迭代的同时长期维持系统的安全隔离与高可用性，仍是整个行业面临的持续考验。

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 7 月 2 日发布，记录了 Meta 核心基础设施工程副总裁 Surupa Biswas 在 @Scale 大会上的开场主题演讲。页面介绍提到，Surupa 负责驱动 Meta 旗下应用家族与 Meta AI 的内部云平台，拥有近二十年在编程语言、编译器和分布式系统领域的深厚积累。 ## TL;DR Surupa Biswas 宣布行业已告别 AI 的纯理论推演，正式迈向由自主智能体（Agent）驱动的真实价值落地期，Meta 内部核心基建中已有超过 90% 的日常代码变更由 AI 辅助完成。面对严峻的供应链瓶颈与算力饥渴，Meta 彻底重构底层技术栈，通过 Twine 共享调度层、高带宽直连网络环（Rings）以及 Storage Hub/Storage Light，实现了跨自建数据中心、公有云与新型 AI 云的大规模混合异构算力调度。针对企业级智能体落地，团队构建了 Meta CLI 统一目录、Meta IAC 安全治理通道与 Feather VM 强隔离沙箱，并全面拥抱开源生态、Kubernetes 及轻量化数据库，推动软硬件全栈协同设计以迎接 AI 带来的“电气化时刻”。 ## Atmosphere & style 整场演讲充满前沿架构师的自信与深沉的技术洞察力，Surupa 重返故乡 Bellevue 的开场带来了一种兼具温情与厚重历史感的叙事基调。随着她将话题从宏观行业演进逐步切入硬核的软硬件协同设计、跨云异构调度与微秒级工程优化，语调转向严谨笃定且富有感染力。听众能真切感受到一家顶级科技巨头在面对算力极限与技术范式剧变时，既有推翻既有体系、拥抱开源生态的谦逊清醒，又怀揣着投身基建“全面电气化时代”的豪情与敬畏。 ## Bulletin - [00:01](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1s) Meta 核心基础设施团队中超 90% 的日常代码提交已包含 AI 生成或辅助代码，自主智能体正在重塑拥有数十年历史的底层复杂分布式系统代码库。 - [07:29](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=449s) 为打破自建算力瓶颈，Meta 依托 Twine 共享调度层与 Storage Hub/Light 存储抽象，在多家公有云及 Neo-clouds 上承载了数百兆瓦核心生产级 AI 负载，并统一纳管英伟达、AMD、自研 MTIA 及 Google TPU 异构芯片。 - [10:26](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=626s) Meta 推出高带宽“网络环”（Rings）架构，将临近的自建数据中心与公有云区域高速打通，成功消除跨区法定副本组部署开销，保障前端层与后端层跨云跨区低延迟协同。 - [16:38](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=998s) 面对敏捷实验诉求，Meta 基础设施主动拥抱开源生态，几周内基于 Twine 落地生产级 Kubernetes 服务（MKS），推出支持存算分离与推测执行的 Postgres/DSQLite，并部署轻量 Feather VM 打造安全隔离沙箱。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1s) 行业已从理论探索正式步入 AI 创造实际商业价值的落地期，自主智能体具备跨会话持久记忆，正引领类似移动互联网爆发的范式重构。 - [04:22](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=262s) 基础设施演进经历三部曲：早期为 AI 堆砌算力的“Building for AI”、当前用 AI 重塑工程体系的“Building with AI”，以及探索海量智能体自主演化的“Exploration at Scale”。 - [05:43](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=343s) 电力、长途光纤、晶圆产能及内存储存硬件全线告急，倒逼软硬件协同设计（Co-design），促成 Meta 6.6 GW 核电规划以及太空轨道数据中心等物理层前沿探索。 - [07:29](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=449s) Meta 打破 15 年纯自建传统，构建覆盖公有云和 Neo-clouds 的大规模混合云体系，通过统一 Twine 共享调度层实现英伟达、AMD、MTIA 与 TPU 等异构算力的即插即用。 - [07:29](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=449s) 创新双轨跨云存储架构，借助 Storage Hub 直连本地存储与公有云原生轻量化的 Storage Light，平稳支撑从合成数据生成到超大规模实时模型服务等严苛任务。 - [10:26](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=626s) 创新的抽象网络环（Rings）架构通过超大带宽直连物理临近的多云与自建数据中心，将原本必须同区部署的前后端服务解耦，彻底打破跨区新建与容灾扩展瓶颈。 - [12:01](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=721s) 推出 Meta CLI 作为企业工具目录中枢，并依托 Meta IAC（基础设施即代码）为控制平面操作设立安全护栏，3 个月内打通超 500 个内部系统。 - [13:54](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=834s) 放弃另起炉灶的“平行安全栈”，全盘复用成熟基建，通过物理级硬约束、最小粒度金丝雀灰度及自动回滚机制化解智能体运行风险。 - [13:54](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=834s) 建立全量追踪与自愈反馈闭环（Eval Loop），智能体执行受阻时自动提单、生成修复补丁并跑评测，以极少的人工干预驱动系统全自动进化。 - [16:38](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=998s) 面对前沿敏捷探索，大型基建团队学会“靠边站”，基于 Twine 底座在数周内光速上线标准 CNCF 兼容的 Meta Kubernetes Service（MKS），大幅压缩原型到量产交付的周期。 - [19:21](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1161s) 推进数据基础设施现代化：实现 Postgres 存算分离并引入类 Git 的数据推测执行版本控制，同时孵化轻量免运维的 DSQLite。 - [20:12](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1212s) 革新传统重型 C++ 静态链接通信机制，借助现代化 gRPC、HTTP 端点（Project Hub）及类 Vercel 应用平台 Nest，数月内催生超 22,000 个内部微服务。 - [21:48](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1308s) 自主研发轻量化 Feather VM 沙箱与跨会话持久化存储卷，在确保绝对安全隔离与数据隐私的前提下，为端侧及企业智能体提供长期记忆能力。 - [22:26](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1346s) 构建自底层异构算力供应、硬件抽象层、现代云原生与私有软件服务层至顶层 AI 负载的四层技术架构，实现芯片到应用的全栈协同设计。 - [23:20](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1400s) AI 即将迈入创造指数级价值的阶段，堪比铁路大提速与全面电气化时代，其底层本质依赖系统工程团队在分布式计算与物理层基础基建上的持久大规模扩张。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1s) 回归 Bellevue：从理论探索到 AI 真实价值的解锁 Surupa Biswas 登台向现场观众致意，并感谢 Casey 的开场介绍。本次 @Scale 大会首次在华盛顿州 Bellevue 举行，对 Surupa 而言意义非凡——Bellevue 曾是她生活多年的故乡，如今能与全球顶尖的基础设施工程师汇聚一堂，共同探讨这场前所未有的技术飞跃与行业增长，令人倍感亲切与振奋。当今底层系统所面临的挑战，在体量、复杂度与规模上均创下了历史新高；而 @Scale 大会的魅力正是在于，来自不同公司的系统架构师们能够跨越组织边界，坦诚交换心得，携手攻克此前无人解决过的难题。 回顾过去的分享历程，Surupa 梳理了一条清晰的技术演进脉络：两年前，她重点探讨了“AI 驱动系统革新，同时系统革新反哺 AI 算法”的双向互动；一年前，她与 Peter Hoose 深入拆解了这种动态交互如何自底向上颠覆整个基础设施技术栈，彼时 Meta 团队聚焦的核心指标是快速上市时间（Time to Market）、架构可变通性（Optionality）以及创新速度。当时极度重视“可变通性”，是因为大家预见到了 AI 将释放巨大价值，但尚不确定具体会催生出怎样的产品形态，也不清楚随之而来的底层系统瓶颈究竟会落在何处。而站在当下的时间节点上，行业已彻底告别理论推演，正式步入 AI 创造真金白银价值的落地期。 过去 12 个月中，整个行业最具决定性的分水岭事件莫过于“OpenClaw（或类似自主智能体）时刻”的爆发。在此之前，大众对 AI 的使用方式主要停留在“搜索式”的单次会话（single-session），即一问一答的交互模式；而智能体（Agents）的出现，彻底释放了 AI 替人类端到端完成复杂、高价值工作的潜力。特别是智能体跨会话的持久化记忆能力（persistent memory），赋予了它前所未有的执行力。 在工程研发一线，这种变革尤为剧烈： - **复杂系统代码重塑**：智能体早已超越编写“从 0 到 1”简单新应用的原型阶段，甚至能在拥有数十年历史的底层复杂分布式系统代码库中，精准完成错综复杂的逻辑修改，且代码正确率极高。 - **Meta 内部采纳率**：在 Meta 的核心基础设施工程团队内部，日常提交的代码变更中，超过 90% 都包含由 AI 辅助或自主生成的代码。 - **万亿美元级资本开支**：AI 释放的价值正大规模反哺算力基建。全球前五大超大规模云服务商（Hyperscalers）公开披露的 CapEx（资本支出）累计已达 7000 亿美元，且 Surupa 认为这可能还是一个较为保守的估计。 Surupa 将当前的智能体爆发与当年的“移动互联网时刻”作类比：互联网底层技术虽然早已存在多年，但直到智能手机将网络装进每个人的口袋，海量此前未曾设想的应用场景才真正井喷。同理，Agentic Execution（智能体自主执行）正在重构 AI 的使用范式。例如她从 2021 年起便随身佩戴的 Meta Ray-Ban 智能眼镜，起初仅用于听音频和抓拍照片，而如今她可以直接通过眼镜向身边的 Agent 下达工作指令，让智能体在后台推进事务，自己则专注沉浸在现实生活与工作中。 --- ### [04:22](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=262s) 基础设施演进三部曲：从“为了 AI”到“以 AI 筑基” 面对智能体规模与能力边界的持续膨胀，基础设施构建者该何去何从？Surupa 将技术团队的演进路径划分为三个清晰的发展阶段： ```mermaid flowchart LR accTitle: AI 基础设施演进历程 accDescr: 展示从构建支持 AI 的基础设施，到用 AI 辅助构建系统，再到大规模探索的三个技术发展阶段 subgraph Phase1["第一阶段"] A["Building for AI<br/>（为 AI 构筑基建）"] end subgraph Phase2["第二阶段"] B["Building with AI<br/>（借 AI 之力共建）"] end subgraph Phase3["第三阶段"] C["Exploration at Scale<br/>（大规模探索与未知突破）"] end A --> B --> C ``` 1. **Building for AI（为 AI 构筑基建）**：早期的战略重心在于超前布局算力，解决支撑大模型训练与推理所需的严苛供应链、电力以及异构算力池问题。 2. **Building with AI（借 AI 之力共建）**：当下的实战重点，即把 AI 深度嵌入企业内部工作流与系统工程全生命周期中，利用 AI 优化并重塑基础设施本身。 3. **Exploration at Scale（大规模探索与未知突破）**：未来最为令人兴奋的疆界，探索由海量智能体自主演化带来的全新系统模式与前沿场景。 Surupa 表示，本场演讲将围绕这三大趋势深入展开，复盘其带来的工程扩展难题，并独家揭秘 Meta 内部应对这些挑战的系统架构解法。 --- ### [05:43](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=343s) 供应链极度紧缺下的物理层狂想与“软硬件协同设计” 如果说 AI 竞赛的本质是一场基础设施军备竞赛，那么紧绷到极限的供应链现状就是最现实的写照。当下无论工程师聚焦于物理架构的哪一个切面，几乎全方位面临严重的资源挤渴： - **基础设施要素全线告急**：数据中心电力接入指标、骨干长途光纤网络带宽、先进制程晶圆厂（Fab）代工产能； - **核心算力与存储硬件匮乏**：不仅 GPU 供不应求，在智能体高频执行逻辑驱动下，CPU 也迎来了创新的“高光时刻”；与此同时，不管是高带宽内存（HBM）、标准通用 DRAM，还是大容量机械硬盘，全球市面上的存量都捉襟见肘（正如 Micron 最新财报所反映的供需紧绷态势）。 这种严苛的供应链现实倒逼各大科技巨头跳出传统思维框架，掀起了一场前所未见的底层物理创新潮。例如，Meta 几个月前正式宣布，计划到 2035 年在美国本土解锁高达 6.6 吉瓦（GW）的核能电力供应；而谷歌与 SpaceX 等同行甚至已经将目光投向近地轨道，着手研发“轨道太空数据中心”，数据中心飞向太空已不再是科幻小说的桥段。 在这场物理层变革中，最核心的架构设计思想是**软硬件协同设计（Co-design）**。系统软件栈必须紧贴物理层基础设施的特性一同演进。为此，Meta 工程副总裁 Peter Hoose 将在当天的专题演讲中进一步详述 Meta 在数据中心物理设施与新型硬件层面的技术突破。 --- ### [07:29](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=449s) 拥抱混合云：Meta 跨公有云、自建与异构算力的落地实战 在自建数据中心（On-prem）领域，Meta 自 2011 年起便坚持自主设计并运营内部数据中心，深耕长达 15 年，承载了绝大多数生产级业务。然而在一年半以前，面对突飞猛进的 AI 算力饥渴，团队意识到：为了不惜一切代价获取可用算力，“必须彻底打破过去的壁垒，拥抱在公有云及新型 AI 云（Neo-clouds）上运行生产级工作负载”。 从一年前尝试性接入单一公有云开始，如今的 Meta 已经演进为在多家公有云上并行承载数百兆瓦（hundreds of megawatts）核心生产负载的庞大系统，构建起全球规模数一数二的混合云（Hybrid Cloud）体系。但这绝非开箱即用的简单迁移，为了抹平异构环境的底层差异，Meta 展开了全栈式的系统重构： ```mermaid flowchart TD accTitle: Meta 异构混合云与存储抽象架构 accDescr: 展示 Meta 生产工作负载如何在统一调度层与存储抽象层上支撑多种异构算力和跨云运行 subgraph Workloads["上层生产级 AI 负载"] W1["合成数据生成"] W2["离线/批处理推理"] W3["大规模 RL 强化学习"] W4["实时大规模模型 Serving"] end subgraph Orchestration["统一主机管理与调度层"] TW["Twine 共享抽象层 (Twine Shared Layer)"] end subgraph Silicon["底层异构计算阵列"] direction LR S1["NVIDIA 系列<br/>(H100 / B200 / GB200 / GB300)"] S2["Meta 自研 MTIA"] S3["AMD Instinct<br/>(MI355X)"] S4["Google Cloud TPU"] end subgraph StorageLayer["跨环境存储系统"] direction LR SH["Storage Hub<br/>(跨云访问 Meta 自建主存储)"] SL["Storage Light<br/>(公有云原生低依赖轻量存储)"] end Workloads --> TW TW --> Silicon Workloads -.-> StorageLayer ``` - **统一计算管理与调度**：依托 Meta 内部的核心调度系统构建了 **Twine Shared Layer（Twine 共享调度抽象层）**。该层不仅无缝部署在 Meta 自建数据中心，还成功移植到了各大公有云及 Neo-clouds 上。 - **极致的算力芯片异构性**：在 Twine 的屏蔽封装下，上层业务服务无需为不同芯片编写定制代码。目前这套底座已统一驱动了英伟达全系硬件（H100、B200、GB200、GB300）、Meta 自研 AI 芯片 MTIA、AMD 最新的 MI355X 系列 GPU，并即将全面接入 Google TPU。 - **破除跨地域数据墙的双轨存储策略**：地理跨度带来的海量数据搬迁成本极高，Meta 采取了两手准备：一方面打造了 **Storage Hub** 架构，允许公有云/Neo-clouds 上的算力节点直接高速访问保留在 Meta 本地数据中心内部的存储服务；另一方面，存储团队开发了专门运行于公有云 Twine 共享层之上的 **Storage Light**——一套低依赖度、高度精简的新型轻量级云原生存储栈。 - **阶梯式压测性能调优**：业务迁徙经历了严谨的技术演化路径：最初只运行较为简单的**合成数据生成（synthetic data generation）**，随后拓展到**离线与批处理推理（offline/batch inference）**，如今公有云底座已完全有能力支撑极高吞吐的**强化学习（RL）**训练以及超大规模的**实时线上模型服务（Model Serving）**。 在成功聚合全球零散算力之后，对于习惯了“全域部署（Globally）”或“单地域部署（Region-local）”经典抽象模式的后端工程师而言，又引发了关于分布式系统架构治理与容灾范式的新一轮思考。 ### [10:26](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=626s) 跨云与自建数据中心的扩展瓶颈：抽象网络环架构（Rings） 随着 Meta 开始从各大云服务商接入分布于各地的多区域（Regions），基础设施团队突然遭遇了前所未有的全新扩展瓶颈。在过去，上线一个新的 Region 并非频繁事件；然而随着 AI 算力需求的爆发，新建区域容量的速度直接卡在了产品团队以及 Meta 超级智能实验室（Super Intelligence Lab）获取算力的关键路径（critical path）上。雪上加霜的是，每个 Region 内部都必须部署一套由核心基础服务构成的法定副本组（quorum of services），当不同云厂商和自建数据中心迅速扩张时，重复部署与跨区通信的系统开销变得异常庞大。 面对“如何才能让这个问题真正可解”的难题，工程师们坐在一起抓破头皮构思方案，最终提出了一个优雅的破局方案：“我们何不把地理位置上彼此临近的区域打包？无论是来自公有云提供商，还是 Meta 自建的数据中心，统统用超大带宽的高速网络直连起来。”团队将这种全新的逻辑抽象命名为 **Rings（环）**。 在 Rings 的抽象下，跨服务的部署与通信模式发生了质的改变。例如 Meta 极为庞大的前端层（Front-end tier）在调用后端依赖时，原本必须要求后端全部同构部署在同一个 Region 内；现在只要两个区域同属于同一个 Ring，即便前端在 Region A、后端在 Region B，也能以极低延迟协同工作。目前 Meta 已在生产环境成功上线了两个 Rings，并正全力将该架构向全球多地铺开。 ```mermaid flowchart TD accTitle: Meta Rings 架构示意图 accDescr: 展示自建数据中心与第三方云通过超高带宽直连构成分组 Ring 的结构 subgraph Ring["高带宽网络环 (Ring)"] direction LR subgraph Region1["Region A (Meta 自建数据中心)"] Frontend["Meta 前端层服务"] end subgraph Region2["Region B (第三方云服务商)"] Backend["分布式后端核心服务"] end Frontend <-->|"超大高带宽专网直连"| Backend end ``` --- ### [12:01](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=721s) 面向企业级 Agent 的基建创新：Meta CLI 与 Meta IAC 在解决了支撑 AI 运转的底层算力与网络拓扑之后，Surupa Biswas 将焦点转向了基础设施所释放的应用价值——**如何在企业内部利用 AI Agent 完成真正有价值的严肃工作**。Meta 约在 6 至 9 个月前开启了这段探索旅程，但起步便遭遇了严峻的工程现实：在像 Meta 这样庞大的企业内部，哪怕仅仅是发起会议、发送日历邀请等日常事务，背后都牵涉盘根错节的专有系统与独立工具。如果直接把一个通用智能体扔进企业环境，它根本无法顺利穿透并贯通所有这些工具链。 为此，团队为 Agent 的互操作性打造了两大基石级技术设施： 1. **Meta CLI（命令行目录服务）**：扮演企业内部能力的统一目录中枢。所有散落在各团队的孤立工具和业务系统，都能通过标准规范接入 Meta CLI。令人惊喜的是，在仅仅约 3 个月的时间里，全公司就有超过 500 多个内部工具与核心系统主动接入了该体系，让 Agent 能够极快地完成检索、发现并顺畅调用。 2. **Meta IAC（Infrastructure as Code，基础设施即代码）**：与 Pulumi 创始人 Joe Duffy 所倡导的现代 IAC 理念一脉相承，Meta IAC 专注于为生产环境筑起安全护城河。当智能体开始自主对高敏感资源（尤其是控制平面资源，control plane resources）执行生产变更操作时，Meta IAC 能确保这些操作必须通过声明式的代码逻辑与严格检验通道进行。其核心目标是赋予 Agent 足够强大的控制力，同时绝不给它们任何破坏企业命脉或影响终端用户的机会。 --- ### [13:54](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=834s) 构建 Agent 安全与自愈飞轮：告别“平行安全栈” 业内在探讨企业级 AI 时，往往将视线聚焦在那些最显眼的顶层收益上——比如如何利用 AI 提升员工生产力，或者如何利用 AI 提高系统可靠性。然而，真正释放 Agent 潜力的底座恰恰位于水面之下：必须先为 AI 本身铺平高生产力的道路，更要彻底解决全行业共同焦虑的**智能体安全性（Agent Safety）**。 Surupa 特别强调，Meta 并没有为 Agent 另起炉灶建立一套平行的安全控制栈。相反，Meta 全面继承并复用了十余年来在隐私、数据安全和高可用架构方面积累沉淀的坚实基建，在此基础上进行加固以适应智能体特性： - **刚性安全护栏（Safety Guardrails）**：设立不可逾越的底层硬约束，例如从物理权限上杜绝 Agent 拥有在用户数据库中直接执行删除操作的能力。 - **可恢复性与自动回滚（Recoverability & Automated Rollbacks）**：正视“Agent 必会犯错”的客观规律。系统让 Agent 能够执行最小粒度的金丝雀部署（minimal deployment），紧接着调取经受多年线上考验的服务健康指标进行校验；一旦检测到任何异常，立即触发自动化回滚，最大限度避免对真实用户造成冲击。 - **自主进化的 Eval 闭环（Self-Improving Feedback Loop）**：Meta 践行了类似 Satya Nadella 在近期企业级 Agentic AI 备忘录中提及的理念。系统对全公司所有基于 Meta 基础架构运行的 Agent 执行轨迹进行全量追踪；一旦智能体在某一步骤受阻执行失败，系统将自动提单（File task）、自动提交代码修复补丁（Submit diffs），并在闭环中持续跑评测（Eval loop）。人类工程师只需在关键卡点上介入提供微小的增量改进，就能驱动工作流源源不断地向端到端全自动进化。 --- ### [16:38](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=998s) 开发者体验的自我反思：从重型基建转向拥抱开源生态与 Kubernetes 展望未来的技术演进，Surupa 指出，AI 的真正质变不仅在于追逐 LLM 的前沿评测跑分，更在于以指数级加速人类在物理学、生物学、医学领域的探索进程；对于普通人而言，搭载 Agent 体验的智能眼镜（如 Meta Ray-Bans）正让视障与听障人士以从未有过的方式与所爱之人交流。在这个研发人员可以在一周之内尝试几十种全新想法的爆发期，Meta 基础设施团队迎来了一个“发人深省、令人谦卑的时刻”： > “对于我们这些从事超大规模系统研发的人来说，我们认识到：当大家只想快速对海量点子进行敏捷试验时，大型基础设施有时必须学会‘靠边站’，不要成为阻碍。” 然而，基建团队的决定性价值同样无可替代——必须让原型验证（prototyping）到工业级量产交付（production scale）的跨度从传统的“数月之久”压缩至“几天或数周以内”。为此，Meta 团队选择重回技术初心，全面拥抱开源生态： - 面对类似 GLM 5.2 等模型层出不穷的现状，研究人员渴望直接拉取镜像即可安全验证，而无需等待繁杂的内部私有系统改造。 - 团队全力拥抱 Kubernetes，在不到一个季度（几周）的极短时间内，打造出了完全合规 CNCF 标准的全新 **Meta Kubernetes Service（MKS）**，并直接上线了承载生产工作负载的集群。 - MKS 能够光速落地的秘诀在于其站在巨人的肩膀上：它完全复用了 Meta 现有的 Twine 共享调度底座，原生继承了其世界级的物理机生命周期管理、容量调度与作业编排能力。 --- ### [19:21](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1161s) 数据基础设施的复兴与革新：Postgres、推测执行与 DSQLite 在基础数据系统领域，老牌的 Postgres 数据库在此刻迎来了跨时代的复兴。为了满足 AI Agent 对数据底座的极致要求，Meta 对数据库架构实施了彻底的演进： - **计算与存储完全解耦（Disaggregation of compute and storage）**：使得算力节点与底层存储容量可以完全独立扩缩容，AI Agent 得以在**几秒钟内**按需动态拉起全新的数据库实例。 - **推测执行与版本控制（Speculative Execution）**：针对 AI 研究员的高频诉求，实现了类似代码版本管理（Git-like for data）的机制。研究员和 Agent 可以在数据库中执行试探性的探索写入，在验证完成后可自由决定是一键合流（Merge）还是直接抛弃（Drop）。 - **推出轻量化 DSQLite**：团队紧接着快速孵化了全新的 DSQLite 产品，力求在保留完整 SQLite 原生语义的同时，将运维开销压缩到极致，为智能化实验场景提供无感的数据支撑。 ### [20:12](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1212s) 敏捷开发范式变革：从 C++ 静态链接到 Nest 与 Project Hub 在底层存储方面，各项新型服务均由 Meta 生产级分布式键值存储 ZippyDB 提供强力支撑。针对这一系统，工程团队在此前举办的 AI and Data @Scale 大会上已有过深入剖析。然而，基础设施的进化并未止步于 Kubernetes 集群与数据库层，团队开始自底向上全面审视整个技术栈，以应对一种全新的开发范式：当工程师从 GitHub 下载开源项目尝试快速原型开发时，该如何无缝调用 Meta 庞大的内部基础设施？ 为了打破传统内部工具链的繁琐束缚，Meta 的开发者基础设施（Dev Infra）团队打造了名为 Nest 的应用平台。Nest 类似于企业内部的 Vercel 解决方案，底层构建于 FaaS（函数即服务）与分布式轻量存储 DSQLite 之上。该平台上线后呈现爆发式增长，在短短几个月内就承载了 22,000 多个内部应用的构建，其中便包括 Meta 全员日常使用的内部 AI 助手 MetaMate。 与此相对应的是服务间通信机制的彻底重塑。在 Meta 传统的后端研发流程中，服务通常需要与庞大的 C++ 库进行静态链接，通过库导出的接口实现远程调用。但对于追求敏捷实验、基于开源生态的小型 GitHub 项目而言，这种机制过于沉重。为此，Meta 全面引入了跨机器的 gRPC 通信支持，并将核心后端服务封装为简单直观的 HTTP 端点。如今，这一名为 Project Hub 的统一接入与调用层，已经全面承载了 Meta AI 整个产品级基础设施层的运行与交互。 - **ZippyDB 稳健底座**：作为生产级数据存储核心，为新一代云原生与 AI 基础设施提供高吞吐、低延迟的数据持久化支持。 - **Nest 平台高速渗透**：基于 FaaS 与 DSQLite 构建，以极低门槛支持内部微服务与工具研发，数月内催生超 22,000 个内部应用。 - **Project Hub 通信解耦**：彻底改变重度依赖 C++ 静态链接的遗留调用链路，向现代化的 gRPC 与 HTTP 端点演进，成为 Meta AI 产品的核心接入网关。 ### [21:48](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1308s) 智能体安全沙箱：Feather VM 与持久化状态存储 伴随着智能眼镜等边缘硬件与个人 Agent 的深入融合，用户已经能够通过语音自然地向 Agent 指派现实任务。然而，要让智能体真正接管具有实际业务价值的工作，安全隔离与数据隐私成为了不可逾越的底线。智能体不仅需要独立、可控的代码执行空间，还需要在不同的交互会话之间保持连贯的上下文与状态。 为了解决这一难题，基础设施团队在极短的时间内开启了攻坚，仅用了数周时间便从零到一构建起了一套专用的安全隔离与持久化环境——Feather VM 及其配套存储卷（Storage Volumes）。Feather VM 为智能体任务提供了极轻量化且高安全性的虚拟化运行沙箱，配合会话间持久化存储，使得 Agent 既能拥有长期记忆与上下文关联能力，又绝不会突破安全与隐私边界。这项快速落地的底座架构，成为了 Meta 推动自主智能体走向复杂、高价值生产场景的关键支柱。 - **安全强隔离沙箱**：通过极轻量的 Feather VM，确保 Agent 运行不受信任代码或处理敏感数据时的系统安全。 - **跨会话持久上下文**：借助专属存储卷解决大模型交互中的无状态痛点，为复杂长周期任务提供连贯记忆支持。 - **极致工程交付速度**：整个 Feather VM 与持久化存储体系从技术预研到落地生产，仅历时数周即实现 "0 到 1" 的突破。 ### [22:26](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1346s) 全栈协同设计：Meta 的异构算力与基础设施架构 从底层物理设备到顶层业务逻辑，Meta 正在重构一套庞大而灵活的端到端技术体系。整个基础设施技术栈自底向上清晰地划分为四大层级，其核心设计理念是通过软件抽象屏蔽底层硬件差异，并在传统超大规模私有设施与现代云原生开放标准之间实现深度融合。 ```mermaid flowchart TD accTitle: Meta 核心基础设施技术栈全景图 accDescr: 展示从底层异构算力供应层到顶层各类 AI 工作负载的四层技术架构 subgraph L4["应用层 (Workloads)"] AI["Meta AI / MetaMate / 各类智能体工作负载"] end subgraph L3["平台与软件服务层 (Software Layer)"] Modern["云原生 API (Kubernetes / S3 兼容对象存储)"] Legacy["自研生产级系统 (ZippyDB / 专用 RPC 框架)"] end subgraph L2["基础抽象软件层 (Foundational Software)"] Abstr["硬件抽象层 (屏蔽算力与网络拓扑差异)"] end subgraph L1["异构供应层 (Heterogeneous Supply)"] DCs["自建数据中心 (Data Centers)"] Clouds["公有云与新型智算云 (Neo Clouds)"] end DCs --> Abstr Clouds --> Abstr Abstr --> Modern Abstr --> Legacy Modern --> AI Legacy --> AI ``` 在该体系的最底层，是高度异构的物理算力供应层，广泛涵盖自建数据中心、主流公有云以及专注 GPU 算力的新型智算云（Neo Clouds），容纳了多种截然不同的芯片与硬件拓扑架构；其上的基础软件层负责提供统一的硬件抽象，消除算力供应端的碎片化。在软件服务层，K8s 容器编排体系以及 S3 兼容的对象存储 API，与 Meta 历经十余年打磨的成熟私有基础设施并驾齐驱，共同为顶层复杂的 AI 训练与推理负载提供支撑。这种架构使得工程团队能够实现前所未有的全栈协同设计（Co-design），从芯片、服务器等物理层指标一直贯穿优化至最终的应用层。 ### [23:20](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1400s) 基础设施的“电气化时刻”：迎接 AI 带来的指数级跃迁 回顾技术演进的周期，AI 的发展经历了三个关键阶段：最初是“理论阶段”，整个行业满怀对 AI 潜力的巨大憧憬，在基础设施领域进行狂热的超前布局；随后进入当前所处的“价值初现阶段”，以 AI 编程助手和端侧智能体为代表的产品开始展现出扎实的生产力变革能力；而正在到来的下一阶段，将是 AI 为全球人类创造“指数级价值”的历史性时刻。 对于基础设施工程师而言，AI 的指数级价值必然直接转化为基础设施工程量的指数级增长。这正如人类历史上曾经经历的铁路大提速时代与全面电气化时代，每一次生产力的质变，底色都是庞大、复杂且持久的基础设施大扩张。面对这一充满复杂技术挑战与前所未有增长机遇的历史窗口，整个工程社区应当团结协作，共同攻克物理散热、集群协同、分布式存储及系统工程中最硬核的难关，在历史性的技术浪潮中共同见证由基础设施所开启的崭新时代。 ## Notable quotes > [00:01](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1s) 两年前，她重点探讨了“AI 驱动系统革新，同时系统革新反哺 AI 算法”的双向互动；一年前，她与 Peter Hoose 深入拆解了这种动态交互如何自底向上颠覆整个基础设施技术栈，彼时 Meta 团队聚焦的核心指标是快速上市时间（Time to Market）、架构可变通性（Optionality）以及创新速度。 — Surupa Biswas > [00:01](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1s) 在 Meta 的核心基础设施工程团队内部，日常提交的代码变更中，超过 90% 都包含由 AI 辅助或自主生成的代码。 — Surupa Biswas > [07:29](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=449s) 必须彻底打破过去的壁垒，拥抱在公有云及新型 AI 云（Neo-clouds）上运行生产级工作负载 — Surupa Biswas > [10:26](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=626s) 我们何不把地理位置上彼此临近的区域打包？无论是来自公有云提供商，还是 Meta 自建的数据中心，统统用超大带宽的高速网络直连起来。 — Surupa Biswas > [16:38](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=998s) 对于我们这些从事超大规模系统研发的人来说，我们认识到：当大家只想快速对海量点子进行敏捷试验时，大型基础设施有时必须学会‘靠边站’，不要成为阻碍。 — Surupa Biswas > [16:38](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=998s) 必须让原型验证（prototyping）到工业级量产交付（production scale）的跨度从传统的“数月之久”压缩至“几天或数周以内”。 — Surupa Biswas > [20:12](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1212s) Nest 类似于企业内部的 Vercel 解决方案，底层构建于 FaaS（函数即服务）与分布式轻量存储 DSQLite 之上。该平台上线后呈现爆发式增长，在短短几个月内就承载了 22,000 多个内部应用的构建 — Surupa Biswas > [23:20](https://www.youtube.com/watch?v=Ei4AbefZxbg&t=1400s) 这正如人类历史上曾经经历的铁路大提速时代与全面电气化时代，每一次生产力的质变，底色都是庞大、复杂且持久的基础设施大扩张。 — Surupa Biswas ## Mentioned Surupa Biswas, Meta, Casey, Peter Hoose, Joe Duffy, Satya Nadella, Pulumi, OpenClaw, Meta Ray-Ban, Micron, Google, SpaceX, Google Cloud TPU, NVIDIA, H100, B200, GB200, GB300, AMD, MI355X, MTIA, Twine, Storage Hub, Storage Light, Rings, Meta CLI, Meta IAC, Meta Kubernetes Service (MKS), Postgres, DSQLite, ZippyDB, Nest, MetaMate, Project Hub, Feather VM, GLM 5.2 ## Video description Surupa Biswas is the Vice President of Engineering responsible for Core Infrastructure at Meta, the internal cloud serving Meta’s Family of Apps & Meta AI. Her group builds the online distributed systems that power Meta's offerings, and the platforms Meta engineers build on. Surupa started at Facebook in 2013. Her career spans nearly two decades of working on programming languages, compilers, developer tools and distributed systems at Microsoft and Meta. She holds degrees in Computer Science and Engineering from the University of Maryland in College Park. Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=Ei4AbefZxbg)

[0:01] Surupa Biswas: 两年前，她重点探讨了“AI 驱动系统革新，同时系统革新反哺 AI 算法”的双向互动；一年前，她与 Peter Hoose 深入拆解了这种动态交互如何自底向上颠覆整个基础设施技术栈，彼时 Meta 团队聚焦的核心指标是快速上市时间（Time to Market）、架构可变通性（Optionality）以及创新速度。 [0:01] Surupa Biswas: 在 Meta 的核心基础设施工程团队内部，日常提交的代码变更中，超过 90% 都包含由 AI 辅助或自主生成的代码。 [7:29] Surupa Biswas: 必须彻底打破过去的壁垒，拥抱在公有云及新型 AI 云（Neo-clouds）上运行生产级工作负载 [10:26] Surupa Biswas: 我们何不把地理位置上彼此临近的区域打包？无论是来自公有云提供商，还是 Meta 自建的数据中心，统统用超大带宽的高速网络直连起来。 [16:38] Surupa Biswas: 对于我们这些从事超大规模系统研发的人来说，我们认识到：当大家只想快速对海量点子进行敏捷试验时，大型基础设施有时必须学会‘靠边站’，不要成为阻碍。 [16:38] Surupa Biswas: 必须让原型验证（prototyping）到工业级量产交付（production scale）的跨度从传统的“数月之久”压缩至“几天或数周以内”。 [20:12] Surupa Biswas: Nest 类似于企业内部的 Vercel 解决方案，底层构建于 FaaS（函数即服务）与分布式轻量存储 DSQLite 之上。该平台上线后呈现爆发式增长，在短短几个月内就承载了 22,000 多个内部应用的构建 [23:20] Surupa Biswas: 这正如人类历史上曾经经历的铁路大提速时代与全面电气化时代，每一次生产力的质变，底色都是庞大、复杂且持久的基础设施大扩张。

[0:01] Surupa Biswas: Open canvas era [0:01] Surupa Biswas: Over 90% of daily code changes submitted within Meta’s core infrastructure team now contain AI-authored code, even across multi-decade-old systems. [4:40] Surupa Biswas: Building FOR AI, building WITH AI, and ultimately reaching exploration at scale. [4:40] Surupa Biswas: The modern AI race is fundamentally a physical supply chain race. [10:12] Surupa Biswas: Rings [19:21] Surupa Biswas: Postgres is having a moment, again [19:21] Surupa Biswas: Kind of like source control, but for data [22:26] Surupa Biswas: Railroads moment [22:26] Surupa Biswas: Electrification moment [22:26] Surupa Biswas: Exponential value from AI, the trend is it will result in exponential work in infrastructure.

[0:01] Hey everyone, thank you Casey. Good [0:03] morning. Welcome. Welcome to our first [0:06] at scale event here in Bellevue. [0:09] Bellevue used to be home for me for many [0:11] years, so it feels really special to be [0:13] back here in a room with all of you, [0:16] some of the foremost infrastructure [0:18] builders in the world during this time [0:20] of unprecedented growth and progress. [0:24] The problems that we are solving in [0:25] infrastructure today have unparalleled [0:28] increase in scope, complexity, scale, [0:32] and I love events like at scale because [0:35] this is where we get to come together as [0:36] a community, even though we work in very [0:38] different companies, and exchange notes [0:41] on some of these problems that nobody [0:43] has ever solved before. [0:46] How many of you here were at an at scale [0:48] event in the past? [0:51] Wonderful. So, for those of you who have [0:53] been following the work that we've been [0:55] doing and the things we've been [0:56] discussing here, 2 years ago I had [0:59] talked about this trend where we were [1:01] seeing AI influencing systems [1:03] innovation, and we were also seeing [1:05] systems innovation influencing AI [1:07] algorithms. [1:09] Then a year ago, I talked about how this [1:11] dynamic was leading to us changing [1:14] things up and down the infrastructure [1:16] stack. Peter Who's and I talked [1:18] extensively about this a year ago, and [1:20] we were optimizing for time to market, [1:23] optionality, and innovation. If you look [1:26] at these dimensions, it's all because we [1:28] were trying to move super fast in [1:30] anticipation of all kinds of changes. [1:33] The reason optionality was so important [1:36] to us is because we were anticipating [1:38] this great unlock of value from AI, but [1:41] we didn't quite know what products we [1:43] were going to be building, and of course [1:45] what the specific infrastructure [1:47] challenges would be that would emerge. [1:49] Now, if we look at the time and the [1:51] moment that we are in right now, we've [1:53] very much gone from theory [1:56] to real value from AI. [1:59] If you ask me what the watershed moment [2:01] in the last 12 months has been, it was [2:04] the open claw moment. And I think it's [2:06] because up until then, most of us that I [2:09] know, we were using AI more in a [2:12] search-like fashion. They were single [2:13] session, we would ask some questions, [2:15] get some answers. But agents really [2:18] unlock the ability to complete [2:20] meaningful work end-to-end on our [2:23] behalf. [2:24] And to be able to have persistent memory [2:26] between sessions, which makes it able to [2:29] do much more powerful and meaningful [2:31] things. [2:33] So, we can look at some of the metrics [2:35] here. For all of you here in the room, [2:37] like this one will feel very familiar. [2:40] Agents are now doing coding and not just [2:42] for zero to one, like new applications, [2:45] but even if we look at complex systems [2:47] code bases, systems that might be [2:49] decades old, we are now finding that [2:52] agents are able to code up really [2:54] intricate and complex changes, and often [2:56] time correctly as well. [2:59] If I just look at Meta's core [3:00] infrastructure team and the changes that [3:03] are being submitted on a day-to-day [3:04] basis, over 90% of those now contain [3:07] AI-authored code. [3:09] Where else is this value flowing? Of [3:12] course, it's flowing into AI [3:13] infrastructure itself. I think this [3:15] number I have up here, 700 billion, [3:18] which is such a large number, may in [3:20] fact be a conservative estimate. This is [3:23] simply the sum of the CapEx spend that [3:26] has been announced by the five large [3:28] hyperscalers. [3:30] So, this really is an unprecedented [3:32] moment for all of us. [3:35] If I hearken back to another [3:37] unprecedented moment, it was the mobile [3:40] internet moment. And the reason I like [3:42] to compare these two is because the [3:44] internet had been around for many, many [3:46] years. But having the internet in our [3:48] pocket really opened up the value of the [3:51] internet and a lot of use cases that had [3:54] never existed before. [3:56] And what we are finding is this agentic [3:58] execution is also opening up AI in brand [4:01] new ways. It's not hard to imagine a [4:04] future like these Meta Ray-Bans. I've [4:06] been wearing them, I think since they [4:07] first came out in 2021, mostly for audio [4:11] and taking some pictures at the time. [4:13] But now I can use these glasses to talk [4:15] to my agent, give it things to do while [4:18] I stay really immersed in my day-to-day. [4:22] So, the question for us today and the [4:24] thing I would love to walk you all [4:26] through is as the scope and scale of [4:29] agents grow, what does it mean for this [4:31] community, those of us in the room and [4:34] all of you on the live stream? [4:36] I want to talk about this journey that [4:39] we are on. [4:40] We were originally really focused on [4:42] building for AI. Like when I talk about [4:45] the anticipation of AI and unlocking [4:47] value from AI and building all of this [4:50] infrastructure, that was really all [4:52] about building for AI. If I look at the [4:54] present moment, we are building for AI [4:57] with AI. And then the thing that we are [5:00] looking forward to and are most excited [5:01] about is exploration at scale and all [5:04] the new things that are going to come [5:06] about. [5:07] So, I'm going to talk about these three [5:09] trends from these time periods and the [5:12] scaling challenges that these have [5:14] created for all of us here, which is [5:16] very exciting because this is exciting [5:18] work for all of us to do here. And I'll [5:20] try to leave you all with a [5:22] behind-the-scenes view of how we're [5:23] tackling these things inside Meta. [5:26] So, with when it comes to building for [5:27] AI, we will talk about supply chain and [5:29] supply chain considerations. [5:32] Then I'll talk about building with AI [5:34] inside the enterprise and then finally [5:36] talk about the unique things that are [5:38] coming up with exploration at scale. [5:40] Let's go dig into these one by one. [5:43] Nothing makes it more clear how much of [5:46] the AI race is an infrastructure race, [5:49] then just looking at the supply [5:50] landscape. I don't know, you all can [5:52] like think about the thing that is [5:54] closest to your day-to-day work. It [5:56] could be data center power, it could be [5:58] network fiber capacity, it could be fab [6:01] capacity, it could be silicon. And with [6:04] silicon, it's GPUs, CPUs. I mean, CPUs [6:07] are having a moment at this point in [6:09] time. With agentic execution, I'm seeing [6:12] so much innovation in the CPU space as [6:14] well. But then, if you look at memory, [6:17] whether it's HBM or regular DRAM, or [6:20] even hard drives, there isn't enough of [6:24] any of these resources to be found in [6:26] the market right now. I mean, Micron [6:28] announced their earnings yesterday. That [6:30] on its own kind of tells you the story [6:32] and the landscape that we're dealing [6:33] with. And as a result, I feel across the [6:36] industry and across our companies, we [6:38] are innovating in ways that we couldn't [6:40] have imagined a few years ago. [6:42] For example, just a few months ago at [6:44] Meta, we announced that we will be [6:46] unlocking up to 6.6 gigawatts of nuclear [6:50] power here in the United States by 2035. [6:54] If you think about some of our industry [6:56] peers, like Google and SpaceX, they're [6:59] quite literally working on projects for [7:01] orbital data centers. And sooner or [7:03] later, we're going to have data centers [7:05] in space. [7:07] So, we have a lot of work that's [7:09] happening in the physical layer. And one [7:11] of the things that we're noticing is [7:12] more and more we are designing software [7:15] along with the physical infrastructure. [7:17] So, co-design is a theme that you can [7:19] see across the industry. And later this [7:22] afternoon, my colleague Peter Hoose is [7:24] going to give you a walk-through of many [7:26] of the things that are happening in the [7:27] physical infrastructure layer. [7:29] So, now let's talk about what we've been [7:31] doing at Meta. [7:32] As these supply challenges arose, we've [7:35] been innovating in new ways. So, many of [7:38] you, if you've been following our work [7:39] here at Meta, we've have running our [7:41] production workloads really on Meta [7:43] owned and operated data centers. And [7:45] we've had our own data centers for quite [7:48] a while now, since 2011. So, it's been [7:50] 15 years. [7:52] But 1 and 1/2 years ago, we realized [7:54] that in order to really get capacity [7:56] wherever we could find it, we had to [7:59] really embrace running production [8:00] workloads on public cloud. [8:04] A year ago here, I talked about how we [8:06] had started to experiment with one [8:08] cloud. And now, if you look at us a year [8:12] later, we are running hundreds of [8:14] megawatts of production workloads across [8:16] multiple clouds. And you're going to [8:18] hear a dedicated talk from Dinesh and [8:21] Sargun shortly on our journey to [8:23] building the world's largest hybrid [8:25] cloud. [8:27] But none of this has come for free. It [8:29] has required significant amount of work. [8:31] So, they're going to talk about this [8:33] Twine shared layer, which is our host [8:35] management layer, which now runs not [8:38] just on Meta on premises, but also on [8:41] public clouds, as well as on Neo clouds. [8:44] Just to give you a flavor of the level [8:45] of heterogeneity we're dealing with, we [8:47] today are running our workloads on [8:49] H100s, B200s, GB200s, GB300s. We have [8:54] our own Meta silicon MTIA. We are [8:57] running on AMD GPUs like the MI355Xs. [9:01] And soon, we are also going to be [9:03] running Google TPUs. [9:06] So, it's not just the compute layer that [9:08] we had to abstract out so services on [9:10] top run without needing to do custom [9:13] work for all of these different [9:15] chipsets. We also had to do a lot of [9:17] work on the storage front. And data is [9:20] really cumbersome to move around between [9:22] geographical locations. So, what we did [9:24] is we made it possible to run production [9:27] workloads on a public cloud or a Neo [9:29] cloud, which accesses storage that is [9:31] running on Meta's data centers. This is [9:33] what we call the storage hub. And I'm [9:35] going to touch a bit on this in the next [9:37] section as well. [9:39] But in addition to that, our storage [9:40] team also built a newer, lower [9:43] dependency storage stack called storage [9:45] light, which runs on top of this TW [9:48] shared layer in the public cloud. [9:51] Last but not least, in order to make [9:53] this work, we had to do a lot of [9:55] performance work. We started with [9:56] simpler workloads like synthetic data [9:58] generation, then we went to offline or [10:01] batch inference, but now we're running [10:03] reinforcement learning and also doing [10:05] model serving at scale all on public [10:08] cloud. [10:10] So now, let's talk a little bit about [10:12] what it meant to swoop up capacity [10:14] wherever we could find it. So for those [10:16] of you here who work on back-end [10:17] services, back-end services tend to have [10:20] some of these common abstractions. They [10:21] tend to be the run globally or they run [10:24] as region local services. But as we [10:26] started to bring up all of these [10:28] different regions from different cloud [10:30] service providers, we suddenly realized [10:32] that we are faced with a new scaling [10:34] problem. The first one you see, first [10:36] new region turn-up used to be not a very [10:39] common event. But suddenly, the time it [10:41] takes to turn up new capacity was on the [10:44] critical path of us giving capacity to [10:46] our product teams and to Meta Super [10:47] Intelligence Lab. [10:49] Not only that, each of these regions has [10:52] a quorum of services in them. But if you [10:55] multiply that out across so many this is [10:57] not really a complete true to uh factual [11:00] picture, but it gives you a sense of how [11:02] many of these kinds of things exist now. [11:05] So then that started to become a real [11:07] overhead. So a bunch of my colleagues [11:09] were sitting in a room scratching their [11:10] heads wondering, "What can we do to [11:12] really make this a solvable problem?" [11:15] And they're like, "You know what? I have [11:16] an idea. Why don't we just take the ones [11:19] which are geographically close to each [11:21] other? It doesn't matter if they come [11:22] from a cloud provider or their Meta's [11:24] data centers, and let's connect them [11:27] with this massive high-bandwidth network [11:29] connection." [11:31] This is what we call rings. So, now [11:33] services have these rings abstraction. [11:36] The reason this is cool is you can now [11:38] run like a major service, for example, [11:40] Meta's front-end tier, and instead of [11:43] talking to back-end services, which are [11:45] all co-located in the same region, they [11:47] could be located in another region, but [11:49] inside the same ring. We have now [11:51] deployed two of these rings, and our [11:53] teams are actively working to create [11:55] many of these around many places around [11:58] the world. [12:01] So, all of this capacity, of course, [12:02] this was the building for AI part. Now, [12:04] let's transition to talking about the [12:06] value this is unlocking. And I'm sure [12:08] this is also something all of us are [12:10] working through right now, which is how [12:12] can we use agents to complete meaningful [12:15] work inside the enterprise? We got [12:17] started on this journey, I would say, 6 [12:20] to 9 months ago inside Meta, and we [12:22] immediately ran into a set of [12:24] challenges. The first challenge is [12:26] inside the company, you know, even to [12:27] schedule a meeting, send an invite, all [12:30] kinds of things we do on a day-to-day [12:32] basis, we have many different tools and [12:34] systems that exist. And we found that if [12:36] you just took an agent and tried to ask [12:39] it to complete one of these tasks, it [12:41] wouldn't be able to successfully [12:42] traverse through all these systems. So, [12:44] we built two foundational pieces of [12:47] technology for agent operability. The [12:50] first is Meta CLI. As the name suggests, [12:53] this is the directory service. So, all [12:55] of these different tools and systems got [12:57] plugged into the CLI. It was actually [12:59] wonderful to see. Over just 3 months or [13:01] so, 500-plus tools and systems just [13:04] added themselves to the CLI, so they [13:06] could be discovered and called readily [13:08] by agents. [13:09] We've also been working on Meta IAC, and [13:12] IAC stands for infrastructure as code. [13:15] And later today, you're going to hear [13:17] from the founder of Pulumi, Joe Duffy, [13:20] and he's going to talk about the [13:21] infrastructure at as code solution that [13:23] they've built. [13:24] But what that meant for us internally at [13:26] Meta is we want to make sure that [13:28] whenever agents are making prod [13:31] operations on critical resources like [13:33] control plane resources, they are being [13:35] done in a safe manner. So, we are [13:38] bringing this technology for control [13:39] plane resources to all services inside [13:42] the company that want to make sure that [13:44] agents are powerful, but they do not [13:46] destroy or break any things that are [13:48] sacrosanct to the company or to all of [13:50] the end users that we serve every day. [13:54] So, I want to give you a glimpse of this [13:57] Let me see if I can do justice to this [13:59] diagram, beautiful diagram that one of [14:01] my colleagues created. I would say when [14:03] we talk about AI inside enterprises, we [14:06] mostly talk about these boxes at the [14:08] top. For example, what I was talking [14:10] about was this purple box in the [14:11] right-hand top corner, which is all [14:13] about how can we leverage AI to be more [14:16] productive. We've also been talking [14:18] about how to leverage AI to make [14:20] ourselves more reliable, and you're [14:22] going to hear a lot more about that. [14:24] But, the thing that I discovered was [14:25] critical to unlocking the power of [14:27] agents are these things which are below [14:29] the line, which is we needed to do work [14:32] so AI itself could be productive. And [14:35] then perhaps the most important to the [14:37] crew that's tuned in today to listen to [14:39] all the reliability talks is we are all [14:41] incredibly worried about what agents [14:43] might do to things that we must protect. [14:46] And so, agent safety is another critical [14:49] area of focus and innovation right now. [14:52] So, you're going to hear right after [14:53] this a talk by David Parrott where he's [14:55] going to go in and and share with you in [14:57] depth what we are doing there. What I [15:00] wanted to leave you all with is simply [15:01] the fact that inside Meta, we have not [15:04] created a different parallel stack for [15:06] safety. We are taking all of the [15:08] infrastructure that we've built over the [15:09] years for privacy, for security, as well [15:12] as for reliability, and leveraging that [15:14] and enhancing that for agents. We are [15:17] hard at work building many different [15:19] agent safety guardrails. For example, [15:21] agents should never be able to delete [15:23] things in user databases, for instance. [15:26] But at the same time, we are also really [15:29] making sure that we invest in [15:31] recoverability, because we know agents [15:33] will make mistakes. We are building [15:35] systems such as like an agent can go do [15:38] a minimal deployment, look at the safety [15:41] health checks that we have for all of [15:42] our services, which have been hardened [15:44] over the years, and then based on those [15:46] results, automatically perform rollbacks [15:48] when there are mistakes, to ensure that [15:50] we don't hurt more end users than we [15:53] would ever need to. [15:55] The last thing I'll say here is for [15:57] those of you who read Satya's recent [15:59] memo on agentic AI in the enterprise, we [16:01] are also implementing something similar. [16:04] We are looking at all the trajectories [16:06] across the company that use meta [16:08] infrastructure, [16:10] and we are automatically filing tasks [16:12] whenever the agent falls over and cannot [16:14] complete a particular step, then [16:16] automatically submit diffs and keep [16:18] doing these eval in a loop. The [16:20] aspiration is if you have all of these [16:22] workflows automatically running these [16:24] eval loops, and then people go and make [16:27] them one step better each time, that we [16:29] will have this self-improving thing that [16:31] will make it possible for us to do more [16:33] and more meaningful work in the [16:34] enterprise end-to-end via agents. [16:38] All right. Now, let's switch gears and [16:41] talk about where we might be headed [16:42] next. [16:44] Perhaps the most exciting thing about [16:45] what's going on, I know we talk a lot [16:47] about AI agents in the enterprise, and [16:50] we talk about frontier LLMs and [16:52] benchmarks and things like that. But the [16:54] real unlock from AI is going to be the [16:57] acceleration of progress for all of us [17:01] on the planet. [17:02] And AI is already meaningfully making it [17:05] faster to do research in physics, in [17:07] biology, in medical sciences. I [17:10] mentioned the Meta Ray-Bans that I have [17:11] been wearing since 2021, but the [17:13] Ray-Bans with the agentic experiences is [17:16] making it possible for people who are [17:19] visually impaired or hearing impaired to [17:21] be able to have experiences where they [17:23] can communicate with their loved ones in [17:25] new ways for the first time. [17:27] So, this is really exciting to witness. [17:29] And of course, this is the time when [17:31] people can try out dozens of different [17:33] ideas all within the same week. [17:35] So, what does it mean for us in for [17:37] people? [17:38] So, one of the things that we learned [17:39] and this was a humbling moment for those [17:41] of us here who work in large-scale [17:43] systems. What we realized is when people [17:45] really just want to experiment with a [17:47] lot of ideas quickly, sometimes [17:50] large-scale infrastructure needs to get [17:52] out of the way. [17:54] At the same time, we have a very big [17:56] role to play because once somebody wants [17:58] to take one of these ideas and ship it [18:00] to production, we want to make it so [18:02] people can go from prototyping to [18:05] production scale in days or weeks, not [18:09] in weeks and months. And so, we have a [18:11] critical role to play here. [18:13] So, what did we do about this at Meta? [18:16] For us, it really went back to kind of [18:19] going back to our roots and really [18:21] embracing open source. What we realized, [18:23] you know, the GLM 5.2 model just came [18:25] out. People just want to be able to [18:27] download the model and be able to play [18:29] around with it. We needed to make it [18:31] much easier for researchers and others [18:33] inside the company to take Docker images [18:36] and just run them safely on Meta [18:38] infrastructure. [18:39] So, this is where we really leaned into [18:41] Kubernetes. And I'm proud to share that [18:43] in just a few weeks time, I would say in [18:44] less than a quarter, the team went from [18:47] having no Kubernetes service to having [18:49] this Meta Kubernetes service with a [18:51] production cluster running workloads. [18:53] And the reason we were able to move so [18:55] fast on this is because MKS builds on [18:58] top of the same Twine shared TW shared [19:01] infrastructure that I mentioned earlier. [19:04] So, all of the things around machine [19:05] management, capacity management, job [19:08] management could be inherited, and [19:10] that's what MKS uses as well. And this [19:13] is not just a meta service, this is [19:15] fully CNCF compliant, so you can take [19:17] any Kubernetes thing and run it [19:18] successfully on MKS. [19:21] Now, if we look at databases, well, for [19:24] the databases people in the crowd, [19:25] Postgres is having a moment, again, [19:28] after all these years. And we've now [19:30] made it possible, we disaggregate [19:32] between storage and compute, so they can [19:33] be scaled independently. And we've now [19:36] made it possible for agents to spin up [19:38] databases in just a few seconds. [19:41] We've also implemented features like [19:43] speculative execution, so you can [19:45] actually go and do work in the database [19:48] and then be able to either drop it [19:51] because it's speculative and you don't [19:52] want to continue or merge it. Kind of [19:54] like source control, but for data. And [19:57] this was a top requested feature from [19:58] our AI researchers. [20:00] I'm really also proud of the team for [20:02] moving fast and building this new [20:04] product called DSQLite, [20:06] which gives you full SQL Lite semantics, [20:09] but with minimal operations, and it is [20:12] backed by our production-grade data [20:14] store, ZippyDB. [20:16] If some of you caught the AI and data at [20:19] scale talks last week, Sarang Musti [20:21] there talked extensively about ZippyDB. [20:24] But we didn't just stop with Kubernetes [20:27] clusters and databases. We are looking [20:30] up and down the stack and trying to [20:31] support this paradigm where somebody [20:33] might build a GitHub project after [20:35] downloading some things from the open [20:37] source, and then they really want to be [20:39] able to also leverage the power of meta [20:42] infrastructure. [20:43] So, our dev infra teams built this app [20:45] platform called Nest, which is our [20:47] Vercel-like solution, which is built on [20:50] top of function as a service and [20:51] DSQLite, and it is really taken off like [20:54] gangbusters. I think in just a few [20:56] months, 22,000 internal applications [20:59] have been built on top of it, including [21:02] Meta's internal chatbot, which is called [21:04] MetaMate. [21:07] The second thing is how do you go [21:09] because you see the normal way of [21:10] operating, which I think will be [21:11] familiar to everyone, is you build your [21:13] back-end service and then you statically [21:15] link against C++ libraries, which have [21:17] exports for all of the things that you [21:18] want to call. But, if you're in GitHub [21:21] with the small project rapidly [21:22] experimenting, how are you supposed to [21:24] leverage meta infrastructure? So, we [21:26] have now made it possible to use things [21:29] like GRPC to talk across machines and [21:32] then we've exposed these meta back-end [21:34] services just like HTTP endpoints, so [21:37] you can call into them readily. [21:40] And the entire Meta AI product [21:42] infrastructure layer now uses this thing [21:45] which we internally call Project Hub. [21:48] Last but not least, you know, I talked [21:50] about the glasses and how today already [21:52] I can talk to my agent and give it [21:54] things to do. If I really want this [21:56] agent to be able to do meaningful work, [21:59] we need to do a great job with security [22:01] and with data privacy. So, we set about [22:04] building infrastructure for secure [22:06] isolation, as well as for persistent [22:08] context between sessions with these [22:10] things we call Feather VM and storage [22:12] volumes. This infrastructure was also [22:15] built from zero to one in just a few [22:16] weeks and it is a critical piece of our [22:18] story as we make it possible for people [22:21] to use agents to do more and more [22:23] meaningful work. [22:26] That was a whirlwind tour. To bring it [22:28] all together, I wanted to talk about the [22:30] infra stack at scale. [22:32] At the bottom, we have this [22:34] heterogeneous supply layer where I was [22:36] talking about our data centers, public [22:38] clouds, neo clouds, all of these [22:40] different architectures. Then, we have [22:42] the foundational software layer that [22:44] abstracts away all these differences and [22:47] then we have the software layer that's [22:48] on top. The thing to note here is you [22:51] will see things like K8s and S3 [22:53] compatible APIs for storage alongside [22:56] Meta's large-scale production [22:58] infrastructure that we've been running [22:59] for many years. And then at the very [23:02] top, we have all the different AI [23:03] workloads that I was sharing with you. [23:06] If there's one thing that you take away [23:08] from today's talk is this is such an [23:10] amazing time to be in infrastructure [23:13] because we can co-design things from the [23:16] physical infrastructure layer all the [23:18] way to the application layer. [23:20] So if I ground us back to where we were, [23:23] where we are, [23:25] and where we're going, we were in the [23:27] theoretical phase where we were building [23:29] feverishly in anticipation of the value [23:31] unlocked from AI. [23:33] Today we are in the early stage of [23:36] getting real value from AI. With agents, [23:39] coding has been an amazing domain, but [23:42] where we are going next is exponential [23:44] value. I think the moment that we are [23:46] all waiting for is to see AI unlock so [23:49] much value for all of us on the planet. [23:52] But the good news for those of us here [23:54] in this room and in this talk is that [23:57] exponential value from AI, the trend is [24:00] it will result in exponential work in [24:02] infrastructure. [24:04] We have the railroads moment, the [24:06] electrification moment, and now the AI [24:08] moment. What's common amongst all of [24:10] these is that these were tremendous [24:13] times of growth and potential in the [24:15] domain of infrastructure. [24:17] We are powerful as a community. I really [24:19] hope that we all get to enjoy this [24:22] phase. It It often feels very difficult [24:24] and challenging because there's so much [24:26] to do, but it really is a time for us to [24:28] get together and solve these hard [24:30] problems together. I hope you all have a [24:32] wonderful at scale today. Thank you so [24:34] much.