Meta Adapts Custom Tech Stack to Run Production Workloads on Public Cloud
@Scale
Meta expanded into multi-cloud infrastructure by co-designing hardware SKUs and building automated security proxies to solve massive GPU capacity shortages.
## Background
Published on YouTube by the @Scale channel on July 2, 2026, this presentation features Meta's Dinesh Govindasamy and Sargun Dhillon discussing how Meta extended its massive internal infrastructure to the public cloud. According to the page description, the talk covers Meta's initial cloud integration strategies to meet skyrocketing GPU compute demands and achieve capacity hyperliquidity across platform, OS, and fleet management stacks.
## TL;DR
Driven by the late-2024 AI boom, Meta pivoted from its traditional purely on-premises hardware strategy to rapidly acquire public cloud GPU capacity within three months. To maintain its strict security and operational standards, the team adapted its Offnet services stack and implemented Transparent TLS (TTLS) with eBPF to securely connect cloud instances to internal data centers without code changes. They introduced Public Cloud Manager (PCM) and extended internal Fleet Health tooling to manage multi-cloud maintenance and GPU failures automatically. By co-designing hardware SKUs directly with cloud vendors, Meta achieved seamless multi-cloud integration, enabling new cloud regions to be onboarded in just one week.
## Atmosphere & style
The presentation delivers an authoritative, deeply technical, and fast-paced engineering narrative that conveys the extreme urgency of the modern AI compute race. Speakers Dinesh Govindasamy and Sargun Dhillon present a candid look at hyperscale problem-solving, smoothly passing the mic back and forth as they detail pragmatic architectural tradeoffs. The tone is confident and pragmatic, illustrating how Meta transformed high-stakes security and reliability risks into a streamlined, highly automated hybrid cloud ecosystem.
## Key takeaways
- [00:01](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1s) Historically, Meta relied on an on-premises co-design strategy, tightly coupling custom hardware and software stacks to support rapid scaling.
- [01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) The late 2024 AI boom led Meta to acquire public cloud capacity to solve immediate GPU shortages while building physical data centers.
- [01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) Standardization around NVIDIA's DGX SuperPOD architecture made cloud GPU compute act like a uniform commodity, enabling rapid onboarding in under three months.
- [03:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=211s) Meta adapted its 'Offnet services stack' to run untrusted cloud infrastructure securely using an immutable OS model, attestation, and a dedicated proxy layer within 60 days.
- [06:21](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=381s) Meta extended its automated Fleet Health tooling into third-party clouds with custom diagnostic probes to circumvent cloud telemetry restrictions and ensure reliability.
- [08:03](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=483s) To convert cloud setups into generic capacity, Meta migrated cloud infrastructure to TW Shared and successfully onboarded its legacy 'Web' service to serve production traffic.
- [11:54](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=714s) Meta deployed Transparent TLS (TTLS) leveraging eBPF in the host kernel to automate transport encryption without requiring manual security rule updates or code changes.
- [14:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=871s) Public Cloud Manager (PCM) was created as an abstraction layer to unify GPU lifecycle management, automated repairs, and Maintenance Trains across multi-cloud providers.
- [17:54](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1074s) Meta co-designs cloud hardware SKUs directly with cloud vendors during early ideation, ensuring required telemetry, BMC capabilities, and debuggability are built in from day one.
- [19:50](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1190s) Meta achieved rapid integration metrics: onboarding a new cloud vendor in under a month, a new region in a week, and a new SKU in two weeks.
- [19:50](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1190s) The ultimate vision is achieving full 'capacity hyperliquidity' by extending the hybrid cloud flywheel from GPUs to CPUs and non-GPU accelerators worldwide.
Dinesh Govindaswamy, Director of Engineering supporting the compute infrastructure organization at Meta, opens the presentation alongside his colleague Sargun Dhillon. Dinesh details Meta’s long history of building out one of the most advanced on-premises infrastructure footprints in the tech industry. Starting in the mid-2000s as a single website (`facebook.com`), Meta grew rapidly into the early 2010s, expanding into a broad suite of applications while maintaining a core server-side strategy centered on high-scale interactive serving, APIs, and data services.
To keep pace with hyper-growth, Meta rejected generic public cloud solutions and chose to build its entire software and hardware stack from scratch. This established a tightly coupled co-design flywheel where hardware and software were optimized together—data centers were tailored for internal workloads, and software was shaped around custom hardware designs.
- **Tao:** Meta's specialized distributed data store acting as the heart of the social graph.
- **HHVM (HipHop Virtual Machine):** A custom, highly efficient execution runtime designed specifically for PHP.
- **Thrift:** A high-performance internal RPC framework.
- **Hive:** The primary data warehousing and OLAP engine.
- **Open Compute Project (OCP):** Meta's initiative to design and open-source custom, power-efficient data center hardware.
[01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) The AI Boom and the Decision to Shift to Cloud
By late 2024, a massive surge in AI model advancements—specifically around Large Language Models (LLMs)—forced a shift in Meta’s infrastructure posture. The engineering team realized these frontier LLM techniques needed to be rapidly integrated into Meta's core products. However, these models demanded unprecedented amounts of storage, compute, and, above all, GPUs. While Meta possessed vast reserves of CPU compute and storage, its existing GPU clusters were completely saturated. Realizing that building physical, GPU-dense data centers takes significant time, leadership made the strategic decision to acquire immediate compute capacity through public cloud partners.
- **Onboarding Timeline:** The decision to acquire cloud capacity was finalized in mid-December, and by mid-February (less than 3 months later), capacity was fully handed off to AI training partners.
- **Sense of Urgency:** Dinesh emphasizes that in the modern AI race "every day counts"—unutilized capacity directly wastes capital while delaying model development.
- **NVIDIA DGX SuperPOD Standardization:** Public cloud integration succeeded rapidly because NVIDIA's DGX SuperPOD reference architecture standardized GPU hardware topology across vendors, making GPU compute act like a uniform commodity.
[03:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=211s) Security Architecture and the Offnet Adaptation
Sargun Dhillon takes over to explain the daunting task of securing and validating public cloud capacity within two months. Meta's traditional security model relies on a deeply layered, defense-in-depth architecture: physical security with armed guards at data centers, network security with tight ACLs, transport security with end-to-end cryptographic authentication, and a hardware-rooted cryptographic root of trust running on hardened Meta OS. None of these physical or lower-layer assurances existed in third-party clouds, creating severe security concerns about running proprietary AI model weights on untrusted infrastructure.
To solve this, the engineering team adapted Meta's existing "Offnet services stack." Originally designed to run Meta Network Appliances in untrusted or physically hostile edge locations around the world, the Offnet model assumes the underlying hardware and network are fully compromised.
- **Bootstrapping Process:** A host boots into a Live CD-like immutable OS model, performs rigorous hardware attestation, and only then executes a local installation of Meta OS components and required daemons (such as Twine).
- **Offnet Services Proxy:** Cloud hosts never connect directly to first-party data centers; all traffic routes through a single Offnet proxy layer that audits connections and enforces strict authentication primitives.
- **Turnaround:** Adapting the Offnet stack enabled secure execution of Meta applications in the cloud within just 60 days.
[06:21](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=381s) Solving Cloud Reliability with Fleet Health Tooling
Once workloads began running in the public cloud, reliability became the primary operational hurdle. Meta’s internal data centers rely on "Fleet Health"—an automated telemetry system processing millions of hardware signals to identify failing or degraded hosts and proactively evacuate workloads. Cloud providers, operating at arms-length, lacked deep visibility into customer workloads and could only offer coarse, high-level system checks.
To bridge this gap, Meta’s Fleet Health and Network Production Engineering teams extended internal monitoring tools into cloud environments. Because third-party clouds restrict access to low-level telemetry, the teams created cloud-specific workarounds.
- **Telemetry Restrictions:** Cloud environments hid direct CPU temperature metrics and shared back-end network statistics.
- **Remediation Loop:** By crafting custom diagnostic probes, Meta regained the ability to detect failing nodes in real time, signal cloud providers for automated hardware repairs, and protect long-running training jobs from silent failures.
[08:03](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=483s) Unifying Workloads with TW Shared and Onboarding Web
Dinesh returns to discuss transitioning from single-use, bespoke AI setups to a generic cloud capacity model capable of running any Meta workload. Onboarding individual ML teams manually created friction: applications depended on Meta's internal caches, privacy frameworks, and scheduling assumptions, leading to job failures that "eroded trust" in cloud infrastructure. To create standard "capacity hyperliquidity," Meta executed two key structural decisions: migrating all cloud infrastructure into TW Shared (Twine Shared, Meta’s unified container orchestration pool) and demonstrating that complex legacy services could run in the cloud with zero specialized code changes.
To prove the platform's readiness, the engineering team intentionally selected "Web"—Meta's oldest, most complex PHP/HHVM monolith—as the test workload. Web contains years of accumulated legacy logic and depends on an extensive web of downstream data stores, caches, and configurators.
- **Production Scale:** By the end of the project, approximately 1% of Meta’s live internal employee traffic was actively served from public cloud capacity.
- **Three Strategic Reasons for Web:**
1. *Foundational Role:* Web houses core product logic; co-locating it with cloud services minimizes cross-region latency.
2. *Dependency Graph:* Web communicates with almost every internal service (Memcached, Tao, Service Router); solving Web's networking dependencies solves them for all other applications.
3. *Credibility:* Proving Web could run in the cloud served as the ultimate "if you can make it here, you can make it anywhere" proof concept for internal developers.
[11:54](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=714s) Transport Layer Security with TTLS and eBPF
While the Offnet proxy model worked for simple workloads, it failed to scale for Web. Offnet required security engineers to manually audit every downstream service, open specific firewall ports, and conduct individual security reviews. While feasible for small AI pipelines with a few dependencies, Web depends on hundreds of shifting downstream services, making manual rule-writing unmanageable.
To eliminate this bottleneck, Meta deployed TTLS (Transparent TLS), an internal transport-layer security service that automates encryption and mutual authentication between cloud containers and on-premises data centers.
- **eBPF Interception:** TTLS uses eBPF (Extended Berkeley Packet Filter) hooks in the host Linux kernel to transparently intercept outgoing connections at the transport layer without requiring any code changes from developers.
- **Tunneling Flow:** Outbound traffic from a cloud host is intercepted by a local TTLS sidecar proxy, wrapped in an encrypted TLS tunnel with host identity credentials, and sent to an on-premises reverse proxy that unwraps the payload and forwards plain text to the destination service.
- **Performance & Reliability:** Operating as a local sidecar on every host eliminates intermediate proxy hops, removing single points of failure while drastically cutting latency.
[14:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=871s) Managing GPU Lifecycles with Public Cloud Manager (PCM)
Sargun returns to address the distinct operational challenges posed by GPUs compared to CPUs. Decades of CPU cloud maturity have perfected live migration and transparent maintenance. In contrast, multi-node GPU clusters behave like tightly coupled supercomputers—if a single GPU fails or undergoes maintenance, the entire multi-node training job or cluster fails.
On-premises, Meta handles maintenance via "Maintenance Trains"—an automated system that rolls through the fleet every 45 days to decommission hosts, unprovision OS images, perform hardware repairs, and integrate with Twine so workloads proactively migrate without developer intervention. In the cloud, this had devolved into engineers manually coordinating maintenance over Slack, Jira, and vendor tickets, creating severe reliability issues across multi-cloud deployments.
- **Public Cloud Manager (PCM):** An abstraction layer built over public cloud APIs to standardize lifecycle operations (instance provisioning, termination, and health reporting) across multiple vendors.
- **Automated Maintenance Integration:** PCM translates vendor physical maintenance alerts directly into Meta's internal Maintenance Trains, eliminating manual coordination.
- **Automated Repair Reporting:** When internal Fleet Health detects a failing GPU node, PCM automatically relays diagnostic data to the cloud vendor to initiate hardware replacement.
- **Single-Engineer Implementation:** PCM uses strict interface contracts, allowing a single engineer to onboard a new cloud provider rather than requiring a dedicated operational team.
[17:54](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1074s) Standardizing Fleet Operations and the NPI Co-Design Process
Today, Meta operates its Private Cloud Infrastructure (PCM) seamlessly across multiple cloud providers, with all instances feeding into the exact same production capacity. The architecture is so tightly integrated that both internal operators and users view the entire infrastructure as "one fleet," turning public cloud capacity into just another standardized pool of hardware. However, as Meta has settled into this hybrid model, the ongoing challenge remains enabling new hardware—such as emerging GPUs and specialized AI accelerators—at scale. Meta relies on an extensive New Product Introduction (NPI) process that begins long before any hardware program is officially greenlit. Early in this phase, Meta sets strict functional specifications covering traditional metrics like CPU speeds and memory capacity, but also mandates granular requirements for debuggability, operability, and specific baseboard management controller (BMC) capabilities.
Historically, cloud vendors were unaccustomed to exposing such low-level control, typically offering limited APIs that hid the telemetry necessary to hook into systems like Meta's fleet health engine. To solve this, Meta took an unusual approach for a cloud customer: sitting down with cloud vendors during early ideation and design rather than after hardware SKUs were already built. By bringing their standard on-prem NPI checklist directly to cloud providers during the initial design phase, Meta enforces the same standards across the board, explicitly defining needs for "debuggability, observability, and back-end introspection." This high level of co-design ensures that new public cloud SKUs ship with all required telemetry and capabilities on day one, dramatically accelerating onboarding and ensuring day-one reliability.
* Meta's early-stage NPI process evaluates not just raw performance (CPU speeds, memory limits) but also low-level operability requirements like BMC behavior and controller access.
* Cloud provider APIs traditionally abstract away the granular hardware details required by Meta's automated fleet health and management tools.
* Co-designing hardware SKUs directly with cloud vendors ensures all necessary telemetry, debug capabilities, and back-end introspection are baked in from day one.
[19:50](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1190s) Cloud Integration Onboarding Metrics and the Vision for Hyperliquid Capacity
The public cloud has transitioned from an experimental buffer to a critical, rapidly expanding component of Meta's global infrastructure footprint. Through standardized tooling and early co-design, Meta has achieved industry-leading onboarding speeds for integrated public cloud infrastructure. A completely new cloud vendor can be integrated in less than a month, a new cloud region can be onboarded in about one week, and a new co-designed hardware SKU can be brought online in just two weeks.
However, Meta views this achievement as only the beginning. Having established a continuous "capacity flywheel" across networking, co-design, and fleet health for GPUs, the infrastructure team is extending this playbook to other types of compute, including general CPUs and non-GPU accelerators. The ultimate goal is to achieve true capacity hyperliquidity—a state where Meta can seamlessly tap into any type of compute capacity, anywhere in the world, at any time it is needed, once again transforming how hyperscale infrastructure is built and operated.
* **Vendor Onboarding Velocity**: Less than 1 month to integrate a new public cloud vendor.
* **Region Expansion Velocity**: Approximately 1 week to turn up a new cloud region.
* **SKU Onboarding Velocity**: Approximately 2 weeks to deploy a newly co-designed cloud hardware SKU.
* Meta is expanding this hyperliquid architecture beyond GPUs to encompass general-purpose CPUs and alternative AI accelerators.
## Notable quotes
> [01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) every day counts — Dinesh Govindaswamy
Faced with saturated on-premises GPU capacity during the late-2024 AI boom, Meta integrated public cloud infrastructure into its core fleet by standardizing orchestration, transport security, and vendor hardware co-design, according to engineering leaders Dinesh Govindaswamy and Sargun Dhillon.
Surging AI Demand and Initial Cloud Security Adaptation (1:43)
After years of operating solely on custom on-premises infrastructure optimized around custom systems like Tao, HHVM, and the Open Compute Project, Meta experienced severe GPU cluster saturation driven by frontier large language model development in late 2024. Because physical GPU-dense data centers require significant lead time to construct, leadership opted to lease external capacity, onboarding public cloud resources from a mid-December decision to training deployment in mid-February. Standardization around NVIDIA DGX SuperPOD architectures enabled rapid adoption by treating GPU compute as a uniform commodity across vendors.
Running proprietary model weights on untrusted cloud hardware required circumventing Meta's traditional reliance on physical security guards and hardware-rooted trust. The team adapted its Offnet services stack—originally developed to run edge appliances in hostile environments—treating the host hardware as compromised. Cloud instances booted into an immutable, Live CD-like operating system that performed hardware attestation before installing internal Meta OS components, routing all early traffic through an auditing Offnet proxy layer.
Bridging Observability and Lifecycle Gaps with Fleet Health and PCM (6:21)
Public cloud providers initially offered only coarse health telemetry, hiding low-level details such as CPU temperatures and shared back-end network metrics. Because multi-node GPU clusters fail if a single node degrades, Meta's Fleet Health and Network Production Engineering teams built custom diagnostic probes to regain real-time failure visibility, proactively evacuating workloads and signaling providers for hardware repairs.
To eliminate the manual coordination of hardware repairs across disparate cloud APIs, Meta developed Public Cloud Manager (PCM). PCM serves as an abstraction layer that translates vendor maintenance notifications directly into Meta's automated internal Maintenance Trains, which roll through the fleet every 45 days. This tool allows a single engineer to onboard a cloud vendor while automatically forwarding diagnostic data for failing hardware replacements.
Unifying Workloads via Twine and eBPF Transport Security (8:03)
To prevent cloud capacity from remaining an isolated, brittle silo for specialized AI teams, Meta migrated cloud nodes into TW Shared, its unified container orchestration pool. To prove the infrastructure could support legacy services, the team migrated "Web," Meta's monolithic PHP/HHVM service. Serving approximately 1% of live employee traffic from the cloud, Web validated that dependencies across internal services like Memcached, Tao, and Service Router could operate in third-party environments without specialized code modifications.
Because Web connects to hundreds of internal services, the manual firewall and audit processes required by the Offnet proxy became unworkable. Meta deployed Transparent TLS (TTLS), using host-level eBPF hooks to intercept outgoing transport-layer connections. Outbound traffic is transparently routed through a local sidecar proxy, encrypted into a mutual TLS tunnel, and sent to an on-premises reverse proxy, eliminating intermediate proxy bottlenecks while securing communication back to internal data centers.
Hardware Co-Design and Fleet Expansion Velocity (17:54)
To prevent recurring observability bottlenecks with emerging AI accelerators, Meta extended its early-stage New Product Introduction (NPI) process directly to cloud providers. Rather than selecting pre-built vendor SKUs, Meta works with cloud vendors during initial design phases to mandate operability requirements, baseboard management controller (BMC) capabilities, and granular back-end telemetry before hardware is deployed.
According to Govindaswamy and Dhillon, this standardization enables Meta to integrate a new cloud vendor in under a month, launch a cloud region in approximately one week, and onboard a newly co-designed hardware SKU in two weeks. Internal operators now manage this capacity as a unified fleet, with efforts underway to expand the hybrid orchestration model to general CPUs and non-GPU accelerators.
Meta aims to use this integrated hybrid framework to achieve what it terms 'capacity hyperliquidity'—the ability to dynamically consume heterogeneous compute across global providers. However, maintaining this abstraction across non-standardized hardware depends on continued vendor willingness to expose low-level controller telemetry and adopt collaborative co-design processes.
精炼阅读摘要
Meta 工程团队的 Dinesh Govindaswamy 与 Sargun Dhillon 指出，面对大语言模型爆发带来的 GPU 严重短缺，Meta 打破长期坚持的全自建硬件哲学，通过安全架构重构、运维自动化和早期硬件协同设计，将公有云无缝整合为统一的计算资源池。
算力危机倒逼架构转向与零信任落地 (1:43)
自 2000 年代中期以来，Meta 始终采用软硬件深度协同的自建基础设施模式，自主开发了 Tao、HHVM、Thrift、Hive 并发起开放计算项目（OCP）。然而到 2024 年末，前沿大语言模型的训练需求使得内部 GPU 集群彻底饱和，自建高密度数据中心耗时过长，管理层最终在 12 月中旬决定直接采购公有云算力，并在不到三个月内完成交付。公有云之所以能迅速落地，关键在于 NVIDIA DGX SuperPOD 参考架构实现了 GPU 拓扑的标准化，将 GPU 算力转变为类似通用商品的形态。
面对无法控制物理机房和底层固件的第三方环境，Meta 必须在两个月内解决专有模型权重的安全顾虑。工程团队复用了针对边缘不信任环境研发的 Offnet 服务栈：云端节点采用类似 Live CD 的不可变系统启动，在完成硬件认证后才安装 Meta OS 组件与 Twine 守护进程；所有进出流量均通过专用的 Offnet 代理进行审计与鉴权，禁止直接连通内网数据中心，在 60 天内实现了公有云环境的初步安全闭环。
突破单点代理瓶颈与承载传统单体架构 (8:03)
早期基于 Offnet 代理的方案需要安全团队人工审核每个下游服务并配置防火墙端口，这一流程在接入依赖关系极其繁复的旧系统时失效。为此，Meta 部署了基于 eBPF 技术的透明传输层安全协议（TTLS）。该方案在宿主机 Linux 内核层拦截出站连接，利用本地 Sidecar 代理自动完成双向 TLS 加密与身份凭证封装，再经由内部反向代理转发至目标服务，既避免了开发人员修改代码，又消除了集中式代理带来的单点故障与延迟负担。
为了验证云端平台的承载能力并打破团队间的信任壁垒，Meta 将全部云端资源并入统一容器编排池 TW Shared，并挑选内部最古老、最复杂的 PHP/HHVM 核心单体应用“Web”作为测试对象。Web 系统不仅承载核心业务逻辑，还广泛依赖 Memcached、Tao 及 Service Router 等下游基础组件。该验证最终成功让公有云承担了约 1% 的 Meta 内部员工实时生产流量，证明了复杂遗留服务无需修改代码即可在公有云稳定运行，确立了跨云架构的普适性。
全生命周期抽象与跨云舰队健康治理 (14:31)
相比于具备完善热迁移能力的 CPU，多节点 GPU 集群更接近紧密耦合的超算，单个节点的微小故障即可导致整个分布式训练任务崩溃。由于公有云平台屏蔽了底层遥测数据（如 CPU 温度与后端共享网络状态），且缺乏对上层业务的感知，Meta 将其内部的 Fleet Health 遥测系统和定制诊断探测器延伸至云端，以实时捕获退化节点并主动隔离工作负载。
在运维协同方面，Meta 针对以往依赖 Slack、Jira 和服务工单的繁琐人工沟通，研发了公有云管理器（PCM）。PCM 作为统一的 API 抽象层，将各大云厂商的实例生命周期接口标准化，并将其物理维护通知自动对接到内部每 45 天轮换一次的“维护列车”（Maintenance Trains）中，实现了受损主机的自动驱逐、镜像解绑与厂商工单报修。该抽象层严格规范了接口协议，使单个工程师即可完成新云厂商的基础生命周期对接。
早期协同设计与极速入网指标 (17:54)
为了让公有云实例完全达到内部“单一舰队”的运行标准，Meta 改变了传统云租户的被动验收模式，将自建数据中心的新产品导入（NPI）流程前置到与云厂商的早期设计阶段。Meta 不仅严格规定了 CPU 频率和内存容量等基础性能，还对基板管理控制器（BMC）能力、调试接口以及底层观测性提出了强制要求，确保云厂商交付的新硬件在上线首日即完整内嵌所需的遥测与内省能力。
得益于标准化的工具链与前置协同设计，Meta 显著压缩了云资源的交付周期：接入一家全新的公有云服务商可在 1 个月内完成，开辟一个全新的云可用区约需 1 周，而部署一款经协同设计的新硬件 SKU 仅需约 2 周。目前，公有云已从临时性的缓冲资源转变为核心基础设施的关键组成部分。
Meta 目前正将其基于 GPU 验证的组网、硬件协同设计与健康管理飞轮推广至通用 CPU 以及其他专用 AI 加速芯片。团队的长远目标是构建一种“高流动性算力”（Capacity Hyperliquidity）架构，实现在全球范围内随时按需调用任意类型的计算能力；但如何在维持统一控制平面的同时，应对更多元异构芯片与更复杂的跨云网络环境，仍是持续演进中的工程挑战。
Detailed notes — zh
## Background
本视频由技术大会官方频道 @Scale 于 2026 年 7 月 2 日发布在 YouTube 平台。内容记录了 Meta 计算基础设施工程总监 Dinesh Govindaswamy 与工程师 Sargun Dhillon 的联合演讲，深入剖析了 Meta 因生成式 AI 带来的庞大 GPU 需求而打破十余年全栈自建传统、拥抱多云架构，并将其无缝整合入自身平台、操作系统与舰队管理体系以打造“算力超流动性”的完整历程。
## TL;DR
面对 2024 年底爆发的生成式 AI 算力危机，Meta 在不到 90 天内完成了首批公有云 GPU 算力的采购与投产，并依托 Offnet 边缘栈攻克了公有云零信任环境下的模型资产安全与黑盒硬件健康度治理难题。为了彻底终结烟囱式特化适配，团队将云端容量整体并轨入统一调度资源池 TW Shared，并研发基于 eBPF 的透明传输层安全（TTLS）架构，成功让全公司依赖最复杂的 Web 核心工作负载透明上云。随后，Meta 打造了屏蔽多云差异并联动内部 45 天例行维护的公有云管理器（PCM），甚至反向介入云厂商早期的 NPI 软硬件协同设计，最终实现了跨多云与本地机房即取即用的算力超高流动性（Capacity Hyper-liquidity）。
## Atmosphere & style
整场演讲节奏紧凑、干货极度密集，洋溢着顶级互联网大厂顶尖系统工程师的务实与硬核锋芒。Dinesh 负责高屋建瓴地复盘战略抉择与体系演变，Sargun 则细致拆解 eBPF、硬件远程证明与维护状态机等底层工程黑科技，两人的无缝配合营造出极强的沉浸感与技术说服力。在叙事上，从最初面对 AI 海啸的极限求生，到敢于挑选最脏最难的 Web 依赖作为试金石，再到反向渗透重塑云厂商芯片设计，整场分享宛如一部波澜壮阔的工程攻坚史诗，充满推倒重来、追求极致架构纯粹性的技术气魄。
## Bulletin
- [01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) 面对 2024 年底大模型浪潮引发的极端 GPU 算力饥渴，坚守十余年全栈自建的 Meta 仅用不到 90 天便完成了公有云 GPU 算力的采购与业务交付上线。
- [03:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=211s) Meta 将原用于不安全外部机房的 Offnet 边缘栈移植至云端，通过不可变操作系统、硬件远程证明和反向代理架构，在 60 天内构建出公有云零信任安全基座。
- [11:54](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=714s) 通过引入透明传输层安全（TTLS）及 eBPF 拦截，云上容器无需改动代码即可透明访问核心私有数据中心，成功将最复杂的 Web 工作负载约 1% 员工流量平移至公有云。
- [14:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=871s) 自研的公有云管理器（PCM）抽象屏蔽了多云底层差异，不仅实现了故障自动报修，还将公有云物理维护无缝融入 Meta 内部 45 天周期的“维护列车”调度中。
- [18:10](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1090s) Meta 深入云厂商 NPI 早期设计阶段推行软硬件协同设计，目前接入一家全新公有云仅需不到 1 个月，交付全新可用区仅需约 1 周。
## Key takeaways
- [00:01](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1s) Meta 历史上坚持软硬件全栈自研与 OCP 协同设计，曾因极高的运行效率和定制化优势长期排斥通用公有云。
- [01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) 生成式 AI 与 LLM 爆发造成巨大算力断崖，自建 GPU 机房周期无法满足急迫需求，迫使 Meta 破例直接采购公有云算力。
- [01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) 英伟达 DGX SuperPOD 参考架构将高端 GPU 集群转变为标准化大宗商品，消除了硬件异构壁垒，奠定了极速云端交付基础。
- [03:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=211s) 针对公有云租户环境物理不可控的挑战，Meta 采用 Offnet 边缘技术栈，依托只读不可变操作系统与硬件远程证明构建零信任防线。
- [06:21](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=381s) 云厂商的黑盒监控无法满足大模型分布式训练对亚健康的敏感要求，Meta 通过网络和舰队健康团队研发的软性规避机制实现了异常节点自动隔离与报修。
- [08:03](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=483s) 为了终结按项目打补丁的烟囱式接入，Meta 推动公有云全面并轨 Twine Shared 统一调度池，追求业务免审批、无感知的泛在运行体验。
- [10:34](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=634s) Meta 选择内部架构最古老、依赖最复杂的 Web 工作负载作为试金石，以彻底扫清全量服务的云端接入障碍并树立技术公信力。
- [11:54](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=714s) 传统基于防火墙开孔的安全方案无法适应动态漂移的庞大依赖，Meta 基于 eBPF 与 Sidecar 研发了透明传输层安全（TTLS）以实现无感知安全互联。
- [14:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=871s) GPU 训练的超算属性对底层维护极度敏感，早期公有云上依赖工单与电话的人肉运维难以支撑多云战略扩张。
- [14:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=871s) 公有云管理器（PCM）作为大一统控制面，将公有云底层维护信号转化为内部标准事件，平滑对齐 Meta 既有的 45 天“维护列车”自动化轮转。
- [14:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=871s) PCM 基于标准化接口契约构建，仅需一名工程师即可完成新云厂商适配，为上层呈现统一步调的单一舰队（One Fleet）视图。
- [18:10](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1090s) Meta 将自建机房的 NPI 规范前置引入云厂商研发初期，通过深度软硬件协同设计确保新规格自第一天起具备必要的可观测与可调试性。
- [18:10](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1090s) 从硬件同构、调度并轨到深度协同设计，Meta 正在构建算力超高流动性（Capacity Hyper-liquidity）架构，实现异构基础设施的即取即用。
## Detailed walkthrough
### [00:01](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1s) 坚守十余年的纯自建栈与“软硬件协同设计”飞轮
Meta 计算基础设施工程总监 Dinesh Govindaswamy 与工程师 Sargun Dhillon 共同登台，分享了一个充满戏剧性转折的基础设施演进历程：一家以构建了全球最先进自建数据中心（On-Prem Infrastructure）闻名的科技巨头，究竟是如何一步步迈向公有云的。回顾历史，Meta 在 2000 年代中期凭借单个网站 facebook.com 起步，历经多年高速扩张，到了 2010 年代初便已成为全球规模最庞大的网络站点之一。随着业务版图从单一站点扩展到庞大的应用矩阵，后端服务器面临的挑战始终未变——海量的数据服务、极端密集的交互式服务以及无休止的 API 调用。面对指数级膨胀的流量，市面上的现成方案根本无法支撑，Meta 被迫全面走向底层自主研发，先后孕育了社交图谱核心 Tao、专为 PHP 执行效率极致优化的 HipHop 虚拟机（HHVM）、高性能 RPC 框架 Thrift 以及分布式 OLAP 数据仓库 Hive；硬件层面更是打破常规，通过发起开放计算项目（Open Compute Project, OCP）直接设计并建造属于自己的超大规模数据中心。
在整个科技行业几乎全员拥抱公有云的大浪潮下，Meta 却反其道而行之，选择坚持全栈自建。这种选择的核心底气源于“协同设计”（Co-Design）哲学——当一家公司同时完全掌控软件和硬件时，就能实现软硬件层面的深度契合与联合调优。Meta 的数据中心架构完全是为自身特有工作负载量身定制的，而上层软件栈的设计也紧密结合了数据中心物理特性的反馈。“我们无需去迁就任何通用的基础设施架构，”Dinesh 指出，自研软硬件构成了一个高速运转的迭代飞轮，确保了极高的运行效率与基础设施可靠性。正因这套自建飞轮运行得异常顺畅，公有云在很长一段时间里根本入不了 Meta 基础设施团队的法眼。
### [01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) 大模型海啸与向公有云闪电采购的生死时速
然而，进入 2024 年底，AI 领域掀起了一场极其猛烈的技术巨浪，以大语言模型（LLM）为代表的前沿架构实现了突破性跃升。Meta 的研发团队敏锐地意识到必须将这些前沿技术引入到自身全线模型体系中。但现实极为残酷：这些模型飞跃需要成倍增长的底层存储、通用算力，尤其是极为稀缺的 GPU 算力。当时 Meta 内部虽坐拥海量的 CPU 算力和庞大存储，但集群内原有的 GPU 早已被各项业务彻底打满，处于极限过载运转状态。众所周知，建造配备海量电力供应与专用液冷设施的下一代 GPU 数据中心绝非一朝一夕之功。面对巨大的算力断崖，Meta 做出了一个打破历史惯例的重大抉择：直接向公有云合作伙伴采买 GPU 算力容量，由此拉开了 Meta 现代意义上拥抱公有云的大幕。
整个推进过程是一场与时间的生死竞速。团队在 12 月中旬拍板决定采购云端算力，到了次年 2 月中旬，就已经把可运行的算力资源正式交付给内部的生成式 AI 合作伙伴团队，端到端耗时不到三个月。Dinesh 强调，在激烈的大模型军备竞赛中，"每一天都至关重要（In an AI race, every day counts）"——GPU 闲置一天，不仅意味着数以万计的算力采购资金在凭空流失，更意味着模型训练和迭代进度正在被同行无情拉开差距。公有云之所以能在这次危机中力挽狂澜，很大程度上得益于英伟达（NVIDIA）重塑了行业格局：其推出的 DGX SuperPOD 参考架构将高端 GPU 集群转变为一种标准化大宗商品（commodity）。各大云厂商均遵循完全一致的拓扑结构与硬件规格进行交付，而 Meta 内部自建的 GPU 架构恰好与该标准高度趋同，为快速适配奠定了硬件物理基础。
- **极速交付周期**：从 12 月中旬做出决策，到 2 月中旬交付生成式 AI 业务团队，全流程仅历时不到 90 天，首批测试实例部署更是在 60 天内全部完成。
- **硬件同构利好**：英伟达 DGX SuperPOD 统一了公有云与私有云在 GPU 拓扑、NVLink 互联和节点规格上的标准，消除了硬件异构带来的底层阻碍。
### [03:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=211s) 零信任环境破局：Offnet 边缘栈与云端安全移植
尽管硬件架构相似，但将云算力真正接入 Meta 生产环境却绝非易事。Sargun Dhillon 随后登台阐述了背后的攻坚战。首当其冲的难题是安全底线：团队要在公有云上加载 Meta 核心大模型的未公开权重（model weights），运行深度绑定的 Meta 内部敏感软件，同时还必须保证极致的稳定性，因为分布式训练中途崩溃将导致长达数天的进度损失。Meta 本地数据中心拥有一套极其严密的纵深安全防御体系：底层由持枪荷枪实弹的安保人员守护物理机房，往上是对物理访问和访问控制列表（ACL）层层设防的网络安全层，再往上是基于端到端密码学真实性与双向认证的传输安全层，这一整套逻辑全部锚定在硬件底层的密码学硬件信任根（Hardware Root of Trust）上，并在深度加固、防御零日漏洞的定制操作系统 Meta OS 上运行。然而公有云的租户环境在物理和底层网络层面完全不受 Meta 控制，传统安全防线全部失效。
面对这一僵局，团队打出了手中珍藏的一张王牌——Meta 的 Offnet 边缘服务技术栈（Offnet Services Stack）。在此之前，Meta 除了在自有大型数据中心跑业务外，还在全球各地运营商网络节点部署了被称为 Meta Network Appliances 的边缘设备。这些边缘设备中有一部分恰恰部署在外部环境极为险恶、物理安全性无法受控的第三方机房中。Offnet 栈生来就是为了应对“对物理环境零信任”的极端场景：它从类似 Live CD 的不可变只读操作系统镜像（Immutable OS）启动宿主机，通过远程硬件证明（Hardware Attestation）验证硬件未被物理植入恶意篡改，确认可信后才在本地动态安装 Meta 专用系统组件、Meta OS 守护进程与 Twine 调度服务；更关键的是，这套云端节点绝不直接回连 Meta 核心第一方数据中心，而是全部强制经由 Offnet Services Proxy（反向代理集群）进行通信汇聚、强身份验证与全面流量审计。借助与 Offnet 团队的跨部门协作，团队在短短 60 天内就让公有云实例具备了运行 Meta 应用的安全资质。
```mermaid
flowchart TD
accTitle: Offnet Services Architecture in Cloud
accDescr: 展现从不可变镜像启动、硬件远程证明、本地环境装配，再到通过 Offnet 代理受控接入第一方核心机房的全流程。
subgraph CloudHost ["公有云算力宿主机 (零信任物理环境)"]
A["只读不可变操作系统<br/>(Live CD / Immutable OS)"] --> B["硬件远程证明<br/>(Hardware Attestation 校验)"]
B --> C["本地环境部署与装配<br/>(安装 Meta OS、TW Shared 守护进程)"]
C --> D["大模型分布式训练工作负载<br/>(加载敏感模型权重)"]
end
subgraph NetworkBoundary ["安全边界收敛层"]
E["Offnet Services Proxy<br/>(流量审计 / 统一认证 / 边界隔离)"]
end
subgraph InternalDC ["Meta 第一方核心数据中心"]
F["第一方控制面与核心数据服务<br/>(Caches / Data Stores / Security APIs)"]
end
D -->|"禁止直连机房网络"| E
E -->|"双向认证加密回连"| F
```
### [06:21](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=381s) 可靠性暗礁：穿透黑盒的云端舰队健康度治理
当算力跑起来后，团队迎头撞上了第二道严峻难关——公有云不可控的硬件可靠性。内部业务方习惯了 Meta 本地机房那种为自家应用高度优化、故障特征高度透明的基础设施，而云厂商与企业客户之间长期保持着“臂长原则”（Arms-length Engagement，即边界分明的冷淡隔离状态）。云厂商缺乏对客户上层应用特征的深度观测视角，往往只能依赖极其粗粒度的宏观监控判断物理机状态。相反，Meta 在内部建立了一套成熟的舰队健康度监控系统（Fleet Health），每天吞吐数以百万计的低级硬件遥测信号，通过多维模式识别算法实时判断机器是完全健康、亚健康（Partially Healthy）还是濒临损坏，并驱动调度系统自动将业务平滑驱逐。
在公有云租户模式下，很多底层关键数据对 Meta 完全呈现为黑盒——由于硬件虚拟化与资源共享，Meta 无法直接读取 CPU 的物理核心温度，更无法获取后端交换网络底层的丢包和链路统计数据。为了防止硬件暗病导致大模型训练无故停摆，Meta 派出了 Fleet Health 团队和网络生产工程（Network PE）团队进驻攻坚，针对云环境专门研发了一系列创新的软性规避机制与指标探测工作流（workarounds）。这套方案成功重建了针对云端节点的亚健康检测能力，能够在 GPU 集群发生静默计算错误或网络退化时提前捕捉异常，自动隔离故障节点并精准向云厂商发起退换机维修工单，保障了数千卡规模分布式训练作业的平稳延续。
### [08:03](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=483s) 告别烟囱式适配：统一接入 Twine Shared 调度资源池
度过了最初的救急阶段后，工程负责人 Dinesh 重新接管话题，指出当时的基础设施架构正陷入一个不可持续的循环陷阱。每当有新的机器学习团队申请使用云上算力时，底层平台团队就必须专门坐下来跟他们反复开会沟通需求。因为 Meta 内部的 ML 工作负载极少是自包含的孤立程序，而是高度依赖内部复杂的共享缓存、持久化数据存储体系以及细粒度的安全隐私控制逻辑。在日常研发中，Meta 的工程师（Metamates）早已习惯了将训练任务直接提交给统一调度层，默认底层存在取之不尽的泛在算力，且周边所有依赖微服务均开箱即用。每当算法工程师（MLE）因为云端缺少某个内部微服务依赖而导致训练任务崩溃报错时，都会大幅蚕食云端算力在公司内部的技术声誉。
为了终结这种“按项目打补丁”的烟囱式接入模式，团队做出了两项决定性的战略抉择：
1. **废弃定制配置，全面并轨 TW Shared**：TW Shared 指的是 Twine Shared。Twine 是 Meta 全局核心的容器编排中枢，而 TW Shared 则是全公司所有标准应用共享的基础算力资源池。团队决定彻底消除云端专有特化配置，让云端节点成为 TW Shared 池中的标准公民。
2. **免审批运行全量工作负载**：确立极简体验目标，即无需漫长繁琐的专门安全与架构审计，任何工作负载都能够直接且透明地无缝运行在公有云之上。
这项变革背后的核心洞见在于：只要让云端容量的表现与本地数据中心的 TW Shared 做到毫无二致，Meta 内部任意一名熟悉标准提交流程的工程师，就能够在完全不修改代码、不进行专门培训、甚至根本无需公有云平台团队介入的情况下，零门槛将业务直接甩到云端运行。然而，将 TW Shared 原封不动搬上云端需要跨越巨大的架构鸿沟，因为 TW Shared 内部暗含了对本地网络扁平拓扑、全套周边服务可达性以及内部控制面连通性的强假设。平台团队必须在 Offnet 代理与安全隧道之上完整模拟并穿透这一套环境，同时必须对开发者严格维持原本的 API 契约不变。为了验证这一宏大构想并一举攻破最复杂的内部依赖，团队挑选了整个 Meta 历史最悠久、架构最复杂、承载流量最密集的“终极试金石”工作负载——Web 服务。
### [10:34](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=634s) 攻坚最难堡垒：为什么将 Web 工作负载迁移上云？
面对将 Meta 基础设施向公有云扩展的挑战，团队没有选择容易的轻量任务，而是直接把目标对准了全公司最古老、依赖关系最错综复杂、积累了多年技术债且充满未知数（unknown unknowns）的业务——Web。正如讲者所言：“我们选择的不是一条容易的路，而是选择了一条能证明我们真正做成了的路。”最终，Meta 成功让约 1% 的员工流量完全由公有云算力提供服务。
选择啃下 Web 这块硬骨头背后有三个极具说服力的战略考量：
1. **基石地位（Foundational Role）**：Web 服务是 Meta 绝大部分核心产品与子业务逻辑的运行之所。计划上云的几乎所有云原生工作负载，最终都要和 Web 频繁交互。若能将 Web 直接协同部署（collocate）在云端，不仅能大幅拉低端到端延迟，更能最大限度减少昂贵的跨地域网络流量（cross-region traffic）。
2. **包罗万象的依赖图谱（Dependency Graph）**：Web 服务是全公司庞杂后端设施的交汇枢纽。它需要连接 Memcached、TAO（分布式图数据存储）、配置推送系统（Configurators）、Service Router，以及数以打计的各类数据服务、缓存层与隐私合规系统。团队意识到，如果能让 Web 这张庞大杂乱的依赖网络在公有云中顺畅跑通，实际上就一劳永逸地为全公司所有其他工作负载扫清了障碍。
3. **技术公信力（Credibility）**：在此之前，每当机器学习工程师（MLE）因为某个云端缺失的底层依赖而遭遇任务失败时，基础设施团队在内部建立的信任就会被侵蚀一次。Web 服务正是证明团队技术底气的“试金石”——用老话来说，这就是“如果你能在这里立足，你就能在任何地方立足”（"if you can make it here, you can make it anywhere"）的代表性时刻。
---
### [11:54](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=714s) 传统边界安全的破局：透明传输层安全（TTLS）
在解决依赖接入时，团队撞上了 Meta 既有安全架构的墙壁。在 Meta 的内部安全模型中，任何公有云主机都被默认判定为不可信节点（untrusted）。所有在公有云和 Meta 自建私有数据中心（First-party DC）之间流动的流量，都必须穿过受严格防火墙规则把守的网外服务栈（offnet services stack）。这意味着，云端容器每需要连接一个内部新服务，工程师就得经历一套繁重的审批流程：梳理具体依赖、提交并申请开放防火墙端口、接受安全团队严格审计、最终批准生效。
在初期的 AI 训练场景下，工作负载高度受限，依赖项寥寥无几，人工开几个端口（punched holes）还能勉强应付。但 Web 服务的后端依赖成百上千，不仅伴随着海量的历史黑盒代码，更要命的是，随着 Web 代码高频持续地快速推版（push），其下属依赖关系每天都在动态漂移。靠人工逐个服务开端口、配白名单的传统安全方案彻底失效。
为了打破这个死结，团队引入了 Meta 内部构建的黑科技架构——透明 TLS（TTLS，Transparent TLS）。TTLS 借助主机上的 eBPF 钩子技术，在传输层直接透明拦截云主机发出的所有出向流量，无需业务应用修改任何一行代码，也无需手工重新配置网络拓扑：
```mermaid
flowchart LR
accTitle: TTLS 工作流架构图
accDescr: 展示从公有云容器发起的请求，通过云端宿主机的 eBPF 拦截与 TTLS 代理隧道加密，最终到达 Meta 私有数据中心反向代理并转发至后端服务的全流程。
subgraph Cloud["公有云端 (Cloud Host)"]
direction TB
C["业务容器 (Container)"]
E["eBPF 传输层拦截"]
P["TTLS 代理 (Sidecar Proxy)<br/>TLS 加密 / 凭证证明"]
C -->|出向流量| E
E -->|透明重定向| P
end
subgraph Direct["专线传输"]
Tunnel["双向 mTLS 隧道<br/>(安全认证与加密)"]
end
subgraph DC["Meta 私有数据中心 (Meta 1st-Party DC)"]
direction TB
RP["反向代理 (Reverse Proxy)<br/>解密隧道"]
S["后端服务 (Memcached / TAO 等)"]
RP -->|明文转发| S
end
P -->|安全隧道| Tunnel
Tunnel -->|到达 DC| RP
```
这套设计的核心精髓在于“透明”（transparent）：
- **业务无感知**：业务服务压根不知道自己正跑在公有云里，研发人员完全不必改动任何既有逻辑代码。
- **主机无损集成**：代理进程以 Sidecar 模式常驻于每台服务器节点上，消除了集群级集中代理的单点故障风险（SPOF）。
- **零额外跳点损耗**：由于运行在整机群的每台宿主机内，它彻底剔除了常规中间层网络网关（intermediate proxy hop）带来的高延迟抖动与可靠性惩罚。
TTLS 的落地彻底激活并释放了混合云能力。随着 2025 年全球 AI 算力大战全面升级，Meta 对算力的渴求呈现指数级爆发，公有云顺理成章地再次成为了承载核心负载的关键后盾。
---
### [14:31](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=871s) 破除“超算死锁”与维护泥潭：公有云管理器（PCM）的诞生
接替 Dinesh 发言的 Sargun Dhillon 尖锐地指出了当前算力基础设施的断层。过去数十年间，公有云厂商已将 CPU 算力的运维磨炼得极其成熟，硬件故障和系统维护对上层使用者几乎全透明，虚机热迁移（live migration）技术更是炉火纯青。但 GPU 彻底打碎了这一舒适区——GPU 从来不是孤立单机，其运作模式更贴近于紧密集成的超级计算机（supercomputer）。在分布式训练拓扑中，哪怕单颗 GPU 出现硬件异常，都可能直接导致整个庞大的分布式训练作业崩溃。这导致公有云上的 GPU 软硬件运维工作流极其脆弱复杂。
在 Meta 内部，数据中心早已拥有一套高度成熟、与外部工业界截然不同的自动化机制——“维护列车”（Maintenance Trains）：
- **45 天例行轮转**：每隔 45 天，一趟维护列车就会有序开过整个机群，自动化地下线服务器、注销操作系统、完成物理检修、推行硬件固件升级并重新交付开机。
- **北向感知机制**：列车完全与集群调度器 Twine 深度联动。上层业务容器能提前收到北向预警信号（northbound signaling），在维护前主动迁离（decamp）至备用冗余算力，整个过程对业务用户几乎毫无打扰。
然而在早期的公有云上，这一流程却退化成了落后的“人肉运维”。Meta 工程师不得不守着 Slack、Jira、邮件甚至是云厂商五花八门的定制工单系统，手工跟各家云厂商的技术支持进行电话和工单拉锯协调。随着 Meta 逐步落地多云战略（multi-cloud），不仅可靠性问题频频爆发，团队也根本不可能为每家云厂商单独配置一组全职驻场运维团队。Meta 必须拥有一套大一统的标准化抽象，公有云管理器（Public Cloud Manager, PCM）应运而生。
PCM 在底层公有云原始 API 之上抽象出了一层标准控制面，将裸机云实例创建、物理机维护、实例销毁等操作完全收敛：
```mermaid
flowchart TD
accTitle: 公有云管理器 (PCM) 架构与协同流
accDescr: 展示公有云管理器如何作为抽象契约层，向下屏蔽多云厂商 API 差异并处理物理维护，向上与 Meta 内部健康监控与调度引擎 Twine 无缝对接。
subgraph MetaInternal["Meta 既有内部系统"]
FH["Fleet Health<br/>(故障探测引擎)"]
Twine["Twine 调度系统<br/>+ 维护列车 (Maintenance Trains)"]
end
subgraph PCM_Layer["公共云管理器 (PCM)"]
direction TB
API["标准化接口契约 (Interface Contracts)"]
Scheduler["维护转换引擎<br/>(公有云维护 ➔ 调度入维护列车)"]
Reporter["故障自动化报修管道"]
API --- Scheduler
API --- Reporter
end
subgraph Clouds["异构公有云提供商"]
C1["云厂商 A (AWS / Azure / GCP 等)"]
C2["云厂商 B"]
C3["云厂商 N"]
end
FH -->|上报硬件故障| Reporter
Reporter -->|全自动工单/API 维修| Clouds
Clouds -->|物理变更通告| Scheduler
Scheduler -->|转化为标准事件接入| Twine
PCM_Layer -->|裸机 GPU 转内部统一可用算力| Twine
```
- **维护流程无缝接轨**：PCM 捕获公有云厂商发出的底层物理维护计划，自动将其编译并注入到 Meta 内部的 45 天维护列车日程中，彻底消灭了 Slack 人工盯梢。
- **智能闭环修复**：当 Fleet Health 探针检测到云上 GPU 出现掉卡或性能异常时，PCM 会自动向云厂商 API 报告详尽的排查信息并触发后台硬件更换。
- **超高工程杠杆**：PCM 核心基于一组清晰的接口契约（interface contracts）构建。接入一家全新的公有云服务商，仅需一名工程师去实现这套契约代码即可，而无需动用一整个工程团队。
- **同一舰队视角（One Fleet）**：所有公有云算力均被纳管为标准生产算力。对于 Meta 的运维人员和业务算法工程师而言，无论底层算力分布在哪个提供商，在控制台和调度屏上看到的全都是统一步调的单一 Fleet。
---
### [18:10](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1090s) 深入造芯源头：NPI 早期软硬件协同设计与算力超流动
随着算力环境步入正轨，团队攻克的最后一道硬核关隘是硬件的极速使能。新的 GPU 和各类 AI 加速芯片层出不穷，Meta 内部拥有极其严苛的 NPI（New Product Introduction，新产品导入）流程，在立项之前就会对 CPU 频率、内存带宽、基板管理控制器（BMC）规范与可维护控制器指标提出极其详尽的规范。
传统模式下，公有云厂商只对外开放极其有限、黑盒化的管理 API，完全不暴露底层 Fleet Health 等基础设施所必需的白盒可观测与深层硬件遥测接口。为此，Meta 做了一件对云租户而言极不寻常的颠覆性动作：直接“反向渗透”到云厂商的研发上游。在云厂商刚开始构想（ideation）和早期架构设计的阶段，Meta 团队就直接坐进了他们的设计会议室。
Meta 将自建私有机房使用的全套 NPI 硬件检查清单原封不动地搬给了云厂商的芯片与服务器设计团队，当面提出明确诉求：“这是我们需要的可调试性（debuggability）、这是我们必须掌握的可观测性（observability），这是我们做故障诊断必需的底层自省能力（introspection）。”这种在工业界极为罕见的软硬件协同设计（co-design），确保了公有云厂商产出的全新定制规格在交付第一天（Day 1）就直接具备 Meta 所需的深度透视能力，彻底扫除了上线维护的隐患。
- **闪电级多云交付速度**：
- 接入一家全新公有云提供商：**不到 1 个月**
- 交付上线一个全新的云可用区（Cloud Region）：**约 1 周**
- 接入一款完成协同设计的新型云端 GPU/硬件 SKU：**约 2 周**
在演说尾声，Sargun 展望道，从网络打通、深度协同设计到 Fleet Health 集成，Meta 已经彻底转动了公有云算力飞轮。如今这套体系正向着更多新型专用加速芯片、更广泛的 CPU 资源池化持续外溢。Meta 正在构建一个“算力超高流动性”（Capacity Hyper-liquidity）的世界——无论何时、无论身处何地，任何形态的基础设施算力都能如活水般随时即取即用，再次引领整个数据中心与云计算产业的革新演进。
## Notable quotes
> [00:01](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1s) 我们无需去迁就任何通用的基础设施架构 — Dinesh Govindaswamy
> [01:43](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=103s) 每一天都至关重要（In an AI race, every day counts） — Dinesh Govindaswamy
> [10:34](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=634s) 我们选择的不是一条容易的路，而是选择了一条能证明我们真正做成了的路。 — Dinesh Govindaswamy
> [10:34](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=634s) 如果你能在这里立足，你就能在任何地方立足（if you can make it here, you can make it anywhere） — Dinesh Govindaswamy
> [18:10](https://www.youtube.com/watch?v=p08-ef0QKYQ&t=1090s) 这是我们需要的可调试性（debuggability）、这是我们必须掌握的可观测性（observability），这是我们做故障诊断必需的底层自省能力（introspection）。 — Sargun Dhillon
## Mentioned
Meta, Dinesh Govindaswamy, Sargun Dhillon, Tao, HipHop Virtual Machine (HHVM), Thrift, Apache Hive, Open Compute Project (OCP), NVIDIA, DGX SuperPOD, Offnet Services Stack, Offnet Services Proxy, Meta Network Appliances, Meta OS, Twine, Twine Shared (TW Shared), Fleet Health, Network Production Engineering (Network PE), Transparent TLS (TTLS), eBPF, Public Cloud Manager (PCM), Maintenance Trains, New Product Introduction (NPI), AWS, Azure, Google Cloud Platform (GCP), Slack, Jira, Memcached, Service Router
## Video description
Meta has a massive amount of internal infrastructure, but the demand for GPU compute led us to look to the cloud to grow our footprint. This led to our initial forays into cloud, and what it took to enable and integrate into Meta’s overall infrastructure footprint.
In order to create capacity hyperliquidity, we’ve leveraged multi-cloud as a form of optionality. Come learn how we’ve fully integrated cloud into our platform, OS, and fleet management stack to make the experience ubiquitous for our internal customers.
Learn more about the @Scale conferences here: atscaleconference.com
[Source](https://www.youtube.com/watch?v=p08-ef0QKYQ)
Source quotes
[0:01] Dinesh Govindaswamy: 我们无需去迁就任何通用的基础设施架构
[1:43] Dinesh Govindaswamy: 每一天都至关重要（In an AI race, every day counts）
[10:34] Dinesh Govindaswamy: 我们选择的不是一条容易的路，而是选择了一条能证明我们真正做成了的路。
[10:34] Dinesh Govindaswamy: 如果你能在这里立足，你就能在任何地方立足（if you can make it here, you can make it anywhere）
[18:10] Sargun Dhillon: 这是我们需要的可调试性（debuggability）、这是我们必须掌握的可观测性（observability），这是我们做故障诊断必需的底层自省能力（introspection）。
Source quotes
[1:43] Dinesh Govindaswamy: every day counts
[8:03] Dinesh Govindaswamy: eroded trust
[8:03] Dinesh Govindaswamy: capacity hyperliquidity
[8:03] Dinesh Govindaswamy: if you can make it here, you can make it anywhere
[6:21] Sargun Dhillon: Fleet Health
[3:31] Sargun Dhillon: Offnet services stack
[14:31] Sargun Dhillon: Maintenance Trains
[17:54] Dinesh Govindaswamy: one fleet
[17:54] Sargun Dhillon: debuggability, observability, and back-end introspection
Original transcript
[0:01] Good morning, everyone.
[0:03] I'm Dinesh Govindaswamy, director of
[0:05] engineering supporting the computer
[0:06] infra organization at Meta, and this is
[0:08] Sargun.
[0:12] Together, we are going to talk about how
[0:14] Meta, the company that has built the
[0:15] most advanced on-prem infrastructure,
[0:18] extended it to public cloud.
[0:21] Meta started in mid-2000s with a
[0:23] website, facebook.com. We ran it very
[0:26] successfully for years.
[0:28] By the early 2010, it was one of the
[0:30] biggest websites in the world.
[0:32] Then Meta started investing in
[0:33] applications.
[0:35] But on the server side, the story
[0:37] remained the same. Lots of data
[0:39] services, lots of interactive serving
[0:41] and APIs.
[0:44] We were growing so so so fast that we
[0:46] have to build our own tech.
[0:48] The Tao, the heart of our social graph,
[0:51] a specialized PHP runtime designed for
[0:54] efficiency, the HipHop Virtual Machine,
[0:56] we call it HHVM,
[0:58] a sophisticated RPC system, Thrift, the
[1:01] OLAP engine, Hive,
[1:03] and we even built our own data centers
[1:04] through Open Compute.
[1:07] We So, we built the entire stack. While
[1:09] our industry colleagues have been
[1:11] choosing to use public cloud, we chose
[1:13] to build our own.
[1:16] When you own both software and hardware,
[1:18] you can optimize them together. We call
[1:20] this process co-design.
[1:23] Our data centers are built for our
[1:24] workloads. Our software stack is shaped
[1:27] by the data center design.
[1:29] We didn't adapt to any generic
[1:30] infrastructure, and it's a rapid
[1:32] iteration loop. We build exactly what we
[1:35] need.
[1:36] This flywheel is what makes us efficient
[1:39] and reliable.
[1:40] And we ignored public cloud for many
[1:42] years.
[1:43] But something wild happened.
[1:46] In late 2024, a massive There was a
[1:48] massive leap in the model space,
[1:50] especially around LLMs.
[1:53] We realized we could also incorporate
[1:55] these techniques into models into our
[1:56] models as well.
[1:58] Unfortunately, some of these
[2:00] advancements needed more storage, more
[2:03] compute, and more GPUs.
[2:06] We had massive amount of CPUs and
[2:08] storage, but our GPUs were really busy.
[2:11] And we all know that we cannot build
[2:13] data centers with GPU overnight.
[2:15] So, we made a different decision. We
[2:17] decided to acquire capacity through our
[2:19] cloud partners, and that started our
[2:21] modern journey to cloud.
[2:25] This slide shows the the timeline
[2:28] through which we onboarded our first
[2:29] cloud, and we handed off our capacity to
[2:32] our journey AI partners.
[2:34] We made the decision to acquire capacity
[2:36] by mid-December, by and by mid-February
[2:38] we handed off the capacity to our
[2:40] partners.
[2:41] End to end, we took less than 3 months
[2:44] to onboard this cloud and the and and
[2:46] and transfer the capacity to our
[2:47] partners.
[2:49] Every day, we weren't able to use the
[2:51] GPU capacity, the money was going to
[2:54] waste,
[2:55] and the day we are running behind our
[2:56] models. In an AI race, every day counts.
[3:01] So, why did cloud work here? Nvidia
[3:04] changed the game.
[3:06] The DGX SuperPOD reference platform made
[3:08] GPU compute a commodity.
[3:10] Every vendor builds to it, same
[3:12] hardware, same topology, and our GP
[3:15] infrastructure was already similar to
[3:17] cloud, and that's why it worked.
[3:20] But, it wasn't simple. There were real
[3:22] challenges in terms of security and
[3:24] reliability, and I'll hand it off to
[3:26] Sargun to talk about it.
[3:31] >> Thank you, Dinesh. I'm Sargun Dillon, an
[3:33] engineer in compute infra. I'll be
[3:35] telling you a little bit more about our
[3:36] cloud journey.
[3:37] So, we had to come up with a plan that
[3:38] would give us the confidence to be able
[3:40] to use this cloud capacity in a little
[3:43] less than 2 months.
[3:44] That's how long it would take for the
[3:45] first test instances to be deployed.
[3:48] But, there were some caveats. We had
[3:49] security concerns. We were going to put
[3:51] our weights into the cloud.
[3:53] We were already using many meta
[3:56] technologies as part of the software
[3:58] that would run on this infrastructure.
[4:00] And reliability was paramount because we
[4:03] could not lose days in training.
[4:05] You see, meta stack is built in layers
[4:09] uh for a security story. We start with
[4:12] physical security where our data centers
[4:14] have armed guards.
[4:16] On top of that, we have network security
[4:18] where physical access to our network is
[4:21] limited and we have ACLs in place.
[4:23] On top of that, it's transport security
[4:26] where we have where we have
[4:27] cryptographic authenticity and
[4:29] authentication end-to-end between our
[4:31] systems.
[4:32] And that is all rooted in a
[4:34] cryptographic root of trust that's tied
[4:35] to the hardware.
[4:37] And this runs on top of meta's operating
[4:38] system, meta OS, to ensure that it's
[4:41] hardened against zero days.
[4:42] We had none of this in the cloud and it
[4:44] was the very basis of our security
[4:45] model.
[4:48] Fortunately, we had a trick up our
[4:49] sleeve,
[4:50] the offnet services stack.
[4:53] You see, meta doesn't just run capacity
[4:55] in our own data centers.
[4:56] We in fact run capacity all around the
[4:58] world in the form of meta network
[5:00] appliances.
[5:02] And some of these appliances are
[5:04] deployed in hostile locations. Locations
[5:06] where we don't control the data center,
[5:08] locations where there might be actors
[5:11] that have access to our hardware that we
[5:12] don't want.
[5:13] And this security model lets us run on
[5:16] top of the world where we don't trust
[5:17] anything.
[5:18] You see, the way that it works is that
[5:20] we start from a live CD like model,
[5:23] an immutable operating system.
[5:25] And after that, we perform hardware
[5:27] attestation to ensure that that hardware
[5:30] has not been tampered with. And only
[5:32] after that,
[5:33] do we begin local installation of meta's
[5:35] components and meta software on the host
[5:39] where we can then install meta's
[5:41] operating system and the requisite
[5:43] demons to enable the subsystems to
[5:45] enable TW shared and enable our ML
[5:48] workloads, enabling further operations.
[5:53] This stack does not directly connect to
[5:55] our first-party DCs.
[5:57] Instead, the stack goes through the
[5:59] Offnet services proxy.
[6:00] The Offnet services proxy is a single
[6:03] player where we can audit connections
[6:05] back to our data centers and enforce
[6:07] further authentication primitives.
[6:10] And by adapting the stack and working
[6:12] with the Offnet team, in just 60 days,
[6:15] we were able to enable the first cloud
[6:16] capacity and thus Meta's applications in
[6:19] the cloud.
[6:21] But unfortunately, that was just the
[6:23] start.
[6:24] Um when we started running workloads on
[6:26] the cloud, we found that there were
[6:27] reliability concerns.
[6:29] Our users were used to our own data
[6:31] centers, which have been tuned for our
[6:32] applications, where we had a great
[6:34] understanding of how things worked.
[6:36] But we realized the cloud vendors had
[6:38] different sets of expectations.
[6:41] You see, in our data centers, we have a
[6:42] tool, fleet health. When you run
[6:45] computers at the scale that Meta does,
[6:47] you generate millions of signals.
[6:49] And these signals can be used to detect
[6:51] patterns and to determine if systems are
[6:54] failing
[6:55] or partially healthy or healthy, and
[6:58] that lets us evacuate the workloads from
[7:00] the host or use that in scheduling.
[7:04] In addition to that, uh
[7:06] these signals can be These signals are
[7:09] not seen by the cloud vendors because of
[7:12] their arms-length engagement with
[7:14] customers, but they don't necessarily
[7:16] have a view into the customer's
[7:18] workload. Instead, they have to look at
[7:20] the systems at a very high level.
[7:23] So, we deployed our fleet health and
[7:25] network PE teams to deploy similar
[7:28] tooling to what we had in our data
[7:29] centers in the cloud.
[7:31] We had to introduce a bunch of
[7:32] cloud-specific workarounds because we
[7:34] didn't have access to things like what
[7:36] is the temperature of the CPU.
[7:38] We didn't have access to things like the
[7:39] back-end network statistics because
[7:41] they're shared resources.
[7:43] But after this, after finding these
[7:45] workarounds, we were finally able to get
[7:47] to a place where we were able to detect
[7:49] when systems were failing and send these
[7:51] back to the cloud vendors for repair and
[7:53] get our training running.
[7:54] With that, I'll hand off a little I'll
[7:56] hand off to the next talk about how we
[7:57] enabled the rest of the applications in
[7:58] the cloud.
[8:03] >> Thanks, Serkan.
[8:05] As our cloud implementation matured, we
[8:07] wanted to make it generic, able to adapt
[8:10] to any workload, not just one workload.
[8:14] But we were stuck in a cycle.
[8:16] Anytime we want to onboard a new ML
[8:19] workload, we have to go talk to that
[8:20] team, understand their requirements.
[8:23] Many of these workloads are not built in
[8:25] isolation, rather built with meta stack
[8:27] in mind. They depend on caches, data
[8:30] stores, and they leverage the security
[8:32] security and privacy capabilities.
[8:35] This was very unusual for our metamates,
[8:38] and they were facing a lot of problems.
[8:40] Because they are used to deploying these
[8:42] workloads using a common scheduling
[8:43] layer,
[8:44] and they assume it's big test capacity,
[8:47] and all of the dependent services are
[8:48] already present.
[8:50] We realized we needed to fix this.
[8:54] We needed a way to make sure that we can
[8:55] deploy any workload in cloud. So, we
[8:58] made two different decisions.
[9:00] Two decisions.
[9:01] We needed to migrate from this bespoke
[9:04] uh specialized stack with bespoke
[9:07] configurations into TW Shared.
[9:09] TW Shared stands for Twine Shared.
[9:12] Twine is our container orchestration
[9:14] platform at Meta, and this TW Shared is
[9:16] the unified capacity pool that every
[9:18] application at Meta runs on.
[9:21] The second decision we made is we need a
[9:23] way to prove that we can run any
[9:25] workload in cloud with less auditing.
[9:29] Every time an MLE had a workload failure
[9:32] because of a missing dependency,
[9:34] it
[9:35] reduced the reputation of cloud
[9:37] capacity.
[9:39] Enabling TW shared.
[9:42] The key inside us here is if we can make
[9:45] cloud capacity identical to TW shared on
[9:47] prem,
[9:48] then every engineer in Meta who's
[9:50] already familiar to familiar with how to
[9:52] deploy workload in TW shared
[9:55] can easily deploy in cloud with zero
[9:58] additional work. No onboarding, no
[10:00] special configurations, and no cloud
[10:02] team in the loop.
[10:05] But this was a significant engineering
[10:06] effort.
[10:08] Because TW shared makes assumptions in
[10:10] terms of network topology, the set of
[10:12] available dependent services, and the
[10:14] control plane.
[10:16] We had to make all of this work through
[10:19] the proxy layer and offnet security
[10:21] stack that Sargun described earlier.
[10:24] But we still want to maintain the
[10:26] developer-facing contract the same.
[10:30] In order to achieve all of this, the
[10:31] workload we chose was web. Perhaps our
[10:34] oldest workload in the company with
[10:35] complex set of dependencies, unknown
[10:37] unknowns, and years of legacy.
[10:40] We chose not what is easy, but will
[10:42] prove that we have done it.
[10:45] At the end of it, about 1% of our
[10:47] employee traffic was served from cloud
[10:49] capacity.
[10:52] Why did we choose web? Three reasons.
[10:56] The first one is the foundational role.
[10:58] Web is where the vast majority of Meta's
[11:02] product and sub-business logic lives.
[11:04] And every cloud that we were planning to
[11:06] onboard to every workload that we were
[11:08] planning to onboard to cloud talks to
[11:10] web.
[11:11] By collocating web, we reduce the
[11:14] latency and also minimize the
[11:16] cross-region traffic.
[11:18] The second is the dependency graph. Web
[11:20] talks to everything. Web talks to
[11:22] memcache, Tao, configurators, service
[11:25] router, dozens of data services, caches,
[11:28] and privacy systems.
[11:29] If we can make web's sprawling
[11:32] dependency work from cloud, we have
[11:34] effectively solved the problem for every
[11:36] other workload.
[11:38] The third reason is the credibility. As
[11:40] I said earlier, every time an MLE
[11:43] engineer MLE's workload failed because
[11:45] of a missing dependency, it eroded
[11:47] trust.
[11:49] Web is our if you can make it here, you
[11:52] can make it anywhere moment.
[11:54] But we had a problem with our offnet
[11:56] security stack approach.
[12:00] You see, Meta's security model treats
[12:02] every cloud host as untrusted.
[12:06] All traffic between cloud and the Meta
[12:09] first-party DC services goes through our
[12:12] offnet services stack governed by
[12:14] network firewall rules.
[12:17] For each service the cloud workload
[12:19] needs to talk to,
[12:20] someone has to go identify the
[12:22] dependencies,
[12:23] open up the firewall rule,
[12:25] get a security review, and get it
[12:27] approved.
[12:29] For our initial set of AI AI workloads,
[12:31] it was fine. It was a constrained set of
[12:33] workloads, maybe a handful of
[12:35] dependencies, and we punched holes and
[12:37] moved on.
[12:38] But web doesn't have a handful of
[12:40] dependencies. It has hundreds of them.
[12:44] Some of them are unknown unknowns and
[12:46] years of accumulated legacy.
[12:48] And with every web push, the
[12:50] dependencies shift.
[12:52] So, the approach of opening individual
[12:54] ports per service simply doesn't scale.
[12:59] This is where TTLS transparent TLS the
[13:01] game.
[13:03] TTLS is a Meta's internal service that
[13:06] transparently encrypts and authenticates
[13:08] traffic between endpoints.
[13:11] It leverages eBPF hooks to intercept the
[13:14] outgoing connections on the cloud host
[13:15] at the transport layer.
[13:18] No application changes, no manual
[13:20] reconfiguration of services.
[13:22] Here's how it works.
[13:24] When a container in the cloud makes a
[13:25] connection to the service running on the
[13:27] on-prem DC.
[13:29] The TTLS proxy running on the host
[13:31] intercepts it, encrypts it, and wraps it
[13:34] in a TLS tunnel, and forwards it to the
[13:36] destination.
[13:37] It also proves that it is originating
[13:39] from a meta provisioned host with proper
[13:41] credentials.
[13:43] On the first-party DC side, the reverse
[13:45] proxy
[13:46] unwraps the tunnel and forwards the
[13:48] plain text to the destination service.
[13:50] The keyword here is transparent.
[13:53] Services don't know they're in cloud.
[13:55] Developers don't need to change their
[13:57] code. It runs on host as a sidecar, and
[14:00] there's no single point of failure, and
[14:03] works for any service.
[14:05] Because it runs on every server in the
[14:07] fleet, we eliminate the reliability and
[14:10] performance cost of an intermediate
[14:11] proxy hop.
[14:13] With TTLS in place, we unlocked
[14:16] everything.
[14:18] But as the battles of AI 2025 heated up,
[14:21] we realized that we needed more and more
[14:23] capacity, and yet again, cloud was the
[14:25] answer.
[14:26] I'll hand it off to Sargun to talk more
[14:28] in detail.
[14:31] >> So, you see, all of this new capacity,
[14:34] all of this new AI capacity, is coming
[14:36] at an incredible clip across all of
[14:38] industry.
[14:40] And we've begun to run into a problem.
[14:42] GPUs are unlike CPUs in that the cloud
[14:44] vendors have had decades to mature their
[14:47] CPU offerings.
[14:48] It means that things like failures and
[14:50] maintenance are transparent to users.
[14:53] Live migration is perfected.
[14:55] GPUs don't fit into these paradigms.
[14:58] They don't act as individual computers.
[15:01] In fact, they act closer to a
[15:03] supercomputer that when one GPU fails,
[15:06] it can cause your entire job or your
[15:08] entire cluster to fail.
[15:09] And this has resulted in incredibly
[15:11] complex workflows around maintenance.
[15:15] This is another in area where Meta has
[15:18] diverged from the rest of industry.
[15:20] We have a system called maintenance
[15:22] trains.
[15:23] Every 45 days, a maintenance train rolls
[15:27] through the fleet,
[15:28] decommissions servers, unprovisions the
[15:30] OS, sets up physical maintenance,
[15:34] and allows those hosts to be upgraded
[15:36] and reprovisioned. And this is all
[15:38] integrated with Twine.
[15:40] Through Twine, workloads have northbound
[15:42] signaling about these trains.
[15:44] They're able to decamp to additional
[15:46] capacity we provision for them,
[15:48] and users rarely notice that they
[15:51] happen.
[15:53] Unfortunately, in the clouds, this
[15:55] became a set of engineers manually
[15:57] coordinating with the vendors to make
[15:59] sure that this happened.
[16:01] They had to coordinate this all by hand.
[16:04] And this was not reasonable as we
[16:07] started to go to multiple cloud
[16:08] providers. They had Slack, they had
[16:11] Jira, or even bespoke ticketing systems.
[16:14] And we weren't used to this because on
[16:17] prem, everything has become fully
[16:18] automated.
[16:20] And it meant that
[16:22] we had to adapt to these different
[16:23] workflows, and that resulted in a number
[16:25] of reliability issues.
[16:27] And we could not reasonably staff a team
[16:28] per cloud. We needed to have one way to
[16:30] do things.
[16:32] And that resulted in the advent of PCM,
[16:35] the public cloud manager.
[16:37] PCM is a layer of abstractions over
[16:40] public cloud APIs and flows.
[16:44] Things like maintenance,
[16:45] starting an instance, terminating an
[16:47] instance.
[16:48] And it helped integrate with the
[16:50] infrastructure that we already had.
[16:54] Public cloud manager takes that bare
[16:56] metal cloud capacity, does GPUs, and
[16:59] converts it quickly into something that
[17:01] can be used internally.
[17:03] On the flip side, when fleet health
[17:05] detects that there's a failure, PCM is
[17:08] responsible for communicating that
[17:10] failure and information about it to the
[17:12] cloud vendor so they can repair the
[17:14] instance.
[17:16] Maintenance trains. The fact that we
[17:18] have them and the fact that our cloud
[17:20] vendors have physical maintenance, PCM
[17:22] brings them together. PCM can go ahead
[17:24] and take a physical maintenance from a
[17:26] vendor and schedule it into a
[17:27] maintenance train. This meant no more
[17:29] slack and no more manual coordination.
[17:32] You see, the key of PCM is that it
[17:34] doesn't replace our internal systems. It
[17:37] feeds into them. It works in concert
[17:38] with them.
[17:39] And PCM is built around a set of these
[17:41] interface contracts.
[17:43] And these interface contracts can be
[17:45] implemented by a single engineer for a
[17:48] given cloud, not an entire team.
[17:51] Making adding a new provider easier than
[17:53] ever.
[17:54] And today we're able to operate PCM
[17:56] across multiple providers
[17:59] all feeding into the same production
[18:00] capacity.
[18:02] Our operators see one fleet, our users
[18:04] see one fleet, making cloud capacity
[18:06] just another kind of capacity.
[18:10] As we've comfortably begun to settle
[18:12] into the cloud, our last challenge is to
[18:14] continue to enable this hardware. New
[18:17] GPUs, new accelerators are coming about
[18:19] always.
[18:20] See, Meta has an extensive NPI process,
[18:23] new product introduction, that starts
[18:25] even before a hardware program is
[18:27] greenlit.
[18:28] Early on in this process, we not only
[18:30] set forth functional specifications,
[18:32] things like CPU speeds, amounts of
[18:34] memory, but we also talk about
[18:37] debuggability,
[18:38] operability. What does the BMC have to
[18:41] have? What do management controllers
[18:42] have to have?
[18:47] And cloud vendors are not used to
[18:48] exposing these. Cloud vendors are used
[18:50] to exposing limited APIs, and these
[18:53] limited APIs don't have those details
[18:55] that I talked about earlier, those
[18:56] details that things like fleet health
[18:58] plug into.
[18:59] So, we've begun to do something somewhat
[19:01] unusual for a
[19:02] a cloud customer. We've begun to sit
[19:05] with these vendors during ideation and
[19:08] early design,
[19:09] not after they've built these keys.
[19:12] And this has unlocked a level of
[19:14] co-design that's rarely seen in
[19:16] industry.
[19:17] We bring our NPI checklist, the same one
[19:20] that we use for our on-prem hosts, to
[19:23] these cloud vendors as they're beginning
[19:25] to design these new SKUs.
[19:27] And we tell them the same things. We
[19:28] tell them, "Here's the debuggability we
[19:30] need. Here's the observability we need.
[19:32] And here's the back-end introspection we
[19:34] need."
[19:35] And this has unlocked something super
[19:37] valuable.
[19:39] These new SKUs come with all of the
[19:41] information we need and all of the
[19:42] capabilities we need from day one.
[19:44] And it's accelerated our ability to
[19:46] onboard new SKUs and make sure that
[19:48] they're reliable.
[19:50] So, where does that put us today?
[19:53] The cloud has become a critical part of
[19:55] Meta's infrastructure. It's here to
[19:57] stay. It's growing and it's growing
[20:00] immensely.
[20:02] We have record times of onboarding new
[20:05] clouds. In a little bit less than a
[20:07] month, we can onboard a new vendor.
[20:09] In about a week, we can onboard a new
[20:11] cloud region. And it takes us about 2
[20:13] weeks to onboard a new cloud SKU that's
[20:16] gone through this co-design process.
[20:19] But the journey isn't done yet.
[20:21] As we've begun to unlock this cloud
[20:23] capacity flywheel with networking,
[20:26] co-design, fleet health, and more, we've
[20:28] begun to look to other kinds of cloud
[20:30] capacity, other kinds of accelerators,
[20:32] CPUs, and more.
[20:34] And this will help us unlock a world
[20:36] where capacity is hyper liquid, where we
[20:38] can tap any capacity anywhere, anytime
[20:40] we need. Helping us change the industry
[20:42] one more time.
[20:44] Thank you.
