Meta Uses AI Models to Build and Manage Its Own Growing GPU Networks

@Scale

Meta deploys internal artificial intelligence to bridge a widening divergence gap between exponentially complex network infrastructure and a flat engineering headcount.

AI-assisted summary of Live from SCCC: Lightning Talks Introduction - AI for NI | Mohab Gawish, Meta — original source. Check the original for full context.

Source published: 2026-09-13

Detailed notes

## Background Published by @Scale on YouTube on August 31, 2026, this video features Meta network engineer Mohab Gawish presenting at the Santa Clara Convention Center. According to the page description, Gawish introduces the lightning talk series "AI for NI" to explain Meta's strategy for managing AI network capacity.

## TL;DR Meta network engineer Mohab Gawish introduces a series of lightning talks detailing how Meta leverages AI to build, scale, and maintain its own AI network infrastructure. He highlights the "divergence gap" caused by exponentially increasing network complexity across diverse hardware/software stacks alongside a flat workforce headcount. To solve this, Meta applies AI across a four-stage capacity management pipeline consisting of Plan, Design, Deliver, and Run. Ultimately, this approach creates a virtuous cycle where AI demand fuels infrastructure growth, while AI tools empower engineers to manage that very same infrastructure.

## Atmosphere & style The presentation carries a professional, technical, and forward-looking tone characteristic of an engineering conference talk. Gawish speaks with structured clarity and collaborative energy, welcoming the audience and setting the stage for his colleagues Ben, Maz, and Yuling. The talk feels insightful and grounded, demystifying the hidden operational complexities behind headline compute numbers with precise industry concepts like the "divergence gap" and compounding pipeline stages.

## Key takeaways - [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) Mohab Gawish introduces Meta's lightning talk series on 'AI for NI' at the Santa Clara Convention Center, featuring team members Ben, Maz, Yuling, and himself. - [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) The series focuses on a closing feedback loop: using AI internally to manage, build, and sustain Meta's high-scale AI network infrastructure. - [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) Network capacity management spans a complete pipeline that transforms raw AI demand signals into a fully operational network. - [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) Extreme heterogeneity across GPU generations, NICs, topologies, and data center constraints drives true operational complexity beyond basic scale numbers. - [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) The 'divergence gap' describes the widening distance between exponentially growing network complexity and a relatively flat engineering headcount. - [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) Bridging the divergence gap serves as Meta's primary motivation for integrating AI into network engineering workflows. - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) Meta structures network capacity management into four continuous pipeline stages: Plan, Design, Deliver, and Run. - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) The Plan stage converts raw workload demands into actionable capacity, timeline, and location requirements. - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) The Design stage establishes architectural blueprints, covering network topologies, transport protocols, hardware specs, and software stacks. - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) The Deliver stage handles physical rack rolling, cabling, configuration, and validation tests to take the network live. - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) The Run stage focuses on maintaining performance for AI workloads, executing defense-in-depth measures, and mitigating failures. - [07:17](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=437s) Operational complexity compounds multiplicatively across stages, as design choices create expanding sets of delivery and run requirements. - [07:17](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=437s) AI creates a virtuous cycle where compute demand drives infrastructure expansion, while AI tools simultaneously optimize and scale that same infrastructure.

## Detailed walkthrough ### [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) Introduction to AI Network Capacity Management

Mohab Gawish, a network engineer at Meta, opens the session by welcoming the audience at the Santa Clara Convention Center. Joined by his colleagues—Ben, Maz, and Yuling—he introduces a series of lightning talks designed to explain Meta's strategy for managing AI network capacity. Unlike traditional presentations focused solely on the hardware built to support artificial intelligence, this lightning series highlights a closing feedback loop: how Meta leverages AI to build and sustain that very same infrastructure.

The capacity management lifecycle spans a full pipeline—from taking raw demand signals for AI workloads on one end to delivering a fully operational, live network on the other. While Meta and other hyperscalers continue to rapidly scale compute capacity and expand their power footprint to support large-scale AI workloads, Gawish notes that these large headline numbers hide the true underlying operational complexity.

- **Speakers:** Mohab Gawish (Meta Network Engineer), alongside colleagues Ben, Maz, and Yuling. - **Core Concept:** Utilizing AI internally as an operational tool to manage, build, and support high-scale AI network infrastructure.

---

[01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) The Heterogeneity Challenge and the Divergence Gap

The primary operational challenge is not just the sheer volume of power or compute, but the extreme heterogeneity across hardware and software stacks. scale-out network infrastructure relies on diverse combinations of GPUs and NICs—incorporating technologies like Meta RoCE (RDMA over Converged Ethernet) and MRC (Meta Remote Compute). Each GPU generation, vendor, and hardware type introduces unique requirements regarding power, bandwidth, network topology, transport protocols, and data center constraints (such as physical space, cooling, and power distribution).

This compounding variety creates what Gawish defines as the "divergence gap." While power demands grow linearly and infrastructure complexity scales exponentially due to varied hardware and software variables, the human workforce available to build and manage this infrastructure remains relatively flat. The core motivation for implementing AI in network engineering is to bridge this gap between exponential system complexity and finite headcounts.

- **Primary Driver of Complexity:** Heterogeneous hardware/software environments, varying GPU/NIC specs, topology formation, data center constraints (cooling, space, power). - **The Divergence Gap:** The widening distance between exponentially increasing network complexity and a flat headcount of network engineers.

---

[04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) The Four Stages of Network Infrastructure Management

Meta’s network capacity management operates as a continuous four-stage pipeline: **Plan**, **Design**, **Deliver**, and **Run**.

- **Plan:** Translates raw AI demand signals into concrete capacity requirements, determining exactly how many GPUs are needed, by what date, and in which locations. - **Design:** Defines high-level architectural blueprints based on planning signals. This includes making decisions on network topology, transport protocols suited for specific GPU/NIC capabilities, hardware selection, and software stack choices. - **Deliver:** Executes the physical and logical rollout. Activities include rack rolling, fiber cabling layout, device power-up, configuration deployment, and validation tests until the network is live. - **Run:** Manages ongoing network operations. This phase ensures high performance for demanding AI workloads, implements defense-in-depth measures, and handles hardware/software failures in a timely manner.

---

[07:17](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=437s) Compounding Complexity and the Virtuous Cycle

The pipeline stages are connected with multiplication signs rather than simple arrows to signify how operational complexity compounds from phase to phase. A single new feature selected during the design phase can fork into numerous operational variables down the line. For example, adopting emerging standards like Ultra Ethernet and features like packet trimming for rapid failure detection requires checking hardware compatibility, setting specific configurations, and diagnosing novel failure modes during the delivery and run phases.

Using AI creates a "virtuous cycle" where AI drives the demand for compute capacity while simultaneously acting as the primary force multiplier to build, scale, and manage its own underlying infrastructure. Gawish concludes the intro by handing the stage over to Ben to present the "Plan" stage, followed by Maz on "Design", Gawish returning for "Deliver", and Yuling closing with "Run".

> "AI drives capacity demand, and at the same time, AI helps us build the infrastructure that powers itself."

## Notable quotes > [07:17](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=437s) AI drives capacity demand, and at the same time, AI helps us build the infrastructure that powers itself. — Mohab Gawish

## Mentioned Mohab Gawish, Ben, Maz, Yuling, Meta, Santa Clara Convention Center, Meta RoCE, MRC, Ultra Ethernet

[Source](https://www.youtube.com/watch?v=GYTPXFQ8GJc)

中文简报

Meta利用AI模型构建与管理日益庞大的GPU网络 Meta部署内部人工智能，以弥合指数级增长的复杂网络基础设施与保持平稳的工程团队规模之间日益扩大的脱节差距。 Meta网络工程师Mohab Gawish介绍了一系列闪电演讲，详细阐述了Meta如何利用AI来构建、扩展和维护自身的AI网络基础设施。他重点强调了跨多样化软硬件技术栈呈指数级增长的网络复杂度与基本持平的人工团队规模之间所形成的“脱节差距”（divergence gap）。为解决这一问题，Meta将AI应用于涵盖规划（Plan）、设计（Design）、交付（Deliver）和运行（Run）的四阶段容量管理全流程中。最终，这种方法形成了一个良性循环：AI需求推动基础设施扩张，而AI工具则赋能工程师有效管理这一日益庞大的基础设施。 Meta 网络工程师 Mohab Gawish 介绍了“AI for NI”系列闪电演讲，重点展示 Meta 如何在内部利用 AI 来构建、管理与维持其超大规模 AI 网络基础设施的闭环。 (0:02) GPU 代际、网卡（NIC）及数据中心物理限制等极端的软硬件异构性，导致网络复杂度呈指数级上升，而工程师人力保持相对稳定，弥合这一“分歧差距”（divergence gap）是引入 AI 的主要驱动力。 (1:48) Meta 将网络容量管理流程划分为“计划（Plan）、设计（Design）、交付（Deliver）和运行（Run）”四个连续阶段，实现从原始需求信号到网络完全投运的全生命周期管理。 (4:57) 由于跨阶段的技术决策会导致复杂度呈乘数级复合递增，Meta 致力于打造良性循环，让 AI 在驱动算力需求的同时，反哺并加速自身底层基础设施的构建与运维。 (7:17)

Reading summary

According to Meta network engineer Mohab Gawish, expanding AI workloads create an operational "divergence gap" between exponential infrastructure complexity and finite human headcounts, requiring Meta to deploy AI as an internal tool to manage its network capacity. Hardware Heterogeneity and the Divergence Gap (1:48) Gawish states that the primary operational hurdle in scaling AI networks is not merely compute volume, but extreme heterogeneity across hardware and software environments. Scale-out clusters rely on diverse combinations of GPU generations, vendors, and network interface cards (NICs), utilizing specialized technologies like Meta Remote Compute (MRC) and Meta RoCE (RDMA over Converged Ethernet). Each hardware combination introduces distinct requirements for transport protocols, network topologies, physical data center space, cooling systems, and power distribution. This compounding diversity drives what Gawish defines as the divergence gap: system complexity grows exponentially alongside linear power increases, while the engineering headcount available to deploy and manage this infrastructure remains flat. Meta positions internal AI adoption as the necessary mechanism to bridge this widening operational divide. Compounding Multipliers Across the Capacity Lifecycle (4:57) Meta structures its network capacity management across a continuous four-stage pipeline: Plan, Design, Deliver, and Run. The planning stage converts raw demand signals into specific GPU counts, target dates, and geographic locations. The design phase establishes architecture blueprints, including topologies, transport protocols, and hardware-software selections. The deliver phase oversees physical rack rolling, cabling layout, power-up, configuration, and validation testing, while the run phase handles real-time workload performance, defense-in-depth measures, and failure remediation. Gawish emphasizes that complexity compounds multiplicatively across these stages rather than linearly. A single architectural selection made during design—such as adopting Ultra Ethernet standards and packet trimming for rapid failure detection—forks into multiple operational dependencies, requiring custom compatibility checks, specific configurations, and unique diagnostic workflows during the deliver and run phases. Closing the Loop in a Virtuous Cycle (7:17) To counter compounding operational overhead, Meta uses AI as a direct force multiplier within infrastructure workflows. Gawish outlines a closed-loop "virtuous cycle" wherein AI workloads drive the demand for massive compute capacity, while AI-driven tools are simultaneously applied to design, construct, and sustain the underlying network infrastructure. Gawish presents internal AI-driven network management as an operational imperative for hyperscale survival, serving as the conceptual foundation for detailed stage-by-stage implementations presented by team members Ben (Plan), Maz (Design), Gawish (Deliver), and Yuling (Run).

精炼阅读摘要

Meta网络工程师Mohab Gawish指出，面对AI算力异构复杂性的指数级增长与工程团队人力有限之间的“分化差距”，Meta正将AI作为核心运营工具，以此构建并维护支撑AI负载的底层网络基础设施。异构系统引发的分化差距挑战 (1:48) Gawish强调，AI网络基础设施的主要运营挑战不仅在于电力和计算容量的绝对规模，更在于软硬件栈的高度异构性。横向扩展的网络依赖于多种GPU与NIC的组合，涉及Meta RoCE（基于融合以太网的RDMA）和MRC（Meta远程计算）等技术。不同厂商与代际的GPU硬件对供电、带宽、网络拓扑、传输协议以及数据中心的散热和空间提出了互不相同的约束条件。这种复杂性的复合增长导致了所谓的“分化差距”（divergence gap）：在基础设施变量与运维复杂度呈指数级攀升的同时，负责构建和管理系统的网络工程师团队规模却相对保持平稳。因此，仅依靠人力难以填补这一扩展缺口，推动了将AI引入网络工程实践的需求。网络容量的四阶段全生命周期管理 (4:57) Meta将网络容量管理流水线划分为四个连续阶段：规划（Plan）、设计（Design）、交付（Deliver）和运行（Run）。规划阶段负责将原始AI工作负载需求转化为具体的容量指标，明确在特定时间和地点所需的GPU配额；设计阶段则确立架构蓝图，根据GPU和NIC的规格确定网络拓扑、传输协议以及具体的软硬件栈方案。在执行层面，交付阶段涵盖机架就位、光纤布线、设备加电、配置部署与验证测试等物理和逻辑交付；运行阶段则负责维护高负载下的网络性能，实施纵深防御策略，并对突发的软硬件故障进行及时响应与处置。复杂度的乘数效应与AI闭环赋能 (7:17) 该流水线各阶段之间的复杂度呈乘数级叠加而非简单的线性递进。在早期设计阶段引入的新特性，会在后续交付与运行中派生出大量运维变量。例如，引入Ultra Ethernet新标准或用于快速故障检测的报文裁剪（packet trimming）特性，均要求后续环节进行额外的硬件兼容性核验、特定配置下发以及针对新型故障模式的排查。 Gawish提出，通过AI赋能能够建立一个“良性循环”：AI工作负载在推动算力需求扩张的同时，也被用作构建、扩展和运维底层基础设施的核心生产力放大器，从而有效应对端到端系统复杂度的复合增长。 Gawish认为，解决超大规模AI网络运营瓶颈的关键在于闭合AI反哺基础设施的反馈回路；本场闪电演讲后续由团队工程师Ben、Maz、Gawish及Yuling分别展开，依次探讨规划、设计、交付与运行四个环节的具体工程落地。

Detailed notes — zh

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布。页面说明介绍，这是来自 Meta 的工程师 Mohab Gawish 在圣克拉拉会议中心（Santa Clara Convention Center）现场发表的“AI for NI”系列闪电演讲引言分享。 ## TL;DR Mohab Gawish 深入阐述了 Meta 如何利用 AI 反哺自身网络基础设施构建与容量管理，打造 AI 与基础设施的共生良性闭环。面对多代 GPU、专用网卡和物理数据中心边界所带来的严重软硬件异构性，系统复杂度呈指数级爆发，与平缓的团队人力基线之间形成了巨大的“发散鸿沟”（Divergence Gap）。为此，Meta 将容量管理抽象为计划（Plan）、设计（Design）、交付（Deliver）与运维（Run）四大具有连锁倍乘效应的工程阶段，并引入 AI 作为力量倍增器以应对挑战。最后，Mohab 梳理了本场系列闪电演讲中四位工程师各自负责的细分议题与交接节奏。 ## Atmosphere & style 演讲洋溢着顶级互联网大厂技术峰会特有的紧凑、硬核与高信息密度氛围。Mohab 演讲台风从容专业、逻辑层层递进，通过将流程连线刻意替换为“乘号”以及“发散鸿沟”曲线对比等视觉化表达，将复杂庞大的分布式网络痛点阐述得直观而透彻。整场分享兼具宏观架构视野与底层工程洞察，为后续四位讲者的深度技术拆解做了极佳的气氛铺垫与框架搭建。 ## Bulletin - [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) Meta 探讨“AI for NI”工程范式，从传统硬件支撑 AI 转向利用 AI 反哺并自动化管理端到端的 AI 网络基础设施闭环。 - [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) 随着数据中心软硬件异构性激增，工程复杂度呈指数级爆发，与平缓的人力规模之间撕裂出难以逾越的“发散鸿沟”（Divergence Gap）。 - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 网络容量管理包含计划、设计、交付与运维四大阶段，各环节之间呈“相乘”关系，单个变量的引入将引发全流程级联的指数级复杂度扩散。 - [09:03](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=543s) Meta 将 AI 视作核心生产力倍增器，通过在容量管理全生命周期各环节深度集成 AI 工具，填平复杂度与工程人力之间的缺口。 ## Key takeaways - [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) AI 网络容量管理形成了双向共生闭环：一方面网络基建承载大规模 AI 算力，另一方面借助 AI 能力自动化构建与管理网络基建。 - [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) 容量管理的核心目标是将前端粗粒度的 AI 业务与算力需求信号，转化为高可用承载业务流量的在线生产网络。 - [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) 数据中心扩容带来的电力足迹（Power Footprint）增长仅是表象，真正严峻的工程瓶颈在于底层软硬件的严重异构性。 - [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) 单一物理网络需同时兼容不同厂商与多代际的 GPU/NIC 架构，软硬件对网络拓扑、传输协议与物理空间等提出了互斥需求。 - [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) “发散鸿沟”定义了系统复杂度的指数级增长与网络工程团队人力（HC）线性或平缓基线之间的巨大矛盾。 - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 端到端容量管理流水线被严密解构为四大连续阶段：计划（Plan）、设计（Design）、交付（Deliver）和运维（Run）。 - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 计划（Plan）阶段致力于精准映射需求信号，厘清 GPU 接入的时间节点、空间位置与容量规模指标。 - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 设计（Design）阶段结合硬件特性完成全局决策，输出包括网络拓扑、传输协议栈及硬件选型的工程蓝图。 - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 交付（Deliver）阶段涵盖机架进场、光纤铺设、设备上电、网络配置推送与链路连通性验证全套落地动作。 - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 运维（Run）阶段聚焦保障网络超高吞吐与极低时延，通过主动防护与纵深防御应对新型故障模式。 - [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 四大阶段之间具有复杂的“相乘放大效应”，例如设计层引入的新协议特性会连锁引发配置兼容性与未知故障模式的爆炸性增加。 - [09:03](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=543s) Meta 系列闪电演讲由 Ben、Maz、Mohab 和 Yuling 分工接力，分别对应容量管理四阶段的最佳工程实践探索。 ## Detailed walkthrough ### [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) 引言：AI 赋能网络基础设施闭环来自 Meta 的网络工程师 Mohab Gawish 登台向现场观众致意，并介绍了本次闪电演讲（Lightning Talks）的主题与演讲阵容。本场系列分享由他和同事 Ben、Maz 以及 Yuling 共同完成，核心议题聚焦于 Meta 在管理 AI 网络容量（AI network capacity management）方面的工程实践，尤其是探索如何借助 AI 本身的力量来赋能并优化这一全流程。 Mohab 指出，本次演讲与当天早些时候的其他分享形成了极佳的闭环互补：之前的演讲主要探讨的是“如何构建支撑 AI 的网络基础设施”，而本次闪电演讲的核心则是“如何利用 AI 反哺并构建这些关键的基础设施”。所谓 AI 网络容量管理，本质上是一个复杂的工程流水线——它以最前端粗糙的 AI 算力与工作负载需求信号为输入，最终交付并运行一个能够高稳定承载该业务负荷的生产级在线网络。 ```mermaid flowchart LR accTitle: AI与基础设施的良性闭环 accDescr: 展示传统硬件基础设施支持AI模型，而AI反向优化容量管理与基建的互惠循环。 subgraph CoreLoop ["AI 与基础设施的共生闭环"] direction LR Infra["网络基础设施建设 (支持 AI 算力)"] -->|"驱动发展"| AI["AI 算力与工作负载"] AI -->|"赋能与反哺"| Pipe["AI 容量管理自动化 (AI for NI)"] Pipe -->|"交付落地"| Infra end ``` --- ### [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) 异构性激增与“发散鸿沟”（Divergence Gap）与其他超大规模云厂商（Hyperscalers）类似，Meta 近年来为了应对爆发式增长的 AI 工作负载，持续大规模扩充计算容量，导致数据中心的能耗足迹（Power Footprint）急剧扩大。然而，能耗或算力规模这一引人注目的“表层数字”并不是工程挑战的全部真相，它遮蔽了支撑其运转的网络底层所面临的巨大复杂性。这种复杂性主要源于硬件和软件层面极度严重的异构性（Heterogeneity）。以 GPU 及其 Scale-out 网络网卡（NIC）为例（如当天大会提到的 MRC 和 MetaRoCE），不同厂商、不同代际的 GPU 具有完全不同的传输速度、功耗指标以及对网络行为的定制化预期。在实际数据中心中，单一物理网络往往必须同时支撑多代混合部署的 GPU，而这些硬件对网络拓扑规划、传输层协议、软硬件选型等都提出了严苛且互斥的要求；与此同时，所有方案还必须受到数据中心物理空间、冷却方案和供电上限等硬性条件的制约。这便引出了本次分享的核心问题陈述——“发散鸿沟”（Divergence Gap）。Mohab 指出，即便假设数据中心能耗仅随业务呈线性增长，其背后的系统工程复杂度也会呈指数级爆发。如果在面对这种指数级膨胀时，支持团队的人力规模（HC）保持平稳，复杂度与人力之间就会被撕开一条难以逾越的鸿沟： - **线性增长**：数据中心电力预算与基础容量规模。 - **指数增长**：软硬件代际混合、拓扑分支与故障场景叠加导致的工程复杂度。 - **平缓基线**：负责交付和运维这些网络基础设施的工程师团队规模。 - **核心动机**：Meta 引入 AI 技术的核心出发点，正是要利用 AI 充当生产力倍增器，跨越由粉色复杂度曲线与灰色人力曲线所构成的“发散鸿沟”。 --- ### [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 容量管理全流程：从相加到相乘的连锁复杂度 Mohab 将端到端的网络容量管理抽象为一个包含四个连续阶段的工程流水线（Pipeline）：计划（Plan）、设计（Design）、交付（Deliver）和运维（Run）。本次闪电演讲的后续环节将沿着这四大模块逐层拆解 AI 的切入方案： 1. **计划（Plan）**：将业务侧粗粒度的原始 AI 需求信号，转化为精准的网络容量指标，厘清何时、何地需要接入多少规模的 GPU。 2. **设计（Design）**：根据计划指标制定全局工程决策，结合具体的 GPU/NIC 硬件能力定义网络拓扑架构、传输协议选型、硬件设备与软件协议栈，形成供现场工程团队执行的设计蓝图。 3. **交付（Deliver）**：落地执行阶段，涵盖机架进场（Rack Rolling）、物理光纤铺设与布线（Cabling & Fiber Layout）、设备通电上电、网络配置推送与连通性验证，最终上线一个可承载流量的生产网络。 4. **运维（Run）**：保持网络全天候平稳运行，持续满足 AI 分布式训练对网络超高吞吐与极低时延的严苛性能要求，实现故障的主动防护、及时处置与网络纵深防御。 ```mermaid flowchart LR accTitle: 网络容量管理端到端流程 accDescr: 展示从需求到线上运维的四个阶段，阶段之间为乘号关系表示复杂度逐级发散。 subgraph Pipeline ["容量管理四大阶段（复杂度乘积效应）"] direction LR P["1. 计划 (Plan) 需求到容量映射"] -->|"×"| D["2. 设计 (Design) 拓扑、传输协议与软硬件选型"] D -->|"×"| DL["3. 交付 (Deliver) 上架、布线、通电与配置验证"] DL -->|"×"| R["4. 运维 (Run) 高可用保障与故障闭环"] end ``` Mohab 特别强调，各阶段之间在示意图中刻意使用了“乘号（$\times$）”而非普通的递进箭头。这是因为异构性带来的复杂度在流水线各阶段之间是呈几何级数放大扩散的。例如在设计阶段引入一个新型协议变量（如 Ultra Ethernet 联盟推动的 Packet Trimming 等快速故障感知机制），看似只是一个局部的架构设计决策，但它会瞬间分叉并渗透到后续全流程中：硬件是否兼容该特性、如何下发配置脚本、该机制会在网络中诱发哪些新型故障模式（Failure Modes）、以及它如何与周边生态组件交互协同。这种级联效应导致每引入一个新变量，整体维护复杂度就会呈指数级攀升。 --- ### [09:03](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=543s) 良性循环与闪电演讲议程分工在演讲的尾声，Mohab 总结了 AI 与网络基建之间的“良性循环（Virtuous Cycle）”逻辑：AI 工作负载的繁荣持续推动网络算力与带宽需求的增长，而网络工程师则反过来借助 AI 来构建更强大的底层基础设施以承载 AI 自身。AI 在此处扮演着关键的“力量倍增器（Force Multiplier）”，旨在填平复杂度与工程人力之间的发散鸿沟。随后，Mohab 介绍了后续四场紧凑的闪电演讲安排与对应讲者，并正式将讲台移交给下一位同事： - **Plan（计划）**：由 Ben 展开剖析需求与容量规划算法。 - **Design（设计）**：由 Maz 分享网络拓扑与架构决策。 - **Deliver（交付）**：Mohab 本人将再次登台，详解自动化部署与上架交付。 - **Run（运维）**：由 Yuling 压轴分享生产环境的高可用运维机制。 ## Notable quotes > [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) 如何构建支撑 AI 的网络基础设施 — Mohab Gawish > [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) 如何利用 AI 反哺并构建这些关键的基础设施 — Mohab Gawish > [00:02](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=2s) 以最前端粗糙的 AI 算力与工作负载需求信号为输入，最终交付并运行一个能够高稳定承载该业务负荷的生产级在线网络 — Mohab Gawish > [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) 发散鸿沟 — Mohab Gawish > [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) 即便假设数据中心能耗仅随业务呈线性增长，其背后的系统工程复杂度也会呈指数级爆发 — Mohab Gawish > [01:48](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=108s) Meta 引入 AI 技术的核心出发点，正是要利用 AI 充当生产力倍增器，跨越由粉色复杂度曲线与灰色人力曲线所构成的“发散鸿沟” — Mohab Gawish > [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 计划（Plan）、设计（Design）、交付（Deliver）和运维（Run） — Mohab Gawish > [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 异构性带来的复杂度在流水线各阶段之间是呈几何级数放大扩散的 — Mohab Gawish > [04:57](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=297s) 每引入一个新变量，整体维护复杂度就会呈指数级攀升 — Mohab Gawish > [09:03](https://www.youtube.com/watch?v=GYTPXFQ8GJc&t=543s) AI 工作负载的繁荣持续推动网络算力与带宽需求的增长，而网络工程师则反过来借助 AI 来构建更强大的底层基础设施以承载 AI 自身 — Mohab Gawish ## Mentioned Mohab Gawish, Meta, Ben, Maz, Yuling, MRC, MetaRoCE, Ultra Ethernet 联盟, Packet Trimming, Santa Clara Convention Center, @Scale ## Video description Mohab Gawish presents the Lighting Talk Introduction "AI for NI" live at the Santa Clara Convention Center. Learn more about @Scale here: atscaleconference.com [Source](https://www.youtube.com/watch?v=GYTPXFQ8GJc)

Source quotes

[0:02] Mohab Gawish: 如何构建支撑 AI 的网络基础设施 [0:02] Mohab Gawish: 如何利用 AI 反哺并构建这些关键的基础设施 [0:02] Mohab Gawish: 以最前端粗糙的 AI 算力与工作负载需求信号为输入，最终交付并运行一个能够高稳定承载该业务负荷的生产级在线网络 [1:48] Mohab Gawish: 发散鸿沟 [1:48] Mohab Gawish: 即便假设数据中心能耗仅随业务呈线性增长，其背后的系统工程复杂度也会呈指数级爆发 [1:48] Mohab Gawish: Meta 引入 AI 技术的核心出发点，正是要利用 AI 充当生产力倍增器，跨越由粉色复杂度曲线与灰色人力曲线所构成的“发散鸿沟” [4:57] Mohab Gawish: 计划（Plan）、设计（Design）、交付（Deliver）和运维（Run） [4:57] Mohab Gawish: 异构性带来的复杂度在流水线各阶段之间是呈几何级数放大扩散的 [4:57] Mohab Gawish: 每引入一个新变量，整体维护复杂度就会呈指数级攀升 [9:03] Mohab Gawish: AI 工作负载的繁荣持续推动网络算力与带宽需求的增长，而网络工程师则反过来借助 AI 来构建更强大的底层基础设施以承载 AI 自身

Source quotes

[7:17] Mohab Gawish: AI drives capacity demand, and at the same time, AI helps us build the infrastructure that powers itself.

Original transcript

[0:02] Hello everyone. Great crowd. Um my name [0:06] is Muhab. I'm uh a network engineer uh [0:09] at Meta. First of all, thank you all for [0:11] being u with us today and uh me along [0:17] with a group of my colleagues uh Ben Maz [0:21] and Yuling will be walking you through [0:24] uh some of the work that we have been [0:27] doing in terms of managing AI uh network [0:31] capacity. It will be in the form of a [0:33] lightning series. So slightly different [0:35] format from uh the talks that we've had [0:39] uh earlier today [0:41] and uh the focus of the discussion will [0:44] be primarily on how we have been [0:48] leveraging the power of AI [0:51] in this activity. [0:54] So when we are talking about capacity [0:58] management or AI network capacity [1:00] management, what do we exactly mean by [1:03] that? We are essentially talking about [1:06] the pipeline that takes a raw demand [1:09] signal of AI workload and from the other [1:13] side gives us a live operational network [1:17] that is capable of supporting that [1:20] workload. And one of the themes that [1:23] will emerge as we go through uh these [1:26] talks is that it kind of closes the loop [1:30] with the previous talks. The previous [1:32] talks the the focus was primarily on how [1:36] we are building the infrastructure that [1:38] makes AI possible. The theme of our [1:41] talks will be how we are leveraging AI [1:43] to make that infrastructure possible. [1:48] So as it might not be a surprise to any [1:51] anyone of you here is that Meta has been [1:54] like along with many of the other [1:55] hyperscalers. [1:58] We have been on a journey of building [2:00] lots of compute capacity mostly driven [2:04] by AI workload and the power footprint [2:08] has grown quite a bit over uh the last [2:11] few years. And while this headline [2:14] figure um might look large, it's [2:19] actually not the center of the story [2:21] that we want to walk you through today. [2:24] This headline figure when we think in [2:27] terms of the network that needs to [2:29] support it, it actually hides a great [2:32] deal of complexity underneath. That [2:35] complexity stems primarily from the [2:37] heterogeneity of the types of hardware [2:40] and software that the network needs to [2:42] account for. If we pick for example [2:44] specifically on on GPUs and the nicks [2:47] that facilitate its scale out network [2:49] like you you might have been with us [2:51] earlier today with the talks around MRC [2:54] and Metaroki. There is lots of different [2:57] capabilities. Every GPU, every vendor, [3:00] every generation comes with its own set [3:02] of speed, you know, power, its own [3:06] particular demands and expectation uh of [3:10] the network. So we might end up in a [3:12] place where one network has to support [3:14] multiple generations of GPUs with [3:16] varying capabilities and varying [3:19] expectation of the network. And that in [3:21] turn translates to specific demands of [3:24] what that network looks like which may [3:26] influence the topology formation, [3:29] hardware choices, software choices and [3:31] everything that is required to make a [3:34] network operational and able to steer [3:36] packets around. And of course all of [3:39] that is incubated [3:41] within some infrastraints in the data [3:43] center like space for example, cooling, [3:47] power and all of these variables. Now [3:51] with all of these things considered [3:55] we have what we may call the divergence [3:57] gap and that's the key problem statement [4:00] that we're trying to present today and [4:02] also present how we have been attempting [4:05] to solve it through the power of AI. [4:08] If we assume for now that power grows [4:10] linearly, complexity given all of the [4:13] factors that we've just mentioned [4:15] grows exponentially. [4:17] And if we want to manage all of that [4:20] defensively with a flat number of [4:22] people, we have what you see on the [4:24] graph here, what we may call the [4:26] divergence gap. That gap between the [4:28] pink line and uh representing complexity [4:31] and the gray line representing the [4:34] number of people that are meant to [4:35] support it. This is the key motivation [4:38] for our exploration of using AI. It's [4:42] [clears throat and snorts] basically our [4:43] attempt to bridge the gap between [4:46] complexity and the number of people that [4:49] are meant to tackle it as part of our [4:51] journey building AI uh network [4:54] infrastructure. [4:57] Going back to the pipeline, we keep [4:59] referring to the capacity management [5:01] business as a pipeline. This is more of [5:03] a manifestation of what it looks like. [5:06] It's basically four sequential stages. [5:10] Plan, design, deliver, and run. And [5:13] those will be the core of our lightning [5:15] series talks today. We'll have a bit of [5:18] a deep dive in each one of those [5:20] focusing primarily on how AI is being [5:23] leveraged in each one of those. So for [5:27] plan we'll be having we we will be [5:29] basically focusing on [5:32] how a raw demand signal translates to a [5:36] capacity requirement. How many GPUs do [5:39] we need by when and where design is a [5:43] subsequent phase after that that picks [5:45] up that signal and this is where we make [5:48] highlevel decisions around how exactly [5:51] we're going to build the network that [5:53] supports that demand. This includes [5:56] decisions around the network topology. [5:59] For example, what kind of transport do [6:01] we need to account for given the [6:03] specific capabilities of the GPUs and [6:06] the nicks that we'll be using? [6:09] Hardware choices, software choices, and [6:12] basically everything that is required by [6:15] a team of implementers or installers to [6:18] actually build out the network. [6:21] This is followed by the deliver phase [6:24] and this is where we make it happen. Now [6:26] we have the design blueprint from the [6:28] design phase. It's time to actually [6:30] build out the network. So this includes [6:32] all sorts of activities around rack [6:34] rolling cabling fiber layout powering up [6:39] networking devices configuration [6:41] validation and everything that ends up [6:44] with an operational network that is [6:46] capable of supporting the AI workload. [6:49] The last stage is run. This is where we [6:52] simply keep operating the network. And [6:54] that in itself is is a challenge. Like [6:57] we're talking about making sure that [6:59] we're continuously meeting the high bar [7:03] of performance expected by AI workloads. [7:06] Make sure that failures are handled [7:08] timely, that we have the right level of [7:10] defense in depth in the network itself [7:13] to make sure that it continues to [7:14] operate smoothly. [7:17] Now one thing to highlight here is that [7:20] while this is being presented as a [7:22] sequential pipeline, we have [7:25] deliberately chosen the multiply sign [7:27] between stages as opposed to an arrow. [7:31] And this is to go back again to the key [7:34] challenge here that the heterogeneity [7:36] and the divergence [7:38] multiplies from one stage to another. [7:41] one variable, one novel variable in the [7:45] design phase can fork into multiple [7:47] variables in the subsequent delivery and [7:50] run phase. Like when you think about all [7:52] of the novelty that is coming in the AI [7:54] world, all of the new standards are [7:56] emerging and even the different [7:58] companies efforts into building out [8:01] their own technology stack. There are [8:03] lots of things that are causing us to [8:05] rethink how we model how the network [8:09] operates, what failure moods we need to [8:11] think of. Like if we look at something [8:13] like ultra ethernet for example and some [8:15] of the capabilities being proposed to [8:17] expedite how we detect failures, [8:20] features like packet trimming for [8:21] example. While that might sound like a [8:24] new feature in the design phase that we [8:27] make a decision on whether or not to use [8:29] if we go forward with that we'll have to [8:31] account of account on whether hardware [8:34] supports it and if it does support it [8:36] how to configure it and if we configure [8:37] it what failure moods does this novel [8:41] capability bring into the network that [8:43] we have to account for and how it [8:45] interacts with the overall ecosystem [8:47] surrounding it and that's why we think [8:50] that in the fast evolving AI world [8:53] complexity grows exponentially with [8:55] every new variable or variant of a [8:58] component that comes into the network [9:03] and this is what this is how we create [9:06] the f the virtuous cycle. AI drives [9:08] capacity demand and at the same time AI [9:12] helps us build the infrastructure that [9:16] powers itself and that will be the [9:18] centerpiece or the central theme of the [9:21] whole talk today. And the key motivation [9:25] is to use AI as the force multiplier to [9:30] keep up with the growing the [9:31] exponentially growing complexity and [9:34] bridge the divergence gap which we have [9:36] chosen as the key problem statement that [9:39] we want to address. [9:41] So we will have four lightning talks [9:45] focusing on each building block of the [9:48] pipeline. So uh we will have for plan we [9:52] have Ben for design we have mass and [9:55] then I'll come back over to the to the [9:58] stage to talk to deliver and then [10:00] finally we'll have Yuling who will uh [10:02] speak to Ron. Thank you all and I hand [10:06] over to Ben to speak to plan. Thank you.