OpenAI Open-Sources Network Protocol to End AI Training Bottlenecks

OpenAI开源网络协议以消除AI训练瓶颈 多路径可靠连接（Multipath Reliable Connection）协议通过允许GPU集群端点在数毫秒内自主绕过网络硬件故障，消除了传统交换机路由延迟。 本期播客探讨了OpenAI如何通过从零重构数据中心基础设施，克服训练同步的大规模AI工作负载所面临的严峻网络挑战。嘉宾Greg Steinbrecher与Mark Handley详细介绍了多路径可靠连接（MRC）协议的开发细节；这一与主要技术合作伙伴联合构建的新型开放标准协议，借助数据包裁剪和静态路由等技术，能在毫秒级内绕过持续发生的硬件故障。通过消除网络瓶颈并缓解尾部延迟，这种协同设计理念大幅提升了GPU利用效率与训练可靠性。最后，讨论还将这些高度复杂的地面工程突破与天基计算在后勤物流层面的不切实际进行了对比。 训练前沿 AI 模型需要海量顶级 GPU 保持严密的同步协同，单一链路的延迟或故障都会迫使整个集群停顿等待，促使 OpenAI 采用基础设施工程师与模型研究团队紧密并肩的“协同设计（Co-design）”模式。 (5:04) 在数千台交换架构成的大规模集群中，硬件及光链路损坏已是统计上的必然，且同步任务极易受最坏路由拥塞（P100 问题）制约，这要求网络系统必须在设计底层原生具备容错能力。 (8:50) OpenAI 联合微软、英伟达、AMD、博通及英特尔推出了 MRC（多路径可靠连接）协议，利用“数据包修剪”（Packet Trimming）在拥塞丢包时仍向前转发包头，彻底消除了乱序与丢包歧义并实现即时重传。 (15:38) MRC 具备类似“自愈”的容错特性，端点可在毫秒级内自主探测并绕过故障路径，避开了传统 BGP 路由协议缓慢的收敛过程，并允许超大规模集群采用纯静态路由以消除交换机控制面的复杂性。 (18:32) OpenAI 已通过开放计算项目（OCP）将 MRC 规范确立为行业开放标准，该协议还支持构建层级更少、能耗更低的扁平网络拓扑，将更多宝贵电力直接留给 GPU 算力本身。 (25:05)

OpenAI engineers Mark Handley and Greg Steinbrecher argue that scaling frontier AI models requires replacing conventional, internet-derived data center networking with protocols tailored for synchronized workloads. To eliminate severe tail-latency bottlenecks and constant hardware failures across massive GPU clusters, OpenAI developed and open-sourced Multipath Reliable Connection (MRC), shifting routing intelligence to endpoints and simplifying switch infrastructure. The Fragility of Synchronous AI Workloads (5:04) Traditional internet and web data center architectures rely on the statistical law of large numbers to smooth independent traffic flows. In contrast, Handley and Steinbrecher explain that distributed AI training operates as a single, tightly synchronized computational task across thousands of GPUs. Because GPUs operate in lockstep, any delayed packet, bit flip, or congested link forces the entire cluster to pause, making overall training speed dictated by the absolute worst-case link—the 100th percentile (P100) tail latency. This vulnerability is exacerbated by the explosion of internal "east-west" data center traffic and high component density. A single data center contains several thousand switches and millions of optical links and lasers. Steinbrecher notes that under independent failure assumptions, doubling cluster size halves the mean time to failure (MTTF). In synchronous clusters, transient link congestion or routine physical failures frequently crash training runs or cause substantial idle compute waste. Eliminating Congestion with Packet Trimming and Static Routing (15:38) To overcome routing bottlenecks, OpenAI developed Multipath Reliable Connection (MRC). The protocol sprays packets across multiple network paths to balance loads evenly. To handle the packet reordering that spraying introduces, MRC employs "packet trimming": when switch queues overflow, the switch drops the payload but forwards the packet header to the destination. This header immediately notifies the receiver of congestion, allowing instant retransmission requests without ambiguity over whether packets were lost or merely delayed. MRC also provides endpoint-driven resilience, which the engineers describe as "self-annealing." Endpoints independently detect degraded paths and reroute around them in milliseconds, bypassing the multi-second convergence delays associated with traditional routing protocols like BGP. Because path selection is managed at the endpoints using IPv6 segment routing, OpenAI disabled switch-level dynamic routing protocols entirely. Switches operate with fixed, static routing tables configured at boot, eliminating control-plane software failures and enabling clusters to run uninterrupted even during active physical cable maintenance. Co-Design, Flatter Topologies, and Open Standards (7:12) OpenAI views infrastructure and model training as an interdependent problem requiring continuous "co-design." Workload systems engineers sit alongside AI researchers and remain on call during major training runs, tailoring server topologies directly to model requirements. This co-design revealed that multipath spraying allows data centers to build flatter networks with fewer switch layers, lowering capital expenditures and directing more data center wattage directly to GPUs rather than network equipment. To prevent industry fragmentation, OpenAI standardized MRC in collaboration with Microsoft, NVIDIA, Broadcom, AMD, and Intel, releasing the specification openly through the Open Compute Project (OCP). Handley and Steinbrecher state that unified Ethernet standards are necessary to preserve shared supply chains and make massive collaborative deployments, such as Microsoft's "Fairwater" and "Stargate" data centers, feasible. Multimodal Latency Demands and Physical Constraints (31:55) As AI models advance into multimodal architectures incorporating video and imagery, the volume of data and sensitivity to network latency increase sharply. Steinbrecher points out that as GPU compute times shorten, the window for inter-GPU data transfer narrows, amplifying the efficiency penalty, or "regret," of network delays. Furthermore, increasing network paths without deterministic routing worsens tail latency due to the "ball-and-bin" variance problem, where the spread between average and worst-performing paths widens. While space-based AI supercomputing has been proposed to circumvent terrestrial data center constraints, Handley and Steinbrecher dismiss orbital clusters as practically unfeasible. Speed-of-light delays introduce severe latency penalties for tightly synchronized parallel workloads like Stargate. Moreover, given the steady rate of optical and hardware failures that require daily, hands-on maintenance by technicians from Microsoft and Oracle on Earth, an orbital cluster would rapidly accumulate unserviceable hardware. OpenAI's networking strategy seeks to make cluster protocols entirely invisible to researchers while maximizing compute efficiency per watt. However, maintaining scaling velocity requires resolving strict physical constraints, particularly tail latency in multimodal workloads and the inevitable hardware degradation across millions of terrestrial optical components.

OpenAI网络与系统团队指出，前沿AI模型训练本质上属于高度紧密同步的计算任务，传统基于大数定律和动态路由的互联网式数据中心网络已成为算力扩展的主要瓶颈。为此，OpenAI联合行业伙伴研发了多路径可靠连接（MRC）协议，通过数据包喷洒、数据包裁剪与端点自愈机制重塑网络架构，以保障超大规模GPU集群的训练效率与系统稳定性。 AI同步计算引发的尾部延迟瓶颈 (5:04) Mark Handley指出，传统数据中心网络沿袭互联网架构，依赖统计学上的大数定律来平滑海量独立用户的流量波动。然而，前沿AI训练的负载逻辑截然相反，需要数千颗顶尖GPU在单一任务上进行高精度的步调同步。Greg Steinbrecher解释称，在并行训练中，GPU之间的通信本身就是计算过程的组成部分；只要单颗GPU因宇宙射线发生比特翻转或遭遇网络拥堵，整个集群的其他计算卡都必须同步停滞等待，造成巨额算力浪费。 随着数据中心内部“东西向”通信流量发生指数级爆炸，单座建筑内往往部署有数千台交换机并衍生出数千条潜在物理路径。在常规路由策略下，随机路径选择极易导致局部链路拥塞。由于同步计算对第100百分位（P100）的最差情况极度敏感，最慢的单条链路直接决定了整个训练任务的耗时；而根据“球与箱”模型的统计方差规律，单纯增加网络路径若不配合确定性路由，反而会进一步恶化尾部延迟。 硬件故障必然性与软硬件协同设计 (7:12) 在超大规模AI集群中，硬件损坏已成为不可避免的统计确定性事件。Greg Steinbrecher指出，在组件故障相互独立的数学模型下，集群规模每扩大一倍，系统的平均无故障时间（MTTF）就会减半。更为严峻的是，单颗GPU连接的网络适配器包含带有多路激光器的光学收发模块，单体数据中心内的光学链路多达数百万条，网络组件数量远超计算芯片本身，导致硬件时刻处于物理失效的边缘。 为应对这一现实，OpenAI打破了传统互联网时代将基础设施视为孤立“算力海洋”的分离模式，推行软硬件深度“协同设计”（Co-Design）。底层网络工程师直接与前沿模型算法研究员并肩协作，深度跟进关键训练轮次并实时处理故障中断。这种紧密闭环使系统架构能够直接适配模型算法的实际通信特征，从源头上消除资源瓶颈与重启开销。 MRC协议核心机制：数据包喷洒与裁剪 (15:38) 为从根本上化解网络局部热点，OpenAI团队主导开发了多路径可靠连接（MRC，即Multi-Path RoCE）协议。该架构在总容量充足的网络拓扑中应用“数据包喷洒”（Packet Spraying）技术，将数据流均匀散布到所有可用路径进行负载均衡，从而将潜在拥塞严格限制在多发一收的终点汇聚场景。针对多路径传输必然导致的乱序到达与丢包检测难题，团队采用了创新的“数据包裁剪”（Packet Trimming）机制。 常规交换机在队列溢出时会直接丢弃整个数据包，迫使发送端陷入被动的重传超时等待。而在数据包裁剪机制下，拥塞交换机仅切除数据包的有效载荷（Payload），但仍将微小的头部信息转发至目的地。接收端凭借收到的包头可立即确认该数据包遭遇拥塞受损而非途中延迟，进而瞬时向源端发起精确重传请求，彻底消除了乱序与真正丢包之间的判断歧义，显著压缩了端到端延迟。 毫秒级自愈与全静态路由极简架构 (18:32) MRC协议在网络容错机制上实现了类似“自退火”（Self-Annealing）的端点弹性。传统网络高度依赖BGP等分布式路由协议在交换机之间逐跳泛洪广播链路状态，其全局收敛通常需要数秒乃至数十秒，期间昂贵的计算集群只能闲置。MRC打破了这种分布式协调的限制，网络端点能够在数毫秒内独立检测出故障路径并自主绕行，即使在机房物理布线频繁插拔变动的极端环境下，上层训练任务也完全无感知运行。 依托端点自主绕障能力，OpenAI在大规模生产集群中彻底停用了交换机上脆弱复杂的动态路由协议，转向全局静态路由。交换机仅需在启动时加载固定配置，运行期间路由表永不变更。配合IPv6段式路由（Segment Routing）技术，数据包头部在源端便直接写入了途径交换机的确定性序列，将中间交换机简化为纯粹的转发管道，既规避了网络控制面崩溃的风险，也消除了运维团队排查链路震荡的沉重负担。 开源生态联盟与地面物理现实的坚守 (25:05) 为了防止行业因碎片化的私有专有硬件而割裂，OpenAI通过开放计算项目（OCP）将MRC技术规范完全开源，并联合微软、英伟达、博通、AMD及英特尔等芯片与基建厂商共同推动标准化。团队强调，底层基础设施属于整个AI产业的“共同命运”，基于标准以太网构建扁平化网络不仅大幅减少了中间交换机层级与资本开支，还能降低网络设备的寄生功耗，使数据中心更多电能直接转化为GPU的有效算力。 针对部分舆论构想将AI算力发射至太空以突破地面环境限制的设想，团队从工程与物理规律出发明确予以否定。紧密同步训练受制于严苛的光速传播延迟，且地面集群每日均依赖微软和甲骨文等合作伙伴的专业工程师现场高频更换损耗的光学器件与芯片。在缺乏即时维修可能性的轨道环境中，昂贵且精密的高密度GPU集群将在极短时间内因硬件损耗累积而彻底瘫痪。 Mark Handley与Greg Steinbrecher总结认为，随着前沿多模态大模型对数据吞吐和极低延迟的要求成倍递增，留给网络底层的数据传输时间窗口正在被极限压缩。MRC协议通过软硬件协同与端点智能重塑了超算互联范式，但随着集群规模向更庞大的体量扩张，如何在数百万光链路的物理故障常态下持续维持极低的尾部方差与严苛的能效比，仍是决定AI算力扩展上限的核心工程挑战。

## Background 该内容由 OpenAI 官方 YouTube 频道于 2026 年 6 月 3 日发布。视频为《OpenAI Podcast》第 18 期，主持人 Andrew Mayne 与 OpenAI 核心网络专家 Mark Handley 及工作负载系统专家 Greg Steinbrecher 展开对谈，探讨了训练前沿大模型面临的网络瓶颈，以及 OpenAI 联合芯片巨头推出的全新超算网络设计 MRC（多路径可靠连接）。 ## TL;DR 随着 AI 模型规模爆炸式增长，海量 GPU 的强同步训练使得网络性能不再取决于平均吞吐，而是受制于极端的 P100 尾部延迟与百万级光链路故障。为了解决传统网络动态收敛缓慢引发的集群停摆，OpenAI 联手英伟达、博通、AMD、英特尔及微软研发了“多路径可靠连接”（MRC）技术，通过报文喷射（Packet Spraying）与报文剪裁（Packet Trimming）实现毫秒级端点自愈，并使数据中心交换机退化为全静态路由。该方案已被推向 OCP 开源标准以避免全行业重复造轮子，使基础设施彻底隐形于算法研究员之后，全力赋能下一代超大模型的稳定训练与物理极限突破。 ## Atmosphere & style 对谈散发着硬核系统工程与底层科研深度交融的实战气息，既有学术泰斗的技术沉淀，又有生产一线“救火”工程师的生动幽默。Mark 的学者从容与 Greg 充满激情的调侃形成了极具张力的互补，两人从早年排障的“恐怖回忆”谈到如今即使光纤施工乱晃也依然平稳运行的自豪，传递出一种在极限工程攻坚中取得重大突破的笃定感与“氛围大好”（vibes are very good）的乐观情绪。整体叙事平实而震撼，将深奥的网络拓扑与不可逾越的光速物理定律娓娓道来，既有务实的系统工程洞察，又不失面向未来的远大格局。 ## Bulletin - [00:00](https://www.youtube.com/watch?v=TiW96H5HmAw&t=0s) OpenAI 联合英伟达、博通、AMD、英特尔与微软共同研发并落地多路径可靠连接（MRC）网络协议，已应用于其最新前沿大模型的分布式训练中。 - [15:19](https://www.youtube.com/watch?v=TiW96H5HmAw&t=919s) MRC 核心基于“报文喷射（Packet Spraying）”消除核心拥塞，并结合“报文剪裁（Packet Trimming）”让接收端在队列溢出时仅保留微报头，从而彻底消除丢包歧义并实现毫秒级定向重传。 - [20:38](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1238s) 依靠端点自主探测与极速自退火机制，超算交换机可彻底关闭 BGP 等动态路由协议而全面采用全静态路由，将故障收敛时间从传统网络的数秒降低至毫秒级。 - [25:05](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1505s) OpenAI 决定通过开放计算项目（OCP）将 MRC 捐赠为全行业公开标准，以避免专有网络方案带来的供应链碎片化与重复造轮子。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=TiW96H5HmAw&t=0s) 衡量超算网络成功的终极标准是底层透明化：当系统真正高效稳定时，模型研究人员甚至不需要知道集群底层运行的具体网络协议。 - [04:34](https://www.youtube.com/watch?v=TiW96H5HmAw&t=274s) AI 分布式训练中跨节点通信是数学计算的有机组成部分，所有 GPU 步调一致（Lockstep）协同运算，导致木桶效应被推向极致。 - [09:39](https://www.youtube.com/watch?v=TiW96H5HmAw&t=579s) 传统数据中心依靠大数定律统计复用平滑流量的假设失效，AI 超算网络的整体吞吐完全被最慢的链路（P100 最差情况尾部延迟）所支配。 - [10:59](https://www.youtube.com/watch?v=TiW96H5HmAw&t=659s) 由于无收敛网络包含数百万条光链路，系统规模每扩大一倍硬件平均无故障时间（MTTF）便直接减半，必须在协议层构建容错机制。 - [15:19](https://www.youtube.com/watch?v=TiW96H5HmAw&t=919s) 报文喷射（Packet Spraying）将数据包粉碎分摊至所有可用物理路径，消除了核心网内部由于哈希碰撞形成的局部拥塞热点。 - [15:19](https://www.youtube.com/watch?v=TiW96H5HmAw&t=919s) 报文剪裁（Packet Trimming）在交换机缓冲区满时仅丢弃 Payload 并转发微型 Header，消除了乱序与真正丢包之间的推断歧义。 - [18:55](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1135s) MRC 摒弃了基于 BGP 等多跳广播的慢速集中收敛，赋予网络终端毫秒级感知链路失效并自主绕开故障路径的“自退火”能力。 - [20:38](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1238s) 在端点具备自主选路弹性后，交换机可全面固化静态路由表，彻底消除了动态路由控制面可能引发的脑裂与软件崩溃隐患。 - [20:38](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1238s) 在 Fairwater 超算中心施工阶段，即使机房光纤因频繁物理施工而剧烈闪断，上层模型训练任务仍平稳运行且无工程师感知。 - [25:05](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1505s) 将 MRC 规范全面开源并捐赠给 OCP 是为了打破各硬件巨头的私有技术壁垒，依托开放以太网生态加速全行业 AI 基础设施建设。 - [27:21](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1641s) 光速不可逾越的物理限制导致端口速率演进时在途数据量激增，MRC 原生构建于以太网之上为未来数代太比特硬件预留了扩展空间。 - [28:45](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1725s) MRC 结合 IPv6 段路由（Segment Routing）将智能推向主机边缘，中间层交换机退化为极简转发设备，显著提升了工程确定性。 - [30:36](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1836s) 多路径喷射的高效路径利用率大幅压缩了传统胖树架构的交换机层级，降低了硬件能耗，提升了数据中心每瓦特的有效模型算力。 - [34:59](https://www.youtube.com/watch?v=TiW96H5HmAw&t=2099s) 由于光速时延与太空中无法承受的高昂硬件故障维护成本，将数万张 GPU 的超大模型训练部署到轨道空间在工程上并不可行，地面超算仍是核心路线。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=TiW96H5HmAw&t=0s) 引言与超级计算机训练的新突破 在当今的前沿人工智能研发中，训练超大规模模型绝非单纯堆叠更多 GPU 那么简单。主持人 Andrew Mayne 开门见山地指出，本期节目将深入探讨如何让超级计算机在模型训练中发挥出前所未有的效能。OpenAI 核心网络团队（Core Networking）的 Mark Handley 与工作负载系统团队（Workload Systems）的 Greg Steinbrecher 共同参与讨论。他们带来了一项关键的技术突破——这项网络设计已经应用于 OpenAI 最新前沿模型的训练之中，不仅大幅提升了训练效率，也消除了大模型继续扩展尺寸（Scaling）所面临的核心瓶颈之一。 在团队看来，衡量这项网络技术成功的终极标准其实极为纯粹：“当我们取得最终胜利时，研究人员甚至不需要知道当前集群到底在跑什么网络协议。”系统的底层复杂性被彻底封装，数万块全球顶级的 GPU 得以像一台运转精密的单体计算机一样，协同执行超高强度的单一任务，让更智能的模型以更快的速度诞生。 - **核心使命**：将全球最强 GPU 组成的庞大集群聚合为单一高效的计算实体，消除网络阻塞与扩展阻碍。 - **技术成果**：OpenAI 联手 AMD、博通（Broadcom）、英特尔（Intel）、微软（Microsoft）和英伟达（Nvidia）共同研发了多路径可靠连接（Multipath Reliable Connection，简称 MRC）。 --- ### [00:38](https://www.youtube.com/watch?v=TiW96H5HmAw&t=38s) 从量子计算与互联网协议走入 AI 系统的探索之路 Greg Steinbrecher 与 Mark Handley 各自沿着截然不同却高度互补的路径汇聚到 OpenAI 的底层系统研发中。Greg 本科攻读物理与数学，他着迷于复杂系统的内在规律，尤其热衷于“构建一个彻头彻尾是谎言、却能精准揭示系统本质的极简模型”，并以此建立直觉去理解庞大系统。随后他在攻读博士学位时尝试建造量子计算机，面对主持人的打趣，Greg 自嘲道：“量子计算机目前根本无法正常工作，所以更谈不上扩展性。它未来或许能行，但现在不行。”在研究用于控制光子量子比特的芯片时，Greg 敏锐地发现这些芯片在拓扑上与网络交换机惊人相似。他意识到学术界对真实数据中心工作负载的理解往往停留在脱离实际的“玩具模型”，于是争取到业界公司的资助，在博士后半段全职投身真实网络硬件研究。他发现常规数据中心硬件蕴藏着巨大的未开发潜能，根本不需要玄乎的光学计算芯片。随着 AI 热潮爆发，Greg 从编写底层集群仿真模拟软件，顺理成章地转向直接编写 GPU 间的底层通信软件，并在一年多前加入 OpenAI，专注于提升 GPU 算力利用率、网络去瓶颈以及硬件故障恢复。 Mark Handley 则在网络学术界深耕数十年。在不为 OpenAI 工作的日子里，他是伦敦大学学院（UCL）的计算机科学教授。他的网络研发传奇始于互联网发展的早期阶段，当时他致力于在算力极度羸弱的计算机上实现网络视频会议。团队最终制定的多项关键通信标准，至今仍是全世界每一部智能手机在 4G 和 5G 网络中建立音视频通话的底层基石。然而，跨行业制定国际标准的漫长拉锯让他深感疲惫：“标准化的最大痛点在于你必须让全世界所有参与者都达成一致，这通常需要耗费极为漫长的时间。”相比之下，数据中心网络具有一个无与伦比的优势——创新的规则仅需在同一栋建筑内达成一致即可快速落地。正是这种极高的迭代自由度与紧迫的现实需求，促使 Mark 全身心投入到数据中心网络架构的探索中。 - **Greg Steinbrecher 的轨迹**：物理与数学背景 $\to$ 量子计算芯片光控研究 $\to$ 发现传统网络硬件巨大优化空间 $\to$ GPU 集群通信模拟与底层软件研发 $\to$ OpenAI 工作负载系统。 - **Mark Handley 的轨迹**：伦敦大学学院（UCL）网络系统教授 $\to$ 互联网早期视频会议与流媒体传输协议先驱（相关标准奠定了现代 4G/5G 通信基石） $\to$ 转战数据中心网络前沿。 - **工作负载系统核心目标**：监控模型训练是否达到理论峰值速率、消除网络传输停顿、处理硬件静默故障与异常重启，榨干顶级硬件的“最后一盎司性能”。 --- ### [04:34](https://www.youtube.com/watch?v=TiW96H5HmAw&t=274s) 为什么 AI 模型训练对网络提出了截然相反的要求 随着大模型参数急剧膨胀，技术界在极短时间内将 GPU 集群规模推向了前所未有的量级。Mark 解释道，传统数据中心网络的底层设计哲学脱胎于公共互联网：网络承载着数以百万计的并发用户，每个人发起独立的小型通信会话。在传统网络架构中，通信流越是繁多杂乱，大数定律（Statistics of large numbers）就越能发挥作用，不同会话的波峰与波谷互相抵消，使得整体网络流量在宏观上变得平滑可预测。 然而，AI 模型的并行分布式训练却呈现出完全相反的极端形态。它并非成千上万个互不相干的独立对话，而是让成千上万块全球运算速度最快的 GPU 为同一个目标协同工作。Mark 强调了一个被很多人忽视的核心事实：“GPU 之间的通信本身就是计算过程的有机组成部分。”在每个训练步骤（Step）中，所有 GPU 必须交换权重与梯度，达成全局一致才能进入下一步计算。这种高度同步的集体步调（Lockstep）堪称“你能想象投放在网络上最糟糕、最严苛的工作负载”。一旦某一块 GPU 因局部网络拥塞或单条链路微小抖动而慢了半拍，集群中所有其他价值连城的计算卡都必须被迫停下空转等待；若有一颗来自外太空的高能宇宙射线击中某张卡导致比特翻转（Bit flip），整个计算步骤可能全部作废，系统必须暂停排查甚至被迫回滚到先前的检查点（Checkpoint）。 ```mermaid flowchart TD accTitle: 传统互联网流量与 AI 分布式训练通信模式对比 accDescr: 对比展示传统互联网流量的大数定律平滑特性与 AI 训练中全互联同步依赖特性 subgraph Traditional["传统互联网 / Web 数据中心"] U1["用户 / 任务 A"] --> S1["共享网络交换架构"] U2["用户 / 任务 B"] --> S1 U3["用户 / 任务 C"] --> S1 S1 --> R1["统计复用：流量波峰波谷自然抵消，表现平滑"] end subgraph AITraining["AI 超级计算集群"] G1["GPU 节点 1"] <--> SW["大规模多层交换网络"] G2["GPU 节点 2"] <--> SW G3["GPU 节点 N"] <--> SW SW --> Sync["强同步集合通信 (All-Reduce / 等)"] Sync --> Bottleneck["木桶效应：单点慢 / 丢包 / 翻转 = 全局算力空转"] end ``` - **通信即计算**：AI 训练中跨节点网络交换并不是附加的 I/O 操作，而是多维张量数学运算不可分割的组成部分。 - **木桶短板代价**：单个 GPU 掉队（Straggler）或通信阻塞会导致数万张算力卡瞬间丧失生产力，形成高昂的时间与能耗浪费。 - **传统互联网与 AI 网络的本质差异**：前者依赖大数定律通过无序多流实现统计平滑；后者面临突发性、全量同步、大包密集的极端网络冲击。 --- ### [07:12](https://www.youtube.com/watch?v=TiW96H5HmAw&t=432s) 系统与算法的深度协同：重新构想数千台交换机的数据中心 AI 技术的爆发不仅颠覆了算法本身，也彻底重构了科技巨头设计数据中心的方法论。Greg 指出，在传统的云计算与 Web 时代，底层基础设施团队与顶层业务负载几乎完全脱节，基建团队的唯一目标就是冷冰冰地向外提供一片无差别的“算力海洋”（Ocean of compute）。然而在 OpenAI，系统工程师、网络专家必须与模型算法研究员坐在一起，进行全栈深度联合设计（Co-design）。Greg 的团队就紧挨着核心算法研究员办公，每天讨论算法负载如何以最小开销映射到物理服务器拓扑上，并且系统团队还要全天候轮值大型训练任务的 On-call 警报——“如果大模型训练任务在半夜崩溃且无法自动恢复，我们就会被直接叫醒。” 正是在这种彻夜排障与实战打磨中，团队真切体会到旧有网络体系的巨大痛苦，并下定决心彻底重塑数据中心。Mark 深入解析了超算网络的物理拓扑困境：由于数万块 GPU 必须在毫秒级的同步周期内同时向外吐出海量数据，世界上根本不存在任何单台交换机（乃至单一交换层级）能够提供如此庞大的背板带宽。数据中心必须构建极为复杂的“交换机层级之上的层级”（Hierarchies of hierarchies of switches），整座数据中心建筑内往往部署了数千台高密度网络交换机。 这种极度纵深的网状拓扑带来了极其棘手的新挑战：从集群中的某一块 GPU 发送数据到另一块 GPU，在物理网络中存在数千条完全并行的可选路径。传统网络往往依赖哈希算法或随机选择一条固定路径来传输特定数据流，但在突发性超饱和的 AI 流量冲击下，随机选路极易导致严重的路径碰撞（Hash collision）与网络拥塞。如何在由数千台交换机交织成的浩瀚迷宫中，为最高并发的数据流实时挑选最优路径，成为了下一代数据中心必须跨越的核心技术门槛。 - **组织协同进化**：告别传统互联网“基础设施提供算力、算法团队独自建模”的隔离模式，转向网络、系统、编译器与模型算法的深度协同设计（Co-design）。 - **物理硬件规模**：单个超算中心建筑内包含数千台顶级网络交换机，形成多层胖树（Fat-Tree）等复杂层级结构。 - **核心架构挑战**：端到端节点间存在成千上万条潜在物理网络路径，单流固定路径选路极易引发链路冲突，迫切需要革命性的多路径流量分发协议。 ### [09:39](https://www.youtube.com/watch?v=TiW96H5HmAw&t=579s) 传统统计复用失效与“P100 最差情况”的致命约束 互联网早期赖以生存的核心设计理念是“统计复用”（statistical multiplexing）。在传统网络中，路线选择带有一定的随机几率：如果运气好，没有其他流量争抢同一条路径，传输就会畅通无阻；如果有两个用户撞在同一条路径上，速度就会慢下来；若是十个人同时挤上一条路径，网络就会变得极其缓慢。但在传统网络里，大数定律（law of large numbers）能够抚平这些波动，平均性能足以支撑常规业务。然而，当工程师试图将这种设计搬进现代 AI 数据中心时，它彻底失效了。 ```mermaid flowchart LR accTitle: 传统网络与 AI 训练网络的性能决定机制对比 accDescr: 对比传统网络关注的平均值统计与 AI 分布式同步训练中取决于 P100 最慢链路的瓶颈机制。 subgraph Traditional["传统网络模式：均值统计"] T1["海量随机用户流量"] --> T2["统计复用平滑波动"] T2 --> T3["依赖平均吞吐（Average）"] end subgraph AISync["AI 分布式训练：同步步调（Lockstep）"] A1["数万张 GPU 并行计算"] --> A2["数十万条网络流同步传输"] A2 --> A3["全网最堵塞链路（P100 瓶颈）"] A3 --> A4["整批 GPU 停滞等待最慢节点"] end ``` AI 训练负载具有高度严格的“同步性”（synchronous nature），整个集群所有节点都在“步调一致”（lockstep）地推进。数以千计乃至数万张 GPU 相互通信时，会产生数十万条网络流，遍布数万条物理链路。在这样的系统里，“成对 GPU 之间的平均通信延迟有多快”完全无关紧要，决定全局训练速度的永远是全网表现最差的那一条链路。Greg 强调，AI 网络不再享有仰赖平均指标的奢侈，整个系统的性能完全被尾部延迟（tail latency）所支配，即所谓的“尾部的尾部”——P100（第 100 百分位值）。只要任何一条链路发生拥塞，所有价值连城的 GPU 都必须原地停滞，等待那条最慢的链路传输完毕。 - **平均性能彻底失效**：GPU 必须等齐所有梯度与权重数据才能进入下一步计算，网络性能完全取决于最糟糕的那条物理链路。 - **大数定律的失灵**：无法靠流量随机分布平摊拥塞，极端尾部延迟（P100）成为约束整个超算吞吐的硬性瓶颈。 ### [10:59](https://www.youtube.com/watch?v=TiW96H5HmAw&t=659s) 规模的诅咒：数百万光链路下的平均无故障时间（MTTF）雪崩 当网络规模被推向极致时，硬件故障就不再是偶发事件，而成了常态。哪怕数据中心采购业内顶级的网络设备和高品质光模块，交换机仍会因未知错误陷入混乱而被迫重启，物理链路也随时可能中断。在 P100 机制统治的网络中，一旦某条链路断开，路由协议需要耗费时间重新收敛，将流量迁移到备用路径上。这个切换过程带来的“瞬时卡顿”（glitch）在传统网络中无伤大雅，但在大规模同步训练中，一次通信中断就足以导致整个耗资数千万美元的分布式训练任务彻底崩溃。当主持人追问这种架构究竟会在哪里崩溃时，Mark 干脆利落地回答：“到处都会崩（Everywhere）。” 更残酷的是硬件可靠性的数学法则：在假设故障彼此独立的前提下，系统规模每扩大一倍，平均无故障时间（MTTF）就会直接减半。而在 AI 数据中心内，网络组件的数量往往比 GPU 本身多出几个数量级。即使是最基本的单张 GPU 连接到网卡，如果使用光收发模块，发送端内部通常包含 4 个激光器，对端交换机收发模块又对应 4 个激光器——这意味着仅在 GPU 接入第一级交换机的初始跳数上，光学激光器的数量就已经比 GPU 多出一个数量级。再加上多级复杂交换网络架构（Multi-tier switching fabrics），整个建筑内部汇集了数以百万计的光纤链路。 - **超高网络组件密度**：为了保障庞大算力不被“饿死”（starve），网络带宽无法做收敛设计，导致单体数据中心内光链路数量高达数百万条。 - **无法依靠补丁修复**：传统协议在故障转移时收敛过慢，若强行在旧协议上缝缝补补，根本无法避免训练任务遭遇致命中断。 - **流量爆炸的历史伏笔**：Greg 回忆起他在 2017 年参加 OFC（光纤通信大会）时的一幕：当时 Facebook 的技术专家展示了一张图表，显示其数据中心内部的互联流量正在呈指数级爆炸，而发往终端用户的外部流量却几乎保持平缓。而在那之后登场的 AI 训练，直接将这种原本就令工程师头疼的系统挑战“一路旋扭拉到了 11 档爆表级别（cranks them up to 11）”。 ### [15:19](https://www.youtube.com/watch?v=TiW96H5HmAw&t=919s) MRC 的核心机制：报文喷射（Packet Spraying）与报文剪裁（Packet Trimming） 为了彻底解决拥塞与单点故障对训练造成的毁灭性打击，OpenAI 联手产业界伙伴推出了“多路径可靠连接”（Multipath Reliable Connection，简称 MRC）。这项技术的基础逻辑建立在两大核心技术创新的协同运作之上： ```mermaid flowchart TD accTitle: MRC 报文喷射与报文剪裁处理机制 accDescr: 展示 MRC 协议通过报文喷射均衡负载，并在交换机缓冲区满时剪裁净荷保留报头快速重传的过程。 subgraph Ingress["发送端：报文喷射 (Packet Spraying)"] Src["发送端 GPU 数据流"] --> Spray["报文被细粒度喷射到多条平行路径"] Spray --> P1["路径 A"] Spray --> P2["路径 B"] Spray --> P3["路径 C"] end subgraph Fabric["交换网络：消除热点"] P1 --> SW1["交换机队列正常 -> 完整转发"] P2 --> SW2["交换机队列溢出 (拥塞点)"] P3 --> SW3["交换机队列正常 -> 完整转发"] end subgraph Trimming["报文剪裁机制 (Packet Trimming)"] SW2 -- "丢弃大体积 Payload" --> Trim["切除有效载荷，仅保留微型 Header"] Trim -- "传递截短报文" --> Dst SW1 --> Dst["接收端 GPU"] SW3 --> Dst end subgraph Recovery["接收端：消除模糊性并瞬时重传"] Dst --> Detect["收到被剪裁报头：明确知晓拥塞，非链路乱序"] Detect --> NACK["即刻向发送端请求极速定向重传"] end ``` 第一项核心机制是**报文喷射（Packet Spraying）**。传统路由倾向于让整条流走固定路径以避免乱序，但 MRC 将数据包均匀喷射打散到网络拓扑中能够提供的所有平行路径上。只要物理拓扑设计了足够的无阻塞带宽，核心网络内部的局部拥塞热点就会被彻底消除，全网只剩下一个不可避免的拥塞点——那就是多个发送者同时向同一个目的地发送数据时的“入端收敛”（incast）。然而，报文喷射会带来副产物：不同路径传输时延差异导致数据包乱序到达（jumbled up）。如果此时发生了网络拥塞和丢包，接收端将陷入深层模糊性：它无法确定某个数据包究竟是因为拥塞彻底丢失了，还是仅仅因为走了较慢的路径而迟到了。 为了打破这种进退两难的僵局，MRC 引入了第二项机制——**报文剪裁（Packet Trimming）**。在传统网络中，交换机队列一旦溢出，处理方式就是直接丢弃整个数据包；而在 MRC 中，当交换机即将遭遇队列溢出时，它不会整包丢弃，而是切掉占体积的大部分数据有效载荷（Payload），仅把几微字节的微型数据报头（Header）继续转发给目的端。接收端在收到这个被剪裁的报头后，能够瞬间心领神会：“这个包确实因为拥塞被丢弃了，我不需要继续空等它乱序到达”。目的端随即发出即时重传请求，发送端迅速补发缺失载荷。这种机制彻底消除了超时等待和重传歧义，使乱序容忍与超快拥塞响应得以完美共存。 - **多路径绝对负载均衡**：通过 Packet Spraying 将数据流粉碎分摊至所有可用物理链路，消除了核心网交换机内部由于哈希碰撞形成的拥塞热点。 - **彻底消除丢包歧义**：通过 Packet Trimming 保证控制信令畅通，接收端无需通过漫长的定时器超时猜测丢包，重传几乎是即时触发的。 ### [17:00](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1020s) 几轮往返（RTT）自愈链路故障与“氛围大好”的工程落地 谈到 MRC 对终端用户与 AI 研究人员的实际价值，Greg 表示最直接的反馈就是用户将以更快的速度用上来自 OpenAI 的性能更强、更智能的前沿大模型。MRC 加速了从研发探索到正式上线的整条管线：研究员在提交海量 GPU 训练任务时，再也不用提心吊胆害怕网络丢包导致整组作业崩溃，也无需关注调度系统将自己分在哪个机架，更不必担心与隔壁机架其他人的高负载作业发生冲突。主持人打趣地问：“所以现在整体氛围很好？（The vibes are good?）”Mark 笑着附和：“氛围非常好，感觉棒极了！（The vibes are very good.）” 从工程研发的历程来看，这项技术并不是平地惊雷的凭空捏造，而是吸纳了学术界和工业界过去几十年中的多项网络研究成果，将其巧妙地组合并工程化实现。OpenAI 与核心软硬件伙伴建立攻坚组织，在拿到首批支持硬件后的短短几个月内，就完成了端到端软硬件部署，并在生产集群上全速训练真实模型。 更为惊艳的是其强大的容错自愈能力。由于所有数据流都被均匀喷射在海量路径上，当网络中某根物理链路突然断开时，穿过该区域的所有数据流固然都会受到一丝波及，但受影响的程度微乎其微。MRC 能够在短短几个往返时间（RTT, Round Trip Times）内迅速感知故障链路并彻底停止向其注入流量。原本足以让分布式训练彻底停摆的单点网络断连，在此被降维为无声无息的微小波动，整个集群的训练甚至察觉不到任何异常便能全速继续向前推进。 ### [18:55](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1135s) 告别分布式收敛：MRC 的毫秒级独立自愈 传统网络在链路出现硬件故障时，其应对机制本质上是一个代价高昂的分布式系统共识过程。当一根光纤被拔掉或端口失效，链路两端的交换机固然能察觉，但它们必须将故障信息一级一级向邻近节点通告，往往依赖如 BGP（边界网关协议）这类类似八卦谣言（Gossip）的协议，在网络中经过五跳甚至七跳的逐级扩散，才能让远端交换机获知该路径不可达并重新计算路由。这一收敛过程在尾部延迟严重时动辄需要数秒乃至数十秒。对于动辄投入数万张 GPU 进行协同训练的大模型任务来说，任何一次链路震荡引发的数秒卡顿，都会导致整个集群暂停计算陷入等待。而在超大规模集群中，这类小故障的发生频率呈指数级上升，累积的闲置算力成本不可估量。 OpenAI 研发的 MRC（Multipath Reliable Connection，多路径可靠连接）彻底打破了对全局集中协调或逐跳网络收敛的依赖。格雷格·斯坦布雷彻（Greg Steinbrecher）戏称马克·汉德利（Mark Handley）把这项技术的颠覆性“说得太保守了”（"Mark has mildly undersold this... Mark, come on, man"）。MRC 的核心机制在于让网络中的每一个终端节点（Endpoint）自主探测可用性，并在毫秒级的时间尺度内独立做出反应——一旦发现某条路径不通，立刻在本地停止向该路径发包，无需等待中心权威下发指令，也无需等待网络拓扑重新收敛。集中式控制中心往往是灾难性的单点故障（SPOF），而 MRC 赋予端点的这种极速“自退火”（self-annealing）能力，让整个通信架构对局部软硬件故障实现了无感免疫。 ```mermaid flowchart TD accTitle: 传统网络收敛与 MRC 端点自愈对比 accDescr: 展示了传统网络依靠多跳 BGP 广播的慢速收敛与 MRC 端点毫秒级自主旁路的流程差异。 subgraph TRAD["传统网络故障处理机制（耗时数秒至数十秒）"] T1["物理链路断开"] --> T2["直连交换机感知故障"] T2 --> T3["多跳 BGP Gossip 逐级扩散"] T3 --> T4["全网拓扑重新计算收敛"] T4 --> T5["全局 GPU 训练停滞等待"] end subgraph MRC_FLOW["MRC 端点自愈机制（耗时数毫秒）"] M1["数据包传输失败"] --> M2["网卡/端点独立检测异常"] M2 --> M3["本地路由直接切断该路径"] M3 --> M4["无缝轮换至健康冗余链路"] M4 --> M5["GPU 计算全程不中断保持满载"] end ``` ### [20:38](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1238s) 施工现场的严苛检验与全面转向静态路由 MRC 的真正实战表现是在全新超算数据中心建设期得到的验证。在部署代号为“Fairwater”的超算中心时，机房内部正处于密集的硬件进场与物理组装阶段。成千上万根光纤在不同机房大厅之间交错穿插，现场施工技术人员在组装一个机厅的同时，难免会碰动甚至插拔相邻机厅共享管道的光缆。这种极度频繁的人为干扰导致物理链路起伏极其剧烈，断线频率远超正常运行状态。令人惊叹的是，依托 MRC 的端点自愈机制，OpenAI 在硬件到货短短数月内便直接启动了模型训练；尽管底层物理连接不断闪断，训练任务却如常运行，工程师们“根本不需要在乎，甚至完全没注意到”（"We did not care. We didn't even notice. MRC just took care of it"）。 在验证了 MRC 绕过故障路径的端点弹性后，团队做出了一个极为激进但彻底的技术决策：彻底关闭数据中心交换机上的动态路由协议。动态路由协议、交换机操作系统及其控制面本身就极其臃肿脆弱，经常因软件缺陷或状态不一致引发次生故障。团队意识到，既然底层的 MRC 终端已经能够自主试探出哪些路径畅通、哪些路径阻断，交换机本身就不再需要承担动态探测和路由重算的负担。于是，在最大规模的超算网络中，OpenAI 全面采用了完全静态的路由配置（completely static routing）：交换机仅在开机引导时加载一次固定转发表，此后无论网络发生什么断损，路由表绝不再变更。这一简化直接将沉重、不可靠的网络控制面从集群管理中剔除，带来了前所未有的工程确定性。 - **施工期免维护**：在硬件物理搭建和光纤频繁施工扰动的混乱阶段，AI 训练任务即可提前数月进场稳定运行。 - **消灭控制面复杂度**：交换机彻底剔除 BGP 等动态路由协议，固化静态转发表，消除了控制面收敛失败与脑裂的一切潜在隐患。 - **从秒级到毫秒级**：故障感知由以往全网通告的“数秒至数十秒”骤降至端点感知的“毫秒级”，保障 GPU 时刻处于高产出计算状态。 ### [22:45](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1365s) 巨头跨界联盟与研究员的“静默革命” 构建如此庞大的超算网络并非一家之力。OpenAI 与微软展开了深度联合工程，由微软负责打造其顶级的 Fairwater 数据中心基础架构；与此同时，OpenAI 携手英伟达（NVIDIA）、博通（Broadcom）、AMD 以及英特尔（Intel），共同将 MRC 的技术规范推进为行业标准，并由这些芯片巨头在下一代超算硬件层面予以直接实现。前沿大模型的训练从来不是按部就班的年度软件版本更新，本质上是探索未知的顶级科研工程，每一次集群故障和训练中断都会导致极其昂贵的时间进度预测落空。 在 MRC 诞生之前，早期基础设施团队有着大量痛苦的“恐怖回忆”：网络工程师半夜频繁被警报叫醒，OpenAI 的食堂里常常见到网络运维人员神色沮丧，因为他们根本无法查清为何某些训练作业会无端挂死。而当配备了 MRC 的新集群上线后，内部收到了前所未有的全员正面反馈。模型研究人员甚至逐渐淡忘了底层网络的存在，不再需要询问某个特定集群到底运行的是什么通信协议。虽然底层监控统计数据显示网络设备和光纤每天依然在发生海量故障，但在研究人员的视野中，一切风平浪静。这种“让基础设施在用户面前隐形”的静默，被团队视作衡量底层技术取得真正胜利的唯一标志。 > “我们知道自己真正赢了的时刻，就是研究人员彻底不再提起‘网络’这个词，不再需要去了解当前集群底层到底跑的是什么网络协议的时候。” ### [25:05](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1505s) 将 MRC 捐赠 OCP：基础设施是全行业的共同命运 OpenAI 决定将 MRC 规范全面对外开源，通过开放计算项目（OCP, Open Compute Project）将其确立为全球公开标准。这一决策背后有着深刻的行业理念：AI 基础设施建设的投资规模空前巨大，如果各大软硬件厂商为了追求微小的竞争优势而各自为战、搞封闭技术壁垒，将导致全球供应链极度碎片化，进而拖慢整个人类 AI 产业的迭代步伐。OpenAI 本身便坚定拥抱开源与公开标准，其所有超算网络完全基于以太网（Ethernet）构建，正因以太网是开放且历久弥新的工业标准，整个产业链的协同效率才得以最大化。 正如早期的以太网和万维网（World Wide Web）通过开放共享改变了世界，面对像 Stargate 这样跨越全球多个地点、涉及海量合作伙伴与极限算力规模的超级工程，单纯把算力当作零和博弈是狭隘的。全行业面临的工程挑战极其艰巨，“大家完全没有必要各行其是、反复重新发明轮子”（"what we're trying to do is hard enough without everybody having to reinvent the wheel all the time"）。将 MRC 贡献给整个产业，不仅能让外部企业享受到极高可靠性的网络架构，还能反哺 OpenAI，通过统一的供应链与标准让下一代模型的训练迭代跑出更高的加速度。 | 评估维度 | 封闭专有网络方案 | 基于以太网与 OCP 的 MRC 开放标准 | |:--- |:--- |:--- | | **供应链协作** | 各硬件厂商各自研发私有互联协议，供应链碎片化 | 联合 NVIDIA、Broadcom、AMD、Intel 与微软，形成通用软硬件生态 | | **网络鲁棒性** | 高度依赖复杂交换机控制面与专有重传机制，排障困难 | 端点毫秒级多路径自愈，交换机采用全静态路由，根除软件栈崩溃隐患 | | **运维体验** | 链路微小抖动即可导致大模型训练阻断，工程师频繁深夜抢修 | 网络底层硬件故障率居高不下但对上层完全透明，研究人员零感知 | | **行业生态价值** | 构筑厂商生态围墙，推高多方合作的集成壁垒 | 通过 OCP 标准消除重复造轮子，使全产业能将资源聚焦于模型核心攻关 | ### [27:21](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1641s) 技术边界、光速物理极限与以太网演进 在探讨 MRC 的能力上限时，格雷格阐明 MRC 本质上是一个高度弹性的架构层。它原生构建于以太网基础之上，负责在两台设备完成物理互通的前提下，承载端点层面的静态路由路径选择与高效的拥塞控制（Congestion Control）。当部分链路发生损耗或流量突发时，端点上的拥塞控制算法能够实时调整发包策略，确保网络资源在所有连接之间得到公平且高吞吐的利用。 然而，网络技术始终受到不可逾越的物理定律限制，其中最根本的硬约束便是真空及光纤中的光速限制。光信号在网络两点之间的单程物理时延存在绝对下限，随着交换端口和网卡速率由百吉比特向太比特（Terabit）级别飞速跃升，时延无法按比例缩小，这意味着在任意给定的往返时钟周期内，链路中处于传输途中的在途数据量（Outstanding Data）将成倍暴增。这直接改变了网络的工作点（Operating Point），对端点的缓冲控制、丢包容忍和流控机制提出了更严峻的挑战。MRC 并不能打破物理学的极限，但它通过极具延展性的设计，为应对未来数代硬件迭代与吞吐暴增构建了一套坚不可摧的基础通信底座。 ### [28:45](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1725s) 借力以太网演进与 IPv6 段路由：让网络核心重归极简 以太网在过去数十年间经历了翻天覆地的演进，早已不是十年前、二十年前甚至四十年前的旧模样。OpenAI 选择基于以太网构建方案，核心逻辑正是要直接承接整个全球网络工业界持续创新的时代红利。MRC（Multipath Reliable Connection）的设计精髓在于“将智能推到网络边缘”（edge of the network），只要底层的以太网核心持续扩展，整个超算集群的网络规模就能毫无阻碍地同步放大。 为了将网络内部的复杂性彻底剥离，团队甚至选择直接关闭网络内部的动态路由逻辑，转而采用源路由（source-routed）机制。在实现上，MRC 依托 IPv6 段路由（Segment Routing）技术，在每个数据包的头部就精确标明它必须途径的交换机序列。这种设计使得网络中间层的所有交换机都可以退化为极度纯粹、简单的“傻瓜交换机”（dumb switches）。在构建超大规模且必须绝对可靠的集群时，让网络中间层尽可能扁平简单，能带来难以估量的工程稳定性。 谈及为何将 MRC 确立为开放标准并联合产业链上下游共同推进，团队明确表示这承袭了以太网成功的精神。以太网之所以繁荣，就是因为全行业围绕统一标准向同一个方向合力推进。OpenAI 坚信如果将 MRC 作为闭门造车的专有技术（exclusive），绝不可能走得如此深远；唯有让它成为下一代行业标准，整个生态才能共同应对 AI 超级系统带来的严峻挑战。 ```mermaid flowchart TD accTitle: 基于 IPv6 段路由的 MRC 极简核心架构 accDescr: 展示源端主机利用 IPv6 段路由指定精确交换路径，中间交换机保持极简转发 subgraph EdgeSmart["边缘智能端（GPU / 主机）"] HostA["源端主机 / 边缘智能"] Header["注入精确路径: [SW1 -> SW2 -> SW3]"] HostA --> Header end subgraph CoreFabric["极简网络核心（Dumb Switches）"] SW1["交换机 1 (无复杂路由计算)"] SW2["交换机 2 (仅按段列表转发)"] SW3["交换机 3 (极简转发)"] Header --> SW1 SW1 --> SW2 SW2 --> SW3 end subgraph DestEdge["目的端"] HostB["目标 GPU / 主机"] SW3 --> HostB end ``` ### [30:36](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1836s) 扁平化网络拓扑：削减交换层级与提升每瓦有效算力 从资源利用与能耗效率的角度来看，基础设施的核心目标就是在物理极限内将每一份电力的产出压榨到极致。MRC 带来的一大非显而易见的优势，在于它在多条并行动力链路上高效“喷洒”数据包（packet spraying）的能力，这使工程师得以构建出拓扑结构极其扁平、设备数量大幅减少的新型网络架构。 传统的大规模集群往往需要堆叠多层高密度的交换机网络，层级越多，硬件成本越高，更致命的是交换机本身会吞噬海量电力。借助 MRC 带来的极高路径利用率，OpenAI 得以大幅减少交换机层级，让网络变得更加轻薄扁平。 这种架构革新直接放大了系统“每瓦特所能完成的有效工作量”（useful work per watt）。多余交换机层级被剔除后，宝贵的机房供电得以更直接、更充沛地输送给真正执行模型计算的 GPU 核心，避免算力被外围网络消耗所稀释。 - **极简扁平拓扑**：大幅精简网络交换机层级，降低互连复杂度和单点隐患。 - **能耗向算力倾斜**：消减多余网络设备的空载与转发能耗，显著提升数据中心整体能效比。 - **建造成本骤降**：设备数量锐减直接缩减了组网所需的昂贵光模块与交换机采购开销。 ### [32:07](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1927s) 大模型长尾延迟挑战与全栈协同调优 随着前沿模型从早期的纯文本与受限上下文窗口扩展到复杂的多模态系统（图像、视频混合训练），底层系统承受的压力呈现指数级上升。在这个量级下，集群不仅要吞吐海量数据，更被套上了极其苛刻的超低延迟硬指标（latency bound）。一旦网络传输出现一丝迟缓，在大规模集群与高度优化的计算栈面前，算力等待所造成的“代价与懊悔”（regret）就会被无限放大。 在 OpenAI 内部，算法研究员与算力优化专家夜以继日地压缩 GPU 端的前向与反向传播耗时，然而 GPU 跑得越快，留给网络传输数据的窗口就缩得越窄——如果网络成为那块拖后腿的短板，算法人员的一切心血都将归零。同时，随着集群规模激增，最慢计算节点的方差急剧攀升，如何控制长尾延迟（tail latency）成为决定生死的核心战役。 这里隐藏着一个反直觉的网络统计学陷阱：如果在没有 MRC 机制的情况下盲目将带宽分散到更多物理路径上，长尾状况反而会严重恶化。这就像“将同样数量的球掷入更多的桶中”（throwing the same number of balls into more bins），最差的桶与平均水平之间的偏离比例会急剧恶化。MRC 依靠严格的确定性源路由与精密的动态负载均衡，精准避开了路径拥塞失衡的陷阱。这种底层硬件能力与上层工作负载的深度协同，体现了 OpenAI 独特的纵向整合（vertical integration）优势：底层硬件工程师透彻理解算法工作负载，而算法人员同样深谙网络交换机内部的物理机理。 ### [34:59](https://www.youtube.com/watch?v=TiW96H5HmAw&t=2099s) 太空 AI 计算的遐想与地面超级集群的现实抉择 针对业界近期讨论“将 AI 算力部署到太空卫星”的热门话题，交流中剖析了推理与训练两种场景的本质区别。如果是单个卫星挂载几块 GPU 进行局部的推理计算，完全具备可行性；但若要将动辄数万张卡、跨越数万公里的分布式超大模型训练搬到轨道空间，物理法则就成了不可逾越的高墙。 光速在空间传播的固定延迟是分布式训练的死穴，而更为严峻的现实壁垒则是“硬件背景故障率”（background rate of failures）。每一代全新 GPU 的算力更高、成本更昂贵，结构也更加脆弱。在类似星际之门（Stargate）级别的地面超算中心中，每天都有来自微软（Microsoft）和甲骨文（Oracle）的技术工程师在机房一线进行硬件巡检与物理维护。一旦将集群送入轨道，面对随时可能宕机或出现静默错误的精密硬件，太空维保在工程和经济上几乎是天方夜谭。 尽管从物理学和工程浪漫主义的角度来看，“在太空部署维修工程师”或建立轨道超级数据中心的构想令人着迷，但务实的工程现实却异常残酷。单单是在地球上协同多家行业巨头将 MRC 协议从草案落地为可运行的高性能网络，就已经历了无数次现场调试与极限攻坚。当前的破局重心依然必须坚决放在地球表面：脚踏实地建造更多超大规模的地面计算中心（terrestrial compute centers），以此加速推动人类社会整体“净智能总量”（net amount of intelligence）的爆发式增长。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=TiW96H5HmAw&t=0s) 当我们取得最终胜利时，研究人员甚至不需要知道当前集群到底在跑什么网络协议。 > [00:38](https://www.youtube.com/watch?v=TiW96H5HmAw&t=38s) 量子计算机目前根本无法正常工作，所以更谈不上扩展性。它未来或许能行，但现在不行。 — Greg Steinbrecher > [00:38](https://www.youtube.com/watch?v=TiW96H5HmAw&t=38s) 标准化的最大痛点在于你必须让全世界所有参与者都达成一致，这通常需要耗费极为漫长的时间。 — Mark Handley > [04:34](https://www.youtube.com/watch?v=TiW96H5HmAw&t=274s) GPU 之间的通信本身就是计算过程的有机组成部分。 — Mark Handley > [07:12](https://www.youtube.com/watch?v=TiW96H5HmAw&t=432s) 如果大模型训练任务在半夜崩溃且无法自动恢复，我们就会被直接叫醒。 — Greg Steinbrecher > [10:59](https://www.youtube.com/watch?v=TiW96H5HmAw&t=659s) 到处都会崩（Everywhere）。 — Mark Handley > [17:00](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1020s) 氛围非常好，感觉棒极了！（The vibes are very good.） — Mark Handley > [18:55](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1135s) Mark has mildly undersold this... Mark, come on, man — Greg Steinbrecher > [20:38](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1238s) We did not care. We didn't even notice. MRC just took care of it > [22:45](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1365s) 我们知道自己真正赢了的时刻，就是研究人员彻底不再提起‘网络’这个词，不再需要去了解当前集群底层到底跑的是什么网络协议的时候。 > [25:05](https://www.youtube.com/watch?v=TiW96H5HmAw&t=1505s) what we're trying to do is hard enough without everybody having to reinvent the wheel all the time ## Mentioned OpenAI, Mark Handley, Greg Steinbrecher, Andrew Mayne, AMD, Broadcom, Intel, Microsoft, Nvidia, Oracle, Multipath Reliable Connection (MRC), Fairwater, Stargate, Open Compute Project (OCP), 以太网 (Ethernet), 伦敦大学学院 (UCL), IPv6 段路由 (Segment Routing), BGP (Border Gateway Protocol), OFC (光纤通信大会), Facebook ## Video description Training frontier models isn’t as simple as adding more GPUs—one small problem and the whole coordinated dance falls apart. OpenAI’s Mark Handley and Greg Steinbrecher discuss how a new supercomputer network design, used to train some of the company’s latest models, keeps the whole system moving in lockstep, even with record numbers of GPUs. They break down Multipath Reliable Connection, a new protocol OpenAI developed with AMD, Broadcom, Intel, Microsoft, and Nvidia, and why they’re making it available for the whole industry to use. Chapters 00:00 Intro 00:39 Greg and Mark's paths to OpenAI 04:34 Why training AI stresses networks differently 10:05 Bottlenecks, failures, and the cost of waiting 15:19 How Multipath Reliable Connection works 18:59 A protocol to route around failures 25:05 Why OpenAI is making MRC an open standard 35:09 Could AI compute move to space? [Source](https://www.youtube.com/watch?v=TiW96H5HmAw)

[0:00] 当我们取得最终胜利时，研究人员甚至不需要知道当前集群到底在跑什么网络协议。 [0:38] Greg Steinbrecher: 量子计算机目前根本无法正常工作，所以更谈不上扩展性。它未来或许能行，但现在不行。 [0:38] Mark Handley: 标准化的最大痛点在于你必须让全世界所有参与者都达成一致，这通常需要耗费极为漫长的时间。 [4:34] Mark Handley: GPU 之间的通信本身就是计算过程的有机组成部分。 [7:12] Greg Steinbrecher: 如果大模型训练任务在半夜崩溃且无法自动恢复，我们就会被直接叫醒。 [10:59] Mark Handley: 到处都会崩（Everywhere）。 [17:00] Mark Handley: 氛围非常好，感觉棒极了！（The vibes are very good.） [18:55] Greg Steinbrecher: Mark has mildly undersold this... Mark, come on, man [20:38] We did not care. We didn't even notice. MRC just took care of it [22:45] 我们知道自己真正赢了的时刻，就是研究人员彻底不再提起‘网络’这个词，不再需要去了解当前集群底层到底跑的是什么网络协议的时候。 [25:05] what we're trying to do is hard enough without everybody having to reinvent the wheel all the time

[0:00] Greg Steinbrecher: We know we've won when researchers stop needing to know what network protocol this particular cluster is using. [0:38] Greg Steinbrecher: unknowably complicated [0:38] Greg Steinbrecher: a complete and utter lie but tell you something about that system. [0:38] Greg Steinbrecher: don't work and therefore they don't scale yet [3:28] Mark Handley: everybody needs to agree. [3:28] Mark Handley: you only needed to agree with it while building, not the whole world. [5:04] Mark Handley: the communication between the GPUs is actually part of the computation. [5:04] Mark Handley: just about the worst possible workload you could think to put onto a network. [7:12] Greg Steinbrecher: ocean of compute [8:50] tail of the tail [10:59] Mark Handley: carry on and basically not notice. [14:18] Greg Steinbrecher: crank them up to 11. [15:38] better models, more intelligent models, faster from OpenAI. [15:38] The vibes are good? The vibes are good. The vibes are very good. [18:32] like self-annealing [18:32] did not care [21:43] really dumb [25:05] shared fate of the whole industry [25:05] useful work you can do per watt. [35:00] The dreamer side of me says that would be really, really cool... The practical side of me says it's really, really hard to do this stuff on earth.

[0:00] Hello, I'm Andrew Mayne, and this is the [0:02] OpenAI Podcast. On today's episode, we're discussing how [0:05] to make supercomputers better at training [0:07] models. Joining me are Mark Handley from the core networking [0:10] team and Greg Steinbrecher from workload [0:12] systems. They'll discuss how a breakthrough has made [0:16] training more efficient so everyone gets smarter models faster. [0:22] This has really allowed us to remove one [0:24] of the key barriers to continuing to scale. [0:26] We're talking about a lot of the world's [0:28] fastest GPUs and making them all work together on a single task. [0:32] We know we've won when researchers stop [0:33] needing to know what network protocol this particular cluster is using. [0:38] So tell me a bit about your background. [0:46] I started out doing physics and math in undergrad, wanting to basically understand [0:48] how complex systems work. I always liked the part of physics [0:51] that's about how do you take this thing [0:53] that is unknowably complicated and build a simple model [0:55] that is a complete and utter lie but tells [0:57] you something about that system. And then build your [1:00] intuition on that and kind of build more [1:02] complex models. I ended up doing a PhD trying to build quantum computers. [1:06] Ambitious PhD. [1:11] You know, little things, little things. Unfortunately, what I liked is big [1:13] complicated systems. And you'll note that quantum computers [1:15] don't work and therefore they don't scale [1:17] yet. They will someday, but they don't yet work. And so [1:20] I kind of took a look at the chips we were [1:22] designing to control light for quantum computers. [1:24] And I went, huh, that kind of looks like a network switch. [1:27] What if we use this as a network switch? [1:29] And what I found out pretty quickly was [1:31] that academia does not know a whole lot about what [1:33] real data center workloads look like. [1:36] You get a whole bunch of kind of very toy models, but they're not very informative. [1:41] And so I ended up kind of pitching an industry company to get a fellowship. [1:46] They paid for the last two years of my PhD. [1:48] I ended up working there for a while, [1:50] building out, kind of doing initial network hardware [1:53] just to try to understand what is it that [1:56] we actually need from data center networks. [1:59] What I found out was that there's a huge [2:01] amount of headroom on just conventional data center [2:03] networking hardware and lots of room for [2:05] optimization. We did not need my little optical [2:07] chip. We did not need to do anything fancy [2:09] like that. But there's all sorts of really fun problems. [2:14] And then around that time, the whole AI [2:15] boom started to kick off. We decided we needed to [2:18] build big GPU clusters. And in particular, [2:20] we needed to build networks for those GPU clusters. [2:23] And so I got roped in on trying to build [2:24] kind of simulations of those so that we can figure out what to build. [2:28] And then in the process of trying to build [2:31] a simulation of these systems, you learn a lot about how they have to work. [2:35] And at some point I said, well, why don't I just go build the actual thing? [2:39] And so I transitioned from writing software to build simulations to just [2:42] writing the software that allows GPUs to communicate with each other. [2:47] And then a little over a year ago, ended up coming here to OpenAI to do some of the [2:52] same stuff, but to get even closer to the actual model training. [2:57] So the team I'm on is responsible for more [3:00] or less making sure that we'd use the GPUs efficiently. [3:04] So are the models training as quickly as they can be? [3:07] Are we not bottlenecked on the network? [3:10] What do we do when something fails? [3:11] Are we restarting efficiently? [3:12] How do we kind of work around quirks in the hardware? [3:18] and yeah now I get to play with some of [3:20] the most fun hardware in the world and like try to make it — [3:25] try to kind of squeeze every last ounce of performance out of it. [3:28] Mark you've had a considerable amount of [3:29] experience trying to get computers to talk to each other and do meaningful [3:32] work tell me more [3:35] about your background. [3:38] So when I'm not at OpenAI I'm a professor at the University College London and I've [3:41] been doing networking research for more decades than I care [3:42] to think about. Originally, I started [3:44] working out on trying to make the internet do video conferencing. [3:48] And back then, that was a really difficult thing because the computers were so slow. [3:53] And then we came up with a way for doing [3:55] that that suddenly the rest of the world got interested in. [3:58] And so the standards we wrote for doing [4:00] that are now the things that your phone uses to communicate [4:02] with 4G and 5G networks. And so that was [4:05] the first part of my life, was trying to standardize all of [4:09] that sort of stuff, that everybody could actually take advantage of that. [4:12] And the problem with standardizing is everybody needs to agree. [4:15] So it just, it takes a long time to [4:16] actually get everybody to agree on anything. [4:19] A while back, I got interested in what was happening in the data center world. [4:22] And it had the big advantage that you [4:24] could actually do something different because you [4:27] only needed to agree with it while building, not the whole world. [4:30] And so that was how I got into thinking, [4:31] well, this data center networking stuff is a really interesting place to be. [4:34] Well, it's an interesting problem because I think that just the idea that we'd be [4:40] scaling as many GPUs as we are now, it's happened so fast, happened so quickly. [4:46] And often, we're still using GPUs, which are graphical processing units. [4:50] We're just now starting to use next generation chips to do this. [4:54] So how much has been the work regarding [4:56] just to update the way we think about it and update the sort of the tools we're [5:00] using for this, which is what we're here to talk about with. [5:04] Multipath Reliable Connection. [5:05] So from a network point of view, [5:08] the data centers that we traditionally used to build, [5:11] they kind of got derived from what we used to do in the internet. [5:14] And when you do communication in the internet, [5:17] you have lots and lots of people communicate. [5:18] It's like there's a lot of data moving around, [5:22] but they're all doing their own separate conversations. [5:24] And so generally speaking, [5:27] if the more communications you add onto the same shared network, [5:30] the more things smooth out and become even. [5:33] And so that's great. You can take [5:34] advantage of the statistics of large numbers. [5:37] Unfortunately, when you look at what we're [5:39] doing when we're training things, it's exactly the opposite from that. [5:42] We're talking about a lot of the world's fastest GPUs and making them all work [5:46] together on a single task, which is why this stuff gets hard. [5:51] That process of basically ingesting all of this data and allowing the AI model to [5:56] learn from that data in parallel is what allows us to build smarter, better models. [6:03] But let's say one GPU goes a little slow. [6:06] Well, now all the other GPUs have to wait for it. That's all wasted time. [6:10] Or one of the GPUs, you know, a cosmic ray hits it and some bits flip and it stops. [6:15] Well, OK, now that whole step is maybe not [6:18] useful and we have to maybe roll back or kind [6:21] of stop and take stock of what has happened. [6:24] And while we stop and take stock, all the GPUs are not doing useful work. [6:28] Yeah, a key thing here is that the [6:29] communication between the GPUs is actually part of the computation. [6:32] It's like they're doing one big computation across all of them. [6:34] It's not they're doing different things. [6:36] We have to actually have them communicate [6:38] with each other in order to agree on what the result of that step of computation is. [6:43] And that's just about the worst possible [6:44] workload you could think to put onto a network. [6:47] And so the way the industry has evolved over the last couple of decades has been [6:51] slowly coming up with improvements on how we actually do that. [6:55] But until recently, the scale wasn't enough that it really mattered. [6:58] You could get away with doing the same [7:00] sort of things we did on the internet, just bigger. [7:03] But you can't get away with that anymore. [7:05] And that's why we tried to think of [7:06] different ways to actually solve these problems, [7:08] to cope with these very synchronized workloads as we scale things up. [7:12] AI has upended a lot of things in the world. [7:15] It has definitely upended the way that technology companies have to think about [7:19] the data centers that they're building. [7:23] conventional hyperscalers for kind of the [7:26] web era, the teams that built this were very kind of [7:31] disconnected from the individual [7:33] workloads. The goal was to basically just provide an ocean of [7:37] compute. AI has forced us to think very [7:39] differently. And OpenAI in particular has been kind of at the [7:42] forefront of realizing that the systems [7:45] and the design of the systems is integral to the training [7:48] of the models and that you can't just have [7:50] your infrastructure team sit over here in one building [7:53] and kind of just deliver an ocean of [7:54] compute and your model team sit over here and like try to make [7:57] the best you know model that they can on [7:59] that compute you really have to do kind of a co-design [8:02] across these whole things um and so you [8:05] know we on my team sit literally next to the researchers [8:10] and talk with them every day about you [8:12] know how they can best you know how their workloads fit [8:16] best onto the existing servers. And in [8:17] doing that, we learn a lot about where the pain points are. [8:20] And we are on call for the big training [8:22] runs. We get woken up in the middle of the night if [8:25] something is broken and can't be fixed. [8:28] And so through that process, and you start to go, [8:32] oh, well, what if in the next generation [8:33] we fix these problems? What if we didn't build data [8:35] centers with the same properties as we did [8:38] for web scale workloads? What if we instead fix this [8:41] piece or this piece or this piece. And I [8:44] think the network has been a real source of pain for us. [8:50] Yeah. So when you're building a data [8:51] center network, all of these GPUs, because they do [8:54] this computation and they all need to talk [8:55] at the same time, you need a lot of bandwidth. [8:58] And the problem with that is you can't [9:00] build that with a single switch or even a hierarchy of [9:03] switches. You have to build hierarchies of [9:05] hierarchies of switches. And so that means that [9:08] when you communicate from one GPU to a different GPU, [9:11] there are many different paths your traffic could take through that network. [9:14] Thousands of different paths your traffic could take [9:15] through the network because we build [9:17] so many different switches in there. [9:18] There's several thousand switches in one of these buildings. [9:23] Now that gives us an interesting problem, [9:25] which is which path do you take from one place to the other? [9:28] And if you basically have the requirement [9:32] that I want to be able to send from one GPU [9:34] to another GPU as fast as possible, [9:36] and I choose, say, a random path through the network, [9:39] if I get lucky and nobody else chooses the same path, then great. [9:42] But if I get unlucky and two people choose the same path, we go slow. [9:45] And if 10 people choose the same path, we go really slow. [9:47] And so this statistical multiplexing that [9:49] we used to have when we designed the internet [9:51] just doesn't work out very well when we're [9:53] trying to build these networks for our data centers. [9:55] And so that's where we came in to try and design things somewhat differently. [9:58] Maybe to put a fine point on that, [10:00] the synchronous nature of the workload that we talked about earlier [10:03] is why this becomes such a problem. [10:05] It is not about how fast can the average pair of GPUs talk to each other. [10:09] It's always what is the absolute worst case that occurs. [10:13] So if you think about you've got thousands [10:15] of GPUs that are trying to talk to each other. [10:17] So there's tens of thousands or hundreds [10:19] of thousands of network flows on tens of thousands of links. [10:22] What you have to do is you have to look at [10:24] that entire network and go, what is the link that got most bottlenecked here? [10:27] That one link is going to set how fast all of your GPUs are able to work and how much [10:31] time it takes for you to move data through this. [10:34] because everything is proceeding in [10:36] lockstep. And so whereas previously we might have been [10:40] subject to or kind of taken advantage of [10:42] average statistics, we don't have that luxury anymore. [10:45] We instead are subject to like the tail of [10:47] the tail, we call P100, the 100th percentile statistics. [10:52] And that leads to very different systems [10:53] requirements than when you can kind of rely [10:55] on the law of large numbers to take care of you. [10:59] And so the other problem with this is that when you build your networks, we build the [11:02] best possible networks we can. We go to the best [11:03] equipment vendors, we use the best optics [11:05] and so forth. When you really scale things really big, [11:09] things are always going to be failing. [11:11] Links will fail, switches will get confused and will have to [11:14] be rebooted and so forth. Any one of these [11:16] failures is going to affect the traffic running [11:18] over the network. And so if you've got [11:20] this problem where we only care about the 100th [11:22] percentile and then a link fails in the [11:24] network, what happens? Well, stuff will fail. We may have [11:29] time before the routing reconverges and [11:30] then we move the traffic onto a different path. We take a [11:32] glitch there. That can be quite a long [11:34] glitch that can cost us. Or worse, we can actually cause one [11:36] of these communication transfers to fail. [11:38] A single transfer fails and you could end up with the whole [11:41] job crashing. And so we really want to [11:43] avoid that kind of problem. We want to build a way of using [11:48] these networks that is resilient to not [11:50] only the potential transient congestion we might build in [11:53] the network, but also when things fail, we [11:55] just want to be able to carry on and basically not notice. [11:58] But that requires that we actually design [11:59] the network protocols differently from the start. [12:01] You can't retrofit this onto existing network protocols. [12:04] Yeah, it's a very interesting problem as you describe it, [12:07] because I can see that where you could say like, [12:08] oh, well, if we have a thousand GPUs, [12:09] there's only one chance out of 10 that they're going to fail or whatever. [12:12] And now I have a hundred thousand GPUs. [12:14] Well, guess what? I'm going to have a failure all the time. [12:17] And that's what you have to sort of solve for each time you scale it. [12:19] So where does this break? Everywhere. There we go. [12:24] If you think about the meantime between failure of the equipment, [12:27] for any particular range of equipment, [12:29] You've got some time between which [12:30] something will fail somewhere in your building. [12:33] And of course, the bigger we get, with the [12:36] same cost of equipment, that time comes down further and further. [12:40] And eventually you get to a point where actually something is failing sufficiently [12:43] often that you don't get any work done on a large synchronous workload. [12:46] And we can't have that happen. [12:48] So we have to do things differently to make that work. Yeah. [12:51] So the very simple math here is you can basically assume that if failures are [12:55] independent and you double the size of your system, [12:58] you're going to have half the time between failures, right? [13:01] Your mean time to failure goes down by half. [13:04] The important thing to think about here for the network is for every GPU, [13:08] we have tens if not hundreds of network components. [13:12] So even just like, say you've got one GPU connected to one network adapter. [13:17] In that network adapter, if it has an optical transceiver in it, [13:20] maybe you'll have four lasers. [13:22] On the other end of that transceiver, you'll have another four lasers. [13:25] And so already just connecting that one GPU just to its first hop switch, you've [13:29] got an order of magnitude more lasers than you have GPUs. [13:34] Now add in multiple layers of switching [13:36] and you start to get into several orders of magnitude more components in the [13:40] network than you have kind of at the edge of your network. [13:44] Because we need to have so much bandwidth here. [13:46] So we have to build these networks. [13:48] We can't kind of taper them down and only [13:50] have a couple of components in the network because then we'd be starving the GPUs. [13:54] They wouldn't be able to actually kind of [13:56] use their full capability to do math as quickly as possible. [13:59] Instead, we would just be letting them sit idle and wasting time, money, energy. [14:05] We would get trained models more slowly. It would be bad. [14:08] And so we built a very big network. [14:10] But now you have many, many, many more [14:12] components in that network than you have maybe at the edge of your network. [14:15] You have literally millions of optical links within the same building. [14:18] So it's a huge scale. You mentioned the data centers originally were. [14:22] I go do something to search the cloud, to get my email, whatever. [14:25] I may be talking to one server, there might be a backup there. [14:28] The idea that we would be having more computers inside a data center talking to [14:33] each other than we did just a few years ago, people trying to connect to it. [14:38] And so how have these protocols evolved? [14:40] One of the things that got me into [14:42] networking in the first place, I have a very distinct memory of, I was at a [14:46] conference, OFC, it's an optical communications conference back in 2017. [14:51] And they had a presenter from Facebook. [14:53] and he put a chart up that had a stacked [14:55] line chart of the traffic that they serve to their [14:59] end users and the traffic that goes inside [15:01] of their data centers. And the traffic inside of [15:04] their data centers was just exploding, [15:06] even while the kind of amount of traffic that they were [15:08] sending to end users was staying constant. [15:10] And this is way before GPU clusters and AI. So this is, [15:14] what AI does is it takes all of the [15:15] systems challenges that people were having previously [15:18] and it cranks them up to 11. [15:19] So to address this, [15:21] you've been working on a method, [15:24] Multipath Reliable Connection. [15:26] The insight was basically that you have, [15:30] we try to manage congestion in networks. [15:32] And so there are several pieces [15:35] that you can pull together and do this. [15:37] And the first part of the insight was [15:38] if you spray the packets across many paths, [15:40] you can low balance those paths [15:41] through the network really equally. And if you do that [15:45] and you build a network topology that has enough capacity, [15:48] then you don't cause hotspots in the network. [15:50] It leaves you with just one place where you have congestion, [15:53] which is if multiple people try to send to the same destination at the same time. [15:57] But it also leaves you some problems [15:59] because the packets can get jumbled up in transit [16:01] because they're taking different paths. [16:02] And so if you do manage to cause congestion and cause loss, [16:05] it's a little bit difficult to figure out whether you got lost [16:08] or whether you should still be waiting for packets [16:09] because they got jumbled up in transit. [16:12] And so the second piece of this is a technique we call packet trimming, [16:17] which is if you're causing congestion in the network [16:22] and you would overflow a queue, [16:24] normally we'd just drop the packet. [16:26] And then we've got ambiguity. Did it get lost? Did it get lost? [16:28] How long do I need to wait for it? [16:31] But what we do instead is we will trim off the payload of the packet [16:33] and just forward the little tiny packet header to the destination, [16:36] which can immediately request a retransmission [16:38] and we can retransmit that packet. [16:40] And that totally removes the sense of ambiguity [16:42] as to whether we lost packets due to congestion [16:45] or whether we should still be waiting for them because they got reordered. [16:48] Interesting. So just making sure that the part saying, are you there, goes through, [16:53] and then you can figure that out and then send the rest. [16:55] Yes, you really need to know [16:56] that you should still be waiting for it [16:57] or you should not still be waiting for it. [16:59] What does this mean for the end user? [17:00] The biggest thing that this means [17:02] is that you're going to get better models, [17:05] more intelligent models, faster from OpenAI. [17:08] So MRC allows us to accelerate [17:12] every part of our research and deployment pipeline. [17:16] It allows individual users to not worry about their jobs failing, not worry about [17:20] how their job has gotten scheduled and whether the performance of it is going to [17:25] be different because they're placed on the same rack as someone else's job. [17:31] It allows us to train frontier models much [17:34] faster, more reliably, and really just to [17:37] turn the entire crank of that pipeline much faster and much more reliably. [17:43] So you should expect to see an ever [17:46] increasingly exciting pipeline of releases from us. The vibes are good? [17:51] The vibes are good. The vibes are very good. [17:53] The idea came out of a lot of research [17:55] work that we've had over the last few decades. [17:58] We're not fundamentally inventing anything new. [18:01] We're just doing things that other people have invented, but pulling the combination [18:05] together into a set of features. [18:06] So we formed this group of people who are all interested in doing this. [18:12] And last year, we finally got to the point where we were able to deploy this. [18:18] And we went from, in a few months, from [18:20] the first hardware available to actually running [18:23] and training models and it actually all operating. [18:26] So this has the result that we don't cause [18:28] that congestion that we talked about before. [18:32] The second really nice property is that if [18:34] something fails in the network, every single [18:37] one of these flows that go through there will probably be affected by the failure. [18:40] But it'll only be affected a little bit. [18:42] And within a few round trip times across [18:44] the network, we stopped using a failed link. [18:48] And so this problem of links failing, bringing down the network just goes away. [18:53] All of the flows themselves from the [18:55] network interface at one side network interface are [18:57] just avoiding those failures as we go through the network. [18:59] It's like self-annealing. Exactly. Yes. [19:02] So I think Mark has mildly undersold this, but maybe we should- Mark, come on, man. [19:08] So conventionally, when a link goes down [19:11] on a network, what happens is one side of that link, [19:15] the switch at one side of that link, or maybe both sides of that link notices. [19:18] But then it has to tell all of its neighbors that that link went down. [19:22] And then they have to tell all of their neighbors that that link went down. [19:24] And so you have a distributed systems problem. [19:27] It's conventionally solved with a [19:28] technology called BGP, Border Gateway Protocol, [19:31] which is basically just like a gossip [19:33] protocol that allows one link over here to eventually tell [19:37] this switch all the way over here, maybe [19:39] through five or seven hops through the network, then, [19:42] hey, you can't get to this destination if [19:43] you take this link. You have to use these other links. [19:47] That's a distributed systems problem that has a convergence time. [19:50] What MRC has done is it has taken that and [19:53] it has broken the need to coordinate. Every endpoint [19:58] independently very quickly detects, hey, I [20:00] shouldn't use that path and just stops using it. [20:04] And this is maybe counterintuitive because [20:06] you would think, oh, it's easier if I just have some central authority that tells me [20:10] that this link is down and that central authority can distribute that information. [20:15] Anybody who's waiting for a website to update knows that. It's not going to work. [20:18] Right. Central authorities are, generally [20:20] speaking, also known as single points of failure. [20:23] And so instead, what we've done here is we [20:25] no longer have to wait for this whole convergence process to occur, which can [20:28] take seconds or in the tail, tens of seconds. [20:31] Instead, everyone within, generally [20:33] speaking, milliseconds notices and just stops using that link. [20:37] So this is a very big deal because [20:38] previously, you know, the link goes down and the whole job stops for a few seconds [20:43] as we wait for kind of the network to stabilize. [20:47] That's time, again, that the GPUs aren't doing useful work. [20:50] And as you, again, scale up, you're going [20:53] to have more and more and more of those individual little seconds. [20:56] And here now, what we've observed is that [20:59] we turned this on as the data center was being built. [21:04] As Mark said, we were able to get jobs up [21:06] in training within months of hardware arriving. [21:09] There's a lot of manual labor that goes into building one of these buildings. [21:13] There's a lot of shared points where fibers from one data hall are coming in [21:17] and technicians are trying to assemble another data hall or things like that. [21:22] And so what we saw was that because of all of this kind of manual effort that was [21:25] going on, links were going up and down all the time, like even way more often than [21:29] you would kind of hope just due to natural failures. We did not care. [21:33] We didn't even notice. MRC just took care of it. [21:36] It just kind of would detect, hey, can't use that path, move on to the next one. [21:42] Didn't care. It was incredible. [21:43] So the other thing this gives you beyond [21:45] just that is because once it's handling that by itself, once MRC is already [21:49] working around the failures, traditionally we would probably still have been running [21:54] a routing protocol in the network to find the paths that actually work. [21:58] But routing protocols themselves are [21:59] complicated and switches are complicated and switch software is complicated. [22:02] And these are all things that can fail. [22:04] And we realized that actually MRC itself [22:06] was able to figure out which paths were still working. [22:09] And so actually, we just decided that we would turn off the routing protocols. [22:13] We use completely static routing in the switches at the largest possible scale. [22:19] And so some paths are broken. Who cares? [22:21] MRC will find the broken ones that still work and keep going. [22:25] And that just removes a whole set of [22:27] complexity out of our network management that we just don't need anymore. [22:30] We don't care about whether the switch [22:32] control plane has converged because it doesn't need to. It's entirely static. [22:36] They have a configuration that they have at boot time. [22:38] They boot up and they never change their routing tables from then onwards. [22:41] So this is a very big effort working with a bunch of people. [22:45] Care to talk about some of the partners? [22:47] Yeah, we've been working with Microsoft, [22:50] who build our Fairwater data centers for us. [22:53] And then we've been working with NVIDIA, [22:55] Broadcom, AMD, and Intel to standardize this specification. [23:00] And with all of these guys to actually [23:03] build our hardware for our new supercomputers. [23:06] It's interesting because as a user of this [23:08] technology, and you listen to sort of the way we sort of think about things where [23:12] people talk about like, well, when's the next model coming out? [23:15] Like it's a software update that comes up every year. But it's not. [23:18] Every model is essentially a research [23:20] project, and it's dependent upon what goes on in the training. [23:24] And you try to have an estimate of how [23:25] long it's going to take, and we try to predict where that's going to be. [23:28] But this kind of reliability sounds like it's going to be an incredible advantage. [23:33] God, yeah. I would say from talking to the people who [23:38] were here even earlier, I mean, you hear just horror stories about what it was [23:43] like, how often they were getting woken up. [23:47] I remember walking in the cafeteria and [23:48] seeing some of the people working on networking and sort of sad faces because [23:52] they didn't know why something had stopped in a run. And it was just. Yeah. [23:57] We have heard nothing but universally [23:59] positive feedback about how stable the clusters with MRC are, how well they're [24:04] working, how basically the researchers don't have to think about this anymore. [24:09] And then you look at the statistics and you realize just how much stuff is [24:11] breaking all the time and they're not noticing. Yeah. [24:15] To some degree, this is the ideal, right? [24:17] I know I said earlier, we are pushing the limits of infrastructure. [24:22] And so there's never going to be a time that you can just completely ignore infra. [24:26] you know the ideal world right is [24:27] researchers don't have to think about this but that's never [24:29] going to be the case but every time we we [24:31] know we've won when researchers stop saying you know [24:34] network or stops you know stop needing to [24:37] know what network protocol this particular cluster is using [24:42] and yeah it's been it's been really nice [24:44] to basically be able to focus all of because there's [24:47] don't get me wrong there's plenty of other [24:49] things that break there's plenty of other work to do [24:51] but this has really allowed us to remove [24:53] one of the key barriers to continuing to scale [24:55] and to being able to deliver newer and better models on a much, you know, [25:01] we are trying to scale everywhere, including our velocity. [25:05] And you have all decided to make this open to everybody to use. [25:09] Yeah, so the specification is due out through OCP as an open standard. [25:14] And we've, as you say, decided to open this up for everybody to use. [25:21] We're big believers in open standards and open source. [25:25] We're building all of our networks on top [25:26] of Ethernet, which is selfless and open standard. [25:29] And we benefit when the industry has velocity, when the industry can keep up [25:33] with the things that we're trying to do on the challenging side of things. [25:37] And so it's in everybody's interest if we're all trying to actually deploy what [25:41] we think are the best solutions in this space. [25:44] There's no shortage of coverage of the scale of the AI build out. [25:50] I think on a personal level, I think it [25:52] would be a real shame if that supply chain was fractured. [25:56] You have people investing in totally different technologies and underlying [26:01] hardware just because they're trying to get some small advantage. [26:07] I'm really excited that this is going to be an open standard. [26:09] I think it will really benefit other people outside of OpenAI. [26:13] It also benefits all of us if we are kind of all pushing in the same direction. [26:17] Infrastructure is kind of this shared fate of the whole industry. [26:22] And I think it is a very good thing that [26:24] we are open sourcing this and kind of bringing everyone along. [26:28] It seems like it's beneficial, too, [26:30] because everything is becoming very collaborative. [26:33] And even, you know, you take a project like Stargate, which is multiple [26:38] locations, many partners across the world, and Microsoft, Fairwater, and sort of this [26:43] idea that compute is a thing that there's never going to be enough of it. [26:48] and the more we kind of work with each other [26:50] to figure out how to maximize it and continue doing it, [26:52] probably the better it's going to be [26:54] for everybody involved than treating it like, [26:57] I mean, it is a very limited resource, but these protocols, like you said before, [27:02] you know, with Ethernet, whatnot, [27:03] that's really what gave us, you know, what we have and things like the World Wide Web [27:06] and a lot of the cool things [27:07] we realize, oh, share this because what we're going to benefit [27:10] is going to be so much better. Yeah, what we're trying to do is hard enough [27:13] without everybody having to reinvent the wheel all the time. We think this is, [27:17] the right way to go and we'd like [27:18] everybody else to go in the same direction as us. [27:21] Where are the limits of this? [27:23] MRC is a flexible standard. [27:26] It builds on top of Ethernet. [27:27] So as Ethernet scales, so will MRC. [27:31] You can think of Ethernet as kind of the [27:34] protocol that individual devices use to talk to each other. [27:38] MRC sits on top of that. [27:40] It incorporates kind of this static routing that Mark is talking about. [27:45] It incorporates what we call congestion [27:47] control, which is basically if we do end up in situations because of failed links [27:51] or choices that we make about how we send traffic, how the endpoints should react to [27:56] that to make sure that we kind of use the network fairly and efficiently. [28:02] My experience with networking is that there will always be more work to do. [28:05] There's always going to be ways we can improve that, make the network more fair. [28:09] There's fundamental limits on networks. [28:11] Specifically, the speed of light is a known speed limit. [28:14] And so the amount of time that it takes [28:16] for light to get from one point to another in a network has some lower limits on it. [28:20] But we're going to keep making each of those links faster and faster. [28:23] And so that will always kind of change the operating point of basically how much data [28:27] you have outstanding per connection at a given time. [28:30] And that will always require kind of [28:31] ongoing engineering effort to make sure that we're making the best use of the [28:35] hardware that we have in a given generation. [28:38] But I think MRC gives us a very flexible [28:40] and strong base to build on as we continue to push through the next few generations. [28:45] And the key thing is because, as you say, it's Ethernet-based. [28:48] I mean, Ethernet now is not what Ethernet [28:49] was 10 years ago or 20 years ago or 30 years ago or even 40 years ago. [28:53] Ethernet itself has evolved so much over the years. [28:57] And what we're doing is we're taking [28:58] advantage of all that development by the whole world's networking industries. [29:03] And so we want to make sure we carry on [29:05] riding that wave of innovation that has taken Ethernet [29:08] forward. So given that all of that's [29:11] happening anyway, MLC, because it takes the intelligence [29:16] pushed at the edge of the network, we can [29:18] scale the cores of our networks as long as Ethernet [29:21] keeps scaling. And there's no particularly [29:23] obvious reason why that's not going to keep scaling for [29:26] at least the near future. And who knows, [29:29] maybe smarter people than me will figure out how to [29:31] make it work for another 40 years. One of [29:33] the key things we're doing, though, is trying to actually [29:35] move the complexity out of the network. So [29:38] as I mentioned, we turn off the routing and each [29:42] packet is actually source-routed through [29:43] the network. We're using a technique called IPv6 [29:46] segment routing, which allows each [29:48] individual packet's address to list the precise set of [29:51] switches the packet goes through as it [29:53] goes through the network. And that means that the [29:56] switches themselves can be really dumb. [29:58] It's really nice to be able to sort of simplify a [30:01] of the network. And as you're trying to scale and make things scale reliably, [30:04] making the middle of the network as simple as possible has huge benefits to us. [30:07] the network as simple as possible has huge [30:09] benefits to us. The other piece here is, you mentioned [30:12] Ethernet has this rich, wonderful history. [30:14] We're still building on Ethernet. That's because it's [30:16] an open standard. That's because the [30:17] entire industry has bought it and is pushing in the [30:19] same direction. And that's exactly what we [30:21] want to see from MRC also, is we want to see basically [30:24] the next layer get ready for the systems [30:27] challenges of AI and get widely adopted. [30:31] We don't think that this would do as well if it was an OpenAI exclusive. [30:36] With OpenAI investing the time and energy [30:38] and the people into solving these problems, [30:41] it does seem like even though you're [30:43] making MRC available to everybody by having teams now working on this, [30:46] it's got to be providing a lot of benefits and advantage. [30:50] I try to get a lot of benefits. So again, my role is from the perspective [30:54] of like, how do we make the most use of what we have? [31:00] Not literally in a conservationist sense, but in some sense it is, right? [31:03] Like if we're going to be burning the [31:05] power on these things, we want it to be used productively [31:08] and we want it to be used efficiently. Another one of the advantages of MRC is [31:12] because it has this property of allowing us to spray [31:15] over multiple paths, we can build much [31:18] simpler, much smaller networks that have many fewer devices in it. [31:23] So this is not an obvious property, but [31:25] basically we're able to build networks that are much [31:28] flatter and basically have many fewer [31:31] layers of switches and use much less power. They also cost [31:34] a lot less. And the amount of useful work [31:37] you can do per watt goes up when you do that because you're [31:40] not spending extra power on these extra [31:42] layers of switches. The power is more generally going [31:45] directly to the GPUs and allowing them to actually do work. [31:51] When you train models and start off with text models and with certain context [31:53] windows, so there's a certain amount of data that needed to go [31:55] about. And then as you go into multimodal [31:57] models, is there a difference between training, let's say, [32:00] an image model or a video model [32:03] or just a multimodal model? [32:07] I probably can't get into the details of [32:09] model architecture, but I will say that as our models get more advanced, [32:13] the demands that they place on the systems [32:18] get substantially harder. The amount of data that [32:24] you have to move and the latency bound on doing that. [32:26] So basically, the regret that you have if [32:28] your network goes a little bit too slowly just [32:31] gets worse and worse and worse as the [32:33] training cluster size gets bigger and as you continue [32:35] to optimize the rest of the stack. [32:38] Because obviously we have many, many very smart people who are trying to push in the [32:44] same direction and make these models very, very efficient. [32:48] We all want the next version of ChatGPT faster. We want it to be smarter. [32:54] One of the big strengths of OpenAI is that [32:56] we have all of these incredibly smart people pushing in one direction. [33:01] And you have the researchers and you have [33:02] the infrastructure people and everyone kind of knows what our goal is. [33:06] And so we have very smart people who are [33:07] trying to make the work that happens on the GPU go faster. [33:12] And all that means is that now we have a [33:13] tighter bound on how long the network has to transfer things. [33:16] Because if we are the ones who are lagging behind, their work doesn't matter. [33:21] And so our work never stops. [33:24] As these things get bigger, the variance [33:28] in whatever the slowest computation is goes up. [33:33] We have to work on controlling that tail. [33:36] Similarly, on the network side of things, you need to pull things in. [33:41] One piece that when Mark was talking about this splitting over many network links, [33:46] one thing that's mildly not obvious about this is that if you did this without MRC, [33:52] the tail statistics actually get much worse. [33:56] So if you take the same amount of bandwidth [33:58] and have more paths, [34:01] you end up basically having worse tail statistics [34:04] because you're throwing the same number of balls into more bins. [34:07] The ratio of the worst bin to the average bin gets much worse. [34:13] And so that deterministic routing that Mark talked about [34:17] where we can do this kind of very careful load balancing [34:20] across these very huge number of links is [34:23] very important to avoid getting into a bad situation [34:28] that we would then have to kind of have a feedback loop to recover from. [34:32] And so kind of all of these pieces of the [34:35] stack are extremely tightly coupled together, right? [34:38] It is extremely important that we have [34:40] kind of low-level network hardware people who have [34:43] some understanding of what is happening at the workload layer. [34:45] And we have people at the workload layer [34:47] who have some understanding of what the heck is [34:49] going on inside a network switch. [34:52] We really wouldn't be able to push the boundaries of how big [34:54] you can make these systems without that kind of vertical [34:56] integration and everyone pushing in the same direction. [34:59] When people talk about data centers-- [35:00] and you mentioned this earlier-- [35:02] there's what you use for training, there's what you use for inference. [35:04] When I asked ChatGPT, I don't need to talk to the whole data center, I [35:06] need a few GPUs who are going to answer this question. [35:09] And there's been talk about, well, the next step [35:11] is we're going to put things into space. [35:12] And my question's always been like, [35:14] I can get-- like, I have a satellite with some GPUs. It's doing inference. [35:18] But when you're literally spreading things [35:20] out across thousands or hundreds of thousands of miles on a big network, it [35:24] seems like you lose all of your advantage for speed. [35:26] The speed of light is your enemy versus being in one center. [35:29] It's hard to envisage doing the sort of [35:32] training that we do in our Stargate data centers in space. [35:37] Just the latency would be a huge problem. [35:39] And just the background rate of failures would be a problem. [35:43] We have technicians from Microsoft and Oracle who have to go in and fix things [35:46] all the time every day. Hard to do that in orbit. [35:49] Yeah. Yeah. I think you can make all sorts of arguments. And I have [35:54] gotten, I've gone very deep on this. As I [35:55] said, I have a physics background. I'm very interested in, [35:58] you know, I worked with people who designed satellites. You know... [36:03] things like that. Yeah. But I think a lot [36:04] of smart people have had very reasonable arguments both [36:07] ways on that dimension. The major [36:09] barriers, I think, are the rate of failure. I mean, [36:13] every GPU, every generation of GPU, the [36:15] GPU itself gets more powerful and more expensive [36:19] and more important. I think we are doing [36:22] incredible work here on earth to try to kind [36:25] of route around failures automatically but [36:27] i think that you would find yourself with a lot of hardware [36:30] that you couldn't use very quickly if [36:32] you're shipping these things into space now is there a [36:35] world in which you can also put [36:37] technicians in space maybe you can do all sorts of things the [36:42] dreamer side of me says that would be [36:44] really really cool yeah the practical side of me says it's [36:46] really really hard to do this stuff on [36:48] earth yeah like every day we are trying to push limits on all [36:51] sorts of dimensions even just spinning up [36:54] MRC was a huge effort that required very close [36:58] collaboration with us and engineers at a [37:01] number of other companies and it required in some cases [37:04] you know hands-on machines to fix things [37:06] to test things etc these systems are hard enough to build [37:09] and make work and make perform here on [37:11] earth i think trying to push the boundaries of that and [37:15] also adding additional complications, you [37:17] have to make a really strong case for why it makes sense to do it in space. [37:21] So build more terrestrial [37:24] compute centers. [37:27] Please. I mean, that is what we are trying to do here, is build a lot of compute so [37:30] that we can increase the net amount of intelligence in the world. [37:33] That's awesome. Gentlemen, thank you very much.