Meta Opens AI Transport Protocol MetaRoCE to Cut Network Bottlenecks

@Scale

Meta is open-sourcing its MetaRoCE transport protocol and Keysight compliance suite to eliminate vendor lock-in and optimize commodity Ethernet for multi-million GPU AI clusters.

AI-assisted summary of MetaRoCE: Meta’s RDMA Transport | Arvind Srinivasan, Meta and Kingshuk Mandal, Keysight — original source. Check the original for full context.

Source published: 2026-09-13

Detailed notes

## Background Published on YouTube by @Scale on August 27, 2026, this presentation features Arvind Srinivasan from Meta and Kingshuk Mandal from Keysight introducing MetaRoCE—Meta's clean-sheet, edge-driven RDMA transport protocol built for AI clusters scaling to over a million GPUs. The talk and page description highlight how MetaRoCE shifts intelligence to the host NIC on commodity Ethernet to decouple transport from switch topology, while presenting an open conformance framework co-developed with Keysight to build a multi-vendor ecosystem.

## TL;DR Meta introduced MetaRoCE to eliminate network bottlenecks and tail latency in ultra-large AI clusters by moving transport intelligence from switches directly to the edge NIC. The protocol relies on per-packet multipath spraying across independent entropy paths, single multipath connections per endpoint, and per-path congestion control to achieve high resilience and network utilization without requiring proprietary switch extensions. To prevent vendor lock-in and ensure specification accuracy, Keysight developed the MetaRoCE 1.2 Conformance Test Plan—a continuous validation suite that catches subtle edge-case bugs like frozen RNR timers and shared-RTO timer coupling across software, RTL simulation, and hardware silicon.

## Atmosphere & style The session delivers an authoritative, technical, and forward-looking presentation driven by deep system-engineering expertise. The handoff between Meta and Keysight creates a balanced narrative dynamic: Meta establishes the high-stakes architectural vision of scaling multi-million GPU networks, while Keysight provides concrete, diagnostic pragmatism through real-world failure case studies. The tone is collaborative yet practical, underscoring the stark economic reality of hardware tap-outs while inviting the broader network ecosystem to build on open standards.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) Meta designed MetaRoCE to support AI clusters scaling to over 1,000,000 GPUs where traditional RDMA bottlenecks large-scale compute. - [02:01](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=121s) MetaRoCE shifts transport intelligence entirely from switch fabrics to edge NICs, allowing commodity Ethernet switches to use flexible, multiplane topologies like FPF. - [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) The protocol targets 90% network efficiency by using per-packet spray over independent paths, reassembling directly in receiver memory without explicit reorder buffers. - [06:27](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=387s) MetaRoCE achieves resilience by isolating failure and congestion tracking to individual paths, preventing a single degraded link from contaminating healthy ones. - [06:27](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=387s) To eliminate flow-competition bottlenecks, MetaRoCE replaces traditional Queue Pair (QP) proliferation with a unified multipath connection per endpoint. - [08:51](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=531s) Meta opened the MetaRoCE specification to prevent vendor lock-in and partner with the hardware ecosystem on commodity Ethernet deployments. - [09:47](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=587s) In partnership with Keysight, Meta established the MetaRoCE 1.2 Conformance Test Plan to validate implementations across software, pre-silicon RTL, and production silicon. - [12:57](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=777s) Testing with Keysight's Anvil engine uncovered a critical Receiver Not Ready (RNR) timer bug where DUT senders failed to resume transmission after timeout expiry. - [15:06](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=906s) Hardware implementations that consolidate retransmission timers (RTO) into a single per-QP timer severely degrade aggregate throughput by coupling fast path speeds to the slowest path. - [18:43](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=1123s) Meta is expanding its clean-sheet network architecture beyond scale-out transport into scale-up rack networks and regional front-end connectivity.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) Scaling AI Clusters to Millions of GPUs

Arvind Srinivasan, an engineer in Meta's Network Infrastructure group, introduces MetaRoCE—Meta's purpose-built RDMA transport protocol designed specifically for AI networking at scale.

As Meta scales its AI cluster architecture from tens of thousands to over one million GPUs across data halls, data centers, and entire regions, the demands on the underlying network grow exponentially. Connecting a million accelerators requires a staggering web of "trillions of wires," shifting the network's role from background infrastructure to the single defining bottleneck of the system. In modern AI clusters, the interconnect determines whether a massive pool of compute can function cohesively.

The modern AI training step is gated entirely by network performance. Arvind compares collective communication operations to an army on the march: the whole formation moves only as fast as its slowest soldier. Any straggler node, tail latency, or fabric overhead directly translates into billions of dollars of stranded, underutilized compute. Tail latency and network resilience cease to be purely technical details—they become a massive economic challenge.

- Meta's target cluster size: over **1,000,000 GPUs** spanning multiple data halls and regions. - High-scale interconnect scale requirement: "Trillions of wires" to interlink the compute nodes. - Financial impact: Network overhead directly translates to billions of dollars in idle compute time.

---

[02:01](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=121s) Shifting Intelligence from the Fabric to the Edge

RDMA has served as the workhorse for High-Performance Computing (HPC) for decades. However, as cluster sizes scale beyond tens of thousands of nodes, the traditional RDMA architecture begins to break down. Designed to be workload-agnostic, standard RDMA relies on passing coarse hints to the network fabric in hopes that switches can correctly manage flow paths. In practice, this leads to persistent congestion, Pause frame storms, constant operational firefighting of hotspots, and severe bandwidth underutilization due to strict in-order packet delivery.

The fundamental issue is that the switches within the network fabric only see individual packets; they lack visibility into the overall application intent. Only the network edge—the host NIC—has full context.

To solve this, Meta is shifting network intelligence entirely to the edge. When a NIC has fine-grained visibility across individual network paths, it can manage resilience and drive aggregate performance directly. Edge-driven load balancing ensures high availability without requiring the network switches to be complex or "smart." By moving complexity to the edge transport, operators gain the freedom to innovate on underlying switch topologies, enabling TCO-optimized, multiplane fabrics like FPF (Flexible Physical Fabric). Instead of presenting the network to the transport as a single opaque pipe, Meta expose many fine-grained "tiny pipes," enabling precise packet-level decisions while remaining completely transparent to the application.

- Standard RDMA limitations: Flow pinning and in-order delivery cause severe hotspotting and underutilization. - Core philosophy: Network switches only see packets; only the edge (NIC) understands application intent. - Topology benefit: Edge-driven transport unlocks multiplane, TCO-optimized topologies such as FPF.

---

[04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) Core Architecture of MetaRoCE

MetaRoCE is a clean-sheet RDMA transport purpose-built for AI workloads on commodity Ethernet switches. Rather than applying superficial patches to legacy RDMA, Meta architected a solution designed to target three primary dimensions of scale:

1. **Performance at Scale:** Target 90% network efficiency using per-packet multipath spraying and fine-grained load balancing. 2. **Resilience at Scale:** Integrate per-path reliability to maintain high availability without requiring fabric-level intervention or application awareness. 3. **Operations at Scale:** Ensure frictionless adoption for existing applications while running on standard, commodity Ethernet switches without proprietary switch features.

Paths form the foundational building block of MetaRoCE. Traditional RDMA pins each flow to a single network path, creating massive "elephant flows" that cause severe hotspots. MetaRoCE takes the opposite approach: when an application posts multiple messages to the transport, the NIC chunks them into individual packets and sprays them across the entire network fabric.

Each packet takes a different path and is reassembled directly into the receiver's memory buffer without requiring dedicated reorder buffers. Every path is assigned a distinct entropy value fully managed by the transport, maintaining independent reliability and congestion states. A per-path load balancer dynamically routes traffic based on measured Round Trip Times (RTT) and available window sizes, allowing the protocol to scale smoothly across non-uniform RTT environments.

- Network efficiency goal: **90% network utilization**. - Packet handling: Per-packet spraying across independent entropy paths with direct-to-memory reassembly (no explicit reorder buffer required). - Switch hardware requirement: Commodity Ethernet running standard ECN (Explicit Congestion Notification) and ECMP (Equal-Cost Multi-Path)—no vendor-proprietary switch extensions.

---

[06:27](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=387s) Resilience, Congestion Control, and Application Decoupling

At massive scale, hardware and link failures are routine operational occurrences rather than exceptional events. In MetaRoCE, logical transport paths map directly to physical fabric planes. When a plane fails, traffic instantly shifts to alternate healthy planes without application involvement. Because every path independently tracks its own RTT and congestion state, a failure or slowdown on one path never impacts or drags down active healthy paths. This complete lack of cross-contamination builds real-time resilience directly into the transport, eliminating the need for operator intervention or fabric-level recovery routines.

MetaRoCE utilizes sender-based congestion control executed at a per-path granularity. When Explicit Congestion Notification (ECN) signals the transport to back off, only the specific congested path reduces its rate. The total available transmission window is the aggregate sum of all individual path windows, allowing operators to scale total bandwidth by adding paths without increasing protocol aggressiveness. Additionally, receivers send grant-based acknowledgements to compute fair-share receiving rates, guaranteeing rapid convergence during incast events.

On the operational side, traditional RDMA scales by multiplying Queue Pairs (QPs)—forcing applications to manage dozens of uncoordinated flows that compete with one another and break down under load. MetaRoCE replaces this QP proliferation with a single multipath connection. Multiple independent application streams share a single, coordinated congestion control engine that dynamically balances traffic across paths.

- Congestion control model: Per-path sender-driven control with receiver-backed grant acknowledgements. - Incast behavior: Fast rate convergence via explicit receiver fair-share grants. - Connection model: Replaces multi-QP proliferation with a **single multipath connection** per application endpoint.

---

[08:51](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=531s) Opening MetaRoCE to the Ecosystem

Arvind emphasizes that Meta's goal is straightforward: keep GPUs running at full capacity on high-efficiency, resilient network infrastructure without vendor lock-in, proprietary hardware extensions, or per-workload customizations. To achieve this, Meta is fully opening the MetaRoCE protocol specification to the broader networking industry.

However, openness requires rigorous interoperability. For implementations from different silicon and software vendors to work together seamlessly, the community needs a comprehensive compliance suite.

Arvind introduces Kingshuk Mandal, a network engineer from Keysight, to detail how Keysight and Meta partnered to create a robust test and compliance framework for MetaRoCE implementations.

- Hardware strategy: Fully standard commodity Ethernet switches relying exclusively on standard ECN and ECMP. - Ecosystem goal: Eliminate vendor lock-in and proprietary transport extensions across the AI networking landscape.

---

[09:47](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=587s) Building a Conformance Framework from Day Zero

Kingshuk Mandal shifts the focus to the broader network ecosystem. He highlights a core truth of protocol design: a brand-new transport protocol becomes a true ecosystem the exact moment two independent vendors agree on the wire format bit-for-bit. Until that point, a protocol implementation communicating only with itself is merely a private, proprietary protocol.

For a new specification like MetaRoCE, the written specification is the very first "product"—shipping long before silicon is taped out. Conformance testing transforms static written words into testable, automated assertions, uncovering ambiguities early when software edits are still cheap. Once silicon freezes, any ambiguity remaining in the specification becomes permanently baked into the physical hardware, where patching is brutally expensive.

To address this, Keysight built the official **MetaRoCE 1.2 Conformance Test Plan**, which is defined by four core attributes: 1. **Purpose-built:** Developed completely from scratch specifically for MetaRoCE. 2. **Proven against reference code:** Every test is validated against Meta's software reference implementation, establishing a firm, real-world fast-fail bar. 3. **Comprehensive scope:** Covers the entire domain—including header integrity, retransmissions, and congestion control algorithms. 4. **Focus on edge cases:** Tests all recovery corners, including NACKs, tricky timers, timeouts, and loss recovery paths where critical bugs hide.

Keysight's test methodology provides a continuous validation pipeline using the exact same test suite across the entire lifecycle: starting at Day 0 with the software reference implementation, moving to pre-silicon RTL simulation, and ending with production silicon. Catching a protocol bug at the reference stage requires a simple code edit, whereas catching that same bug post-tapout can cost a full silicon spin.

- Conformance standard: Keysight Conformance Test Plan for **MetaRoCE 1.2**. - Unified testing continuum: Same test cases run across **Software Reference → Pre-Silicon RTL Simulation → Production Silicon**. - Economic rationale: Bugs caught in software are quick code fixes; bugs caught after silicon freeze require multi-million-dollar hardware spins.

---

[12:57](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=777s) Case Study 1: Receiver Not Ready (RNR) Timer Freeze

Kingshuk presents a real-world case study using Keysight's Anvil test engine to illustrate the danger of untested edge cases. While normal "happy path" traffic runs smoothly during daily lab testing, "unhappy path" failure modes sit undetected until triggered by complex production workloads like Mixture-of-Experts (MoE).

In MoE workloads, models are split across hundreds of machines. At specific steps, nodes attempt all-to-all communication simultaneously, causing receiver memory buffers to temporarily fill up. The receiver is not broken; it simply needs time to catch up. Under MetaRoCE, the receiver sends a polite Receiver Not Ready (RNR) frame asking the sender to pause transmission for a precise duration on a specific tag.

During testing with the Anvil tool acting as the receiver and a Device Under Test (DUT) acting as the sender, Anvil issued an RNR pause frame for a single tag. The specification mandates that the sender must pause only the affected tag while all other tagged traffic on the Queue Pair continues to make progress. While the DUT correctly paused the specified tag, a critical bug was discovered: the sender **never resumed transmission** after the RNR timer expired, leaving that flow permanently frozen.

- Workload trigger: Mixture-of-Experts (MoE) all-to-all communication bursts. - Expected behavior: Precise, surgically scoped pause on a single tag; immediate resume upon timer expiry; unimpacted tags continue transmitting. - Identified bug: Sender correctly paused the requested tag but failed to resume after the timeout, hanging the flow indefinitely.

---

[15:06](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=906s) Case Study 2: Per-Path Retransmission Timers (RTO)

The second case study examines network-induced failures and the temptation of taking hardware design shortcuts. In MetaRoCE, packets are sprayed across multiple paths simultaneously, and the sender must track retransmission timeouts (RTO) on a per-path basis. If an RTO timer is set too short, the sender prematurely transmits duplicate packets; if set too long, flow completion and overall job completion time (JCT) suffer severely.

The MetaRoCE specification explicitly requires **an independent retransmission timer for every single path**. However, hardware designers face a strong temptation to implement a single shared timer per Queue Pair to save chip state and memory.

Keysight tested this behavior by configuring Anvil to simulate two paths with contrasting characteristics: Path 0 (a fast, low-latency path) and Path 1 (a slow, congested path). - In a **compliant implementation**, Path 0's RTO settles to a low floor while Path 1's RTO independently climbs to absorb its local delay. - In a **flawed implementation** using a single shared timer, the congested path drags the timeout value of the fast path up alongside it.

Sharing timers completely destroys the benefits of multipath routing. A single degraded or "sick" path poisons every healthy path in the connection, forcing an 8-lane network to run at the speed of its single worst lane. Because this flaw never manifests in clean lab environments where all paths are healthy, it only surfaces in production as mysterious latency coupling under heavy load—taking engineers weeks to diagnose without dedicated conformance tools.

- Protocol requirement: Independent Retransmission Timeout (RTO) tracking per path. - Design pitfall: Consolidating into a single shared timer per Queue Pair to reduce hardware state/memory. - Production impact: A single congested path inflates the timeout of all healthy paths, degrading an 8-lane network to the speed of the slowest link.

---

[17:27](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=1047s) The Path Forward for AI Transport Networks

Kingshuk concludes by outlining why early, test-driven specification development is essential for the entire industry: - **For Specification Authors:** Test assertions convert ambiguous language into precise, testable requirements while hardware designs remain fully editable. - **For Network Architects:** Guarantees true multipath performance, proper path separation, and proven failover mechanics verified through rigorous fault injection rather than basic happy-path assumptions. - **For Fabric Operators:** Prevents subtle, workload-skew bugs (such as all-to-all burst stalls) from ever reaching production datacenters.

A protocol with only one implementation remains a private, proprietary technology. By pairing MetaRoCE with a rigorous, Day-0 compliance suite, Meta and Keysight are building an open, multi-vendor ecosystem for next-generation AI networks.

- Final thesis: Hardware is brutally expensive to patch; establishing compliance on Day 0 turns a single company's transport protocol into a robust industry ecosystem.

[18:43](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=1123s) The Horizon of MetaRoCE and Open Collaboration

Arvind Srinivasan takes the stage back from Kingshuk Mandal to outline the current state and future vision of MetaRoCE as Meta pushes the boundaries of AI infrastructure. Emphasizing the immense scale of modern operations, he describes how network demands have reached unprecedented levels as Meta tackles connectivity for several million GPU clusters—a problem space involving "trillion wire connectivity across billions of accelerators." In the face of these staggering requirements, MetaRoCE delivers precisely what AI workloads demand: peak performance paired with high availability, all running reliably on commodity Ethernet without relying on specialized switches or falling into vendor lock-in.

Meta is opening MetaRoCE to the broader hardware and networking industry, inviting the entire NIC ecosystem to implement the protocol. Srinivasan notes that the specification provides full flexibility in design; it strictly defines the wire behavior while leaving vendors free to choose how they build and optimize their hardware implementations. Looking ahead, Meta emphasizes that scale-out transport is only part of the journey. The engineering team is actively tackling challenges across scale-up networking within the rack as well as front-end connections across entire regions, asserting that "every layer of the AI network stack demands the same rethinking, and this is just the beginning."

- Meta is deploying MetaRoCE to solve ultra-large-scale connectivity across multi-million GPU clusters and billions of accelerators. - MetaRoCE runs entirely on commodity Ethernet, eliminating the requirement for proprietary switch features or vendor-locked ASIC designs. - The open specification prioritizes implementation flexibility, defining only the wire behavior to allow diverse vendor architectures. - Future focus areas extend beyond scale-out transport to encompass scale-up rack networks and regional front-end connectivity.

## Notable quotes > [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) trillions of wires — Arvind Srinivasan

> [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) The modern AI training step is gated entirely by network performance. — Arvind Srinivasan

> [02:01](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=121s) only the network edge—the host NIC—has full context. — Arvind Srinivasan

> [02:01](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=121s) tiny pipes — Arvind Srinivasan

> [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) Performance at Scale: — Arvind Srinivasan

> [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) Resilience at Scale: — Arvind Srinivasan

> [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) Operations at Scale: — Arvind Srinivasan

> [06:27](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=387s) single multipath connection — Arvind Srinivasan

> [09:47](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=587s) a brand-new transport protocol becomes a true ecosystem the exact moment two independent vendors agree on the wire format bit-for-bit. — Kingshuk Mandal

> [09:47](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=587s) MetaRoCE 1.2 Conformance Test Plan — Kingshuk Mandal

> [12:57](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=777s) never resumed transmission — Kingshuk Mandal

> [15:06](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=906s) an independent retransmission timer for every single path — Kingshuk Mandal

> [18:43](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=1123s) trillion wire connectivity across billions of accelerators — Arvind Srinivasan

> [18:43](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=1123s) every layer of the AI network stack demands the same rethinking, and this is just the beginning. — Arvind Srinivasan

## Mentioned Arvind Srinivasan, Kingshuk Mandal, Meta, Keysight, MetaRoCE, MetaRoCE 1.2, MetaRoCE 1.2 Conformance Test Plan, Anvil, Ethernet, RDMA, FPF, Flexible Physical Fabric, ECN, ECMP, QPs, Queue Pairs, RNR, RTO, JCT, MoE, @Scale

[Source](https://www.youtube.com/watch?v=SpyWXkWwb1A)

中文简报

Meta开源AI传输协议MetaRoCE以消除网络瓶颈 Meta开源其MetaRoCE传输协议与Keysight合规套件，旨在消除厂商锁定，并针对数百万GPU规模的AI集群优化通用以太网。 Meta推出MetaRoCE，通过将传输智能从交换机直接转移至边缘NIC，消除超大型AI集群中的网络瓶颈与长尾延迟。该协议依靠跨独立熵路径的逐包多路径分发、每端点单多路径连接以及逐路径拥塞控制，在无需专有交换机扩展的情况下实现高弹性和网络利用率。为防止厂商锁定并确保规范准确性，Keysight开发了MetaRoCE 1.2一致性测试计划，该持续验证套件可跨软件、RTL仿真和硬件芯片捕获冻结RNR定时器与共享RTO定时器耦合等细微边缘场景缺陷。针对规模超过 100 万 GPU 的 AI 集群通信瓶颈，Meta 推出 MetaRoCE 协议，将传输控制智能全面转移至边缘网卡（NIC），使通用以太网交换机能高效支持多平面 FPF 拓扑。 (2:01) 该协议通过在独立路径上执行逐包分发（per-packet spray）并在接收端内存直接重组，目标实现 90% 的网络利用率，同时用单一多路径连接取代传统繁冗的 QP 激增架构。 (4:16) MetaRoCE 在单路径粒度上独立跟踪 RTT 与超时重传（RTO），彻底隔离各路径状态，防止单条降级或慢速链路拖慢整个连接的健康路径。 (6:27) Meta 携手 Keysight 建立 MetaRoCE 1.2 一致性测试套件，贯穿软件参考实现、前仿 RTL 到量产芯片全流程，并成功检测出接收端未就绪（RNR）超时后发送端无法恢复传输等严重边缘漏洞。 (9:47) Meta 全面开源 MetaRoCE 规范以避免硬件厂商绑定，在严格规范线缆报文行为的同时赋予实现高度自由，并计划将重构经验扩展至机架内 scale-up 与区域前端网络。 (18:43)

Reading summary

Arvind Srinivasan of Meta and Kingshuk Mandal of Keysight present MetaRoCE, an open RDMA transport protocol engineered to eliminate tail latency and network bottlenecks across AI clusters scaling to over one million accelerators. The architecture shifts network intelligence to host NICs using per-packet spraying over commodity Ethernet, backed by an ecosystem test suite to eliminate pre-silicon ambiguities. Economic and Operational Limits of Legacy RDMA (0:05) As Meta expands AI cluster architectures from tens of thousands toward more than one million GPUs across data halls and regions, the network interconnect becomes the primary operational bottleneck. In modern training workflows, collective communication operations move strictly at the pace of the slowest node. Any straggler latency or fabric overhead directly strands compute capacity, creating an economic challenge involving billions of dollars in idle GPU time. Srinivasan notes that traditional RDMA, designed decades ago for High-Performance Computing, breaks down at massive scale. Because legacy RDMA treats network paths as opaque pipes and pins individual flows to single routes, it relies on coarse switch-level management. This design frequently triggers Pause frame storms, persistent congestion hotspots, and severe bandwidth underutilization due to rigid in-order packet delivery requirements. Edge-Driven Multipath Routing on Commodity Ethernet (4:16) MetaRoCE shifts routing intelligence from network switches to the host NIC, operating on the premise that internal switches only see raw packets while the network edge understands overall application intent. Rather than relying on smart switches or proprietary ASIC features, the transport runs on standard, commodity Ethernet using standard Equal-Cost Multi-Path (ECMP) and Explicit Congestion Notification (ECN). This edge-driven decoupling enables cost-optimized multiplane topologies such as Meta's Flexible Physical Fabric (FPF). To achieve an efficiency target of 90% network utilization, MetaRoCE eliminates traditional elephant flows by chunking application messages into individual packets and spraying them across independent entropy paths. The receiver reassembles these packets directly into its target memory buffer without requiring dedicated reorder buffers. A per-path dynamic load balancer actively directs traffic based on round-trip time (RTT) measurements and current window sizes. Per-Path Congestion Control and Fault Isolation (6:27) MetaRoCE maps logical transport paths directly to physical fabric planes to isolate failures. Because each path maintains its own RTT tracking and congestion state, a slowdown or link degradation on one path does not drag down or cross-contaminate healthy paths. When a fabric plane fails, traffic automatically shifts without application awareness or fabric-level intervention. Congestion control is sender-based and enforced per path. When ECN signals congestion, only the impacted path throttles down, while the connection's total transmission window scales as the aggregate of all individual paths. Receivers issue grant-based acknowledgements to establish fair-share rates and ensure rapid convergence during incast bursts. Operationally, MetaRoCE replaces traditional Queue Pair (QP) proliferation—where applications manage dozens of uncoordinated, competing flows—with a single multipath connection per application endpoint. Pre-Silicon Conformance and Hardware Verification (9:47) To transition MetaRoCE from an internal design into an open, multi-vendor standard, Keysight developed the MetaRoCE 1.2 Conformance Test Plan. Mandal emphasizes that a protocol becomes an ecosystem only when independent vendors agree on bit-level wire formats. Because hardware spins are expensive once silicon freezes, Keysight's test suite runs continuously across the entire engineering pipeline: software reference implementations, pre-silicon RTL simulation, and production silicon. Keysight's Anvil test engine uncovered critical edge-case flaws in non-compliant designs. In one case involving Mixture-of-Experts (MoE) all-to-all communication bursts, a Device Under Test paused when receiving a Receiver Not Ready (RNR) frame for a specific tag, but failed to resume transmission after the timer expired, permanently stalling the flow. In another scenario, test configurations simulated unequal network paths to enforce the rule requiring independent Retransmission Timeouts (RTO) per path. Implementations that improperly shared a single timer across a QP allowed a single congested link to poison the RTO of healthy paths, degrading multi-lane network throughput to the speed of the slowest link. Meta is releasing the MetaRoCE wire specification to the industry, permitting vendors architectural freedom in hardware implementation while relying exclusively on commodity Ethernet switches. Srinivasan indicates that scale-out transport represents only the initial phase, with Meta's network engineering group currently extending similar architectural redesigns to scale-up intra-rack networking and regional front-end fabrics.

精炼阅读摘要

Meta 的 Arvind Srinivasan 与 Keysight 的 Kingshuk Mandal 介绍了专为超大规模 AI 集群设计的 MetaRoCE 传输协议，该方案通过将控制智能从交换机转移至主机网卡，并结合从研发初期介入的一致性测试套件，旨在通用以太网上实现 90% 的网络利用率并构建开放的跨厂商生态。 AI规模化瓶颈与将智能转移至网卡边缘 (0:05) Srinivasan 指出，随着 Meta 的 AI 集群扩展至跨数据大厅和区域的 100 多万张 GPU，底层需要连接“数万亿条线路”，使网络成为整个计算系统的核心瓶颈。在现代 AI 训练中，集合通信操作类似军队行军，整体进度完全受制于最慢的节点；任何长尾延迟、离群节点或织构开销，都会直接导致价值数十亿美元的算力闲置浪费。然而，传统 RDMA 依靠工作流绑定（flow pinning）和严格的保序传输，在大规模场景下极易引发持续拥塞、暂停帧风暴及严重的带宽欠载。 Meta 认为该问题的根源在于交换机仅能处理孤立数据包，缺乏对上层应用意图的全局感知，而网络边缘的主机网卡（NIC）才具备完整上下文。为此，Meta 将网络智能完全转移至边缘网卡，由网卡直接管理路径弹性与细粒度负载均衡，使底层交换机保持通用与精简。这种架构解除了网络复杂性对硬件的绑定，使数据中心能够采用柔性物理网络（FPF）等多平面、低 TCO 的交换拓扑，通过向传输层暴露众多微型管道来实现精准的包级决策。逐包喷洒与路径级拥塞控制架构 (4:16) MetaRoCE 是一套直接运行于通用以太网交换机上的全新 RDMA 传输协议，仅依赖标准 ECN 和 ECMP，不要求任何专有交换机特性，设计目标为达到 90% 的网络利用率。传统 RDMA 将单流绑定至固定路径从而形成造成网络热点的大象流，MetaRoCE 则在网卡处将应用消息切片为独立数据包，并将其逐包喷洒（per-packet spraying）至整个网络。每个数据包携带独立的熵值沿不同路径传输，并在接收端内存直接重组，无需专用的重排序缓冲区。每条路径均由发送端单独监测往返时间（RTT）和拥塞状态，由动态负载均衡器按实时测量结果分发流量。在容灾与连接管理上，MetaRoCE 的逻辑路径与物理织构平面直接映射。当某一物理平面发生故障时，流量会自动且透明地转移至健康平面，单条路径的降速或中断绝不会跨路径污染其他通道。其拥塞控制在发送端以单路径为粒度执行，仅受阻路径会在触发 ECN 时下调传输速率，同时接收端会返回基于许可的确认（grant-based ACK）以保障微突发（incast）事件下的快速收敛。在操作层面，MetaRoCE 摒弃了传统架构中成倍扩展队列对（QP）的做法，每个应用端点仅维持单一的多路径连接，由集中的拥塞控制引擎统一调度。第零天一致性测试与全生命周期验证 (9:47) Keysight 的 Kingshuk Mandal 强调，一项新型传输协议只有在两个独立芯片厂商在物理线缆格式上达到逐比特严格一致时，才能演变为真正的行业生态，否则仅属于私有协议。在硬件流片（tape-out）之前，书面规范本身就是首个交付产品；如果协议歧义未在软件阶段通过测试消除，一旦芯片制造冻结，后续硬件再版（silicon spin）的修复成本将极其高昂。为此，Keysight 联合 Meta 开发了 MetaRoCE 1.2 官方一致性测试计划。该测试套件直接对照 Meta 的软件参考实现进行基准验证，覆盖报头完整性、重传逻辑、拥塞控制算法，并重点针对否定应答（NACK）、复杂定时器、超时及丢包恢复等边缘场景构建测试用例。测试体系在整个硬件生命周期内执行统一标准，从前期的软件参考实现、流片前的 RTL 仿真，一直贯穿至最终的量产芯片，确保多厂商硬件在接入真实数据中心前完全符合规范。极端异常案例：RNR挂死与RTO定时器陷阱 (12:57) Mandal 通过真实测试案例展示了未覆盖边缘用例的潜在危害。在专家混合（MoE）模型进行全对全通信突发时，接收端内存缓冲区容易暂时耗尽并发送“接收端未就绪”（RNR）帧，要求发送端在特定标签（tag）上暂停传输指定时间。规范要求发送端仅暂停受影响标签，其余流量正常行进并在定时器到期后自动恢复。但在使用 Keysight Anvil 引擎对被测设备进行压力测试时发现，发送端虽然正确挂起了指定标签，却在超时后未能恢复传输，导致通信流永久冻结。另一测试案例揭示了硬件设计简化带来的性能隐患。MetaRoCE 规范强制要求每条路径必须具备独立的重传超时（RTO）计时器，但部分厂商为了节省芯片面积和内存状态，倾向于让整个 QP 共享单一计时器。Anvil 模拟快路径与拥塞慢路径的测试表明，共享计时器会导致慢路径的延迟直接拉高快路径的超时阈值。这种设计缺陷破坏了多路径机制的独立性，会导致 8 通道网络整体退化至单条最慢通道的吞吐水平，且在常规实验室测试中极难复现，往往在生产环境高负载下才会暴露为难以排查的延迟耦合。 Meta 已将 MetaRoCE 协议规范面向芯片与网络行业完全开源，仅严格约束物理线缆行为而保留内部实现的架构灵活性。Arvind Srinivasan 总结指出，当前 MetaRoCE 正在被部署用于解决覆盖数百万 GPU 的横向扩展（Scale-out）互联，但传输层重构仅是第一步，Meta 计划未来将相同的重构理念进一步推进至机架内部的 Scale-up 互联以及跨数据中心区域的前端网络架构中。

Detailed notes — zh

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布，来自 Meta 的 Arvind Srinivasan 与是德科技（Keysight）的 Kingshuk Mandal 在 @Scale 大会上的联合演讲。视频简介指出，在 Meta 百万级加速器集群规模下，网络已成为 AI 训练的关键路径，因此 Meta 推出了专为商用以太网打造的全新 RDMA 传输协议 MetaRoCE，并联合 Keysight 推出一致性测试套件，全面向行业开源以共建开放生态。 ## TL;DR 面对百万卡 AI 集群中集合通信长尾延迟造成的巨大算力闲置，传统依赖交换机智能的 RDMA 方案已难以为继，Meta 提出了将网络智能全面下沉至网卡（NIC）的 MetaRoCE 协议。该协议通过每路径粒度的逐包喷洒、免重排序缓冲区设计以及单路径独立拥塞控制，在普通商用以太网上实现了高达 90% 的网络利用率与天然故障隔离，彻底打破了多 QP 恶性膨胀困境。与此同时，Meta 与 Keysight 深度合作打造了严格的一致性测试套件，从 Day 0 软件参考实现贯穿至硬件仿真与物理流片，通过注入 RNR 死锁、独立 RTO 状态污染等真实极端用例，在芯片物理冻结前彻底消除协议歧义，推动跨厂商硬件在标准以太网上的逐比特互通。 ## Atmosphere & style 整场演讲呈现出典型硬核系统架构峰会的严谨与务实，节奏紧凑且信息密度极高。两位演讲者配合默契，Meta 的 Arvind 从宏观的万亿级连接经济账与协议重构切入，语气坚定且极具战略视野；随后 Keysight 的 Kingshuk 则迅速将讨论拉入显微镜级别，带着工程师特有的较真劲头直面极端故障与底层硬件缺陷。现场没有浮夸的营销包装，取而代之的是白板式的机制图解、真实踩坑案例以及对开放生态的深刻呼吁，带给观众一种在芯片流片前夕争分夺秒拆解技术炸弹的专业张力与工业级踏实感。 ## Bulletin - [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) Meta 正式开源专为百万级 GPU 超大规模 AI 训练设计的 MetaRoCE 协议，主张将网络智能全面下沉至边缘网卡（NIC），仅依赖普通商用以太网的 ECN 和 ECMP 即可达到 90% 的网络有效利用率。 - [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) MetaRoCE 彻底摒弃传统将流静态绑定至单链路的设计，采用逐包多路径喷洒与去重排序机制，数据包乱序到达后直接零拷贝写入接收端内存，省去了硬件重排序缓冲区的额外开销。 - [07:14](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=434s) 针对 AI 训练中常见的多 QP 恶性膨胀问题，MetaRoCE 用单条连接整合多路径拥塞控制，并在单路径粒度独立执行 ECN 调速与重传超时（RTO）管理，杜绝局部慢路径污染全局健康带宽。 - [10:47](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=647s) Meta 携手是德科技（Keysight）联合发布行业首个 MetaRoCE 1.2 一致性测试套件，打通从软件参考实现、RTL 仿真到物理芯片的全周期验证连续体，确保硬件流片前线缆行为实现逐比特对齐。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) 百万级 GPU 集群使得长尾延迟直接关联价值数十亿美元的算力闲置浪费，网络韧性与延迟控制已演变为纯粹的算力经济学问题。 - [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) 分布式 AI 集合通信严格受制于“最慢士兵”的木桶效应，任一节点抖动均会拖慢整个训练迭代步。 - [01:59](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=119s) 传统 RDMA 试图向交换网络注入提示信息的模式在大规模场景崩溃，交换网络仅见孤立数据包，唯有边缘网卡洞悉应用的真实通信意图。 - [01:59](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=119s) 智能下沉至网卡后，网络通信从不透明的单一粗管道解耦为细粒度微型管道，使底层得以引入多平面 FPF 等 TCO 优化拓扑。 - [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) MetaRoCE 将应用大消息切片后喷洒至带独立熵标签的多物理路径，实现超 90% 的网络利用率与天然的故障平面隔离。 - [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) 乱序到达的数据包直接根据内存偏移量写入目标缓冲区，完全消除了网卡硬件端设计昂贵重排序缓冲区的负担。 - [07:14](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=434s) 拥塞控制精细化至单路径级别，ECN 降速仅惩罚受阻子路径，配合接收端 ACK 反馈公平份额，实现多对一 Incast 的快速收敛。 - [07:14](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=434s) 以单条多路径连接替代上百个并发 QP，彻底终结了 QP 膨胀与相互缺乏协调导致的拥塞崩溃难题。 - [08:51](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=531s) Meta 坚持反专有绑定，MetaRoCE 仅要求商用交换机具备标准 ECN 和 ECMP，旨在打造无供应商锁定的通用基础设施。 - [08:51](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=531s) 开放生态的成熟标志在于两家及以上独立芯片厂商实现线缆级兼容，一致性测试是实现该契约的强制准则。 - [10:47](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=647s) 规范文本作为交付的首款产品必须在芯片流片前转为可测试断言，在 Day 0 阶段以低成本消除模糊地带，规避昂贵的物理流片重制。 - [10:47](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=647s) Keysight 一致性测试套件贯穿软件参考实现、RTL 仿真与物理芯片，以无缝验证连续体保障统一交付门槛。 - [12:57](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=777s) 测试套件深入覆盖逆境路径，成功捕获 MoE 密集并发突发下网卡接收端未就绪（RNR）超时后永久冻结通信的隐藏死锁缺陷。 - [15:06](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=906s) 网卡若违规共享单一 RTO 定时器，会导致拥堵路径的慢速直接拖累其余健康路径，使多通道高带宽降级至最慢速率。 - [17:27](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=1047s) MetaRoCE 仅严格约束线缆上的传输行为，为生态伙伴保留芯片微架构自主设计自由度，未来将持续向机架 Scale-up 及跨区域互联拓展。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) 百万级 GPU 规模下的网络挑战与经济账在 Meta 的基础设施蓝图中，AI 集群的建设规模正以前所未有的速度演进，从数万张 GPU 狂飙至跨越多个机房数据大厅、数据中心乃至跨区域的百万级 GPU 互联。Meta 网络基础设施团队的 Arvind Srinivasan 指出，为了支撑如此庞大的加速卡互联需求，网络规模呈现指数级暴增，“可以想象，要将数百万个加速器连接在一起，我们需要数以万亿计的线缆”。在这种极度庞大的规模下，网络互联不再只是隐身于后台的辅助性水暖管道（auxiliary infrastructure），而是直接跃升为决定整个 AI 训练成败的关键瓶颈——它决定了这上百万张 GPU 究竟能否作为一个整体协同运转，网络已经处于每一次 AI 训练迭代步（training step）的关键路径之上。 Arvind 用了一个非常形象的比喻来解释这一演变过程背后的残酷现实：“集合通信就像是一支正在行军的军队，整个方阵的前进速度完全取决于走得最慢的那名士兵。”（"Collectives are like an army marching. The entire formation moves at the pace of the slowest soldier."）分布式训练性能完全被集合通信中最慢的节点所卡死。在百万卡集群中，哪怕网络带来一丝一毫的额外延迟开销，都会直接导致价值数十亿美元的算力资源陷入闲置与浪费（stranded compute）。因此，长尾延迟（tail latency）与网络韧性不再仅仅是单纯的技术指标，它们实质上已经演变成了一个关乎真金白银的“纯经济学问题”。这种根本性的范式转变，正是当今超大规模 AI 网络必须直面的挑战。 - **超大规模网络连接**：互联集群跨越机房大厅、数据中心与地域，连接数百万级 GPU 加速器需要构建由数万亿根线缆构成的通信网络。 - **决定性的木桶效应**：AI 集合通信机制严格受限于最慢的通信节点，任何长尾抖动都会直接阻断整个计算流。 - **长尾与韧性的经济代价**：长尾延迟直接导致上百亿美元算力资产搁浅闲置，网络韧性与尾部延迟控制已成为纯粹的算力经济学问题。 --- ### [01:59](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=119s) 传统 RDMA 的困境与“智能下沉至边缘”的范式转移过去数十年间，RDMA 一直是高性能计算（HPC）领域的坚固基石。然而 Meta 在实践中观察到，随着集群节点数突破数万大关，传统的 RDMA 运行模型开始全面崩溃。传统 RDMA 原本设计为与具体工作负载无关，为了让网络跑通，工程师们不得不向网络交换网络（fabric）注入各种提示信息（hints），寄希望于交换机能够自行理解并调整策略。但这种做法的最终结局是无休止的网络拥塞、PFC 暂停帧（pause frames）风暴，以及网络运维人员如同救火队员般在链路热点和带宽严重未充分利用之间疲于奔命。其根本症结在于：交换网络本身根本没有足够的信息，它眼里看到的仅仅是一个个孤立的数据包，只有位于网络边缘的网卡（NIC）才真正洞悉应用的真实通信意图。面对“交换网盲目、边缘知情”的矛盾，Meta 给出的解法是将网络智能全面下沉至网络边缘（NIC）。当网卡在“每路径”（per-path）粒度上掌握了足够的网络状态信息时，它就可以独立管理网络韧性并直接驱动整体通信性能。基于边缘驱动的负载均衡能够维持极高的可用性，而不再依赖底层交换机具备复杂的“智能”。当网络边缘具备了充沛的自主决策能力后，网络团队甚至可以反向解耦并大刀阔斧地革新底层物理拓扑，例如引入极具 TCO（总体拥有成本）优化优势的多平面拓扑（如 FPF 拓扑）。传统网络应用将网络视为一根不可见的黑色粗水管，端点对于管道内部的状况缺乏细粒度感知；而将网络智能移至边缘后，这根粗水管被拆解为暴露给传输层的众多微型管道（tiny pipes），传输层能够做出极细粒度的调度决策，同时对上层应用保持完全透明。 ```mermaid flowchart TD accTitle: 传统网络模型与边缘智能网络模型的对比 accDescr: 对比传统网络依赖交换机智能导致的拥塞，以及MetaRoCE通过边缘网卡感知细粒度路径实现自适应负载均衡。 subgraph Traditional["传统 RDMA 模式（依赖交换网智能）"] direction TB T_App["上层应用"] --> T_Pipe["不透明粗管道 (单一网络流)"] T_Pipe --> T_Switches["盲目交换机 (缺乏应用意图)"] T_Switches --> T_Pain["象流热点 / PFC暂停 / 算力闲置"] end subgraph EdgeIntelligence["MetaRoCE 模式（智能下沉至边缘 NIC）"] direction TB E_App["上层应用 (无感知透明适配)"] --> E_NIC["智能边缘 (NIC 掌握全局意图)"] E_NIC --> E_Pipes["细粒度微型管道 (感知各路径 RTT/拥塞)"] E_Pipes --> E_Commodity["普通商用以太网交换机 (标准 ECMP/ECN)"] E_Commodity --> E_Gain["均衡利用 / 极高可用 / 解耦拓扑创新"] end ``` --- ### [04:16](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=256s) MetaRoCE 核心机制：多路径喷洒与去重排序设计基于“智能边缘”的全新设计哲学，Arvind 正式揭晓了 Meta 专为商用以太网（Commodity Ethernet）上运行的大规模 AI 工作负载从零重构的 RDMA 传输协议——MetaRoCE。MetaRoCE 并没有对传统 RDMA 进行缝缝补补，而是彻底重写，从三个核心维度全面攻克规模化难题：其一，**大规模性能**，通过逐包多路径喷洒（per-packet multipath spraying）与动态负载均衡，达成 90% 的网络有效利用率；其二，**大规模韧性**，通过每路径（per-path）维度的独立可靠性保障，使系统在没有任何交换网层级人工干预的情况下实现高可用；其三，**大规模运维**，确保现有 AI 应用能够零阻力无缝迁移，底层仅需标准商用以太网交换机，完全不需要任何专有的定制交换机特性。在 MetaRoCE 的架构内部，“路径”（Paths）是最基础的核心构建块。传统 RDMA 将每个流静态绑定到单一物理路径上，这必然产生巨大的“大象流”（elephant flows），进而引发严重的链路热点与利用率不足。MetaRoCE 则采取了完全不同的处理逻辑：当应用程序向传输层提交多个消息时，网卡会将这些大消息直接切片分割为众多小数据包，并将其均匀喷洒到整个网络的多条路径上。每个数据包沿不同路径飞行，到达接收端后，直接被组装写入接收端内存缓冲区中，完全不需要额外的重排序缓冲区（reorder buffers）。每条路径都有唯一的熵（entropy）标识，由传输层自主管理且对应用透明。更为关键的是，每条路径独立维护自身的可靠性状态与拥塞状态，从而轻松适应非均匀的往返时延（non-uniform RTT）。负载均衡器会实时依据各路径测得的实际 RTT 和可用窗口动态分配流量。 ```mermaid flowchart LR accTitle: MetaRoCE 消息切片与多路径喷洒机制 accDescr: 展示应用消息被切片为数据包，通过携带不同熵的多路径喷洒到底层网络，并直接按序写入接收端内存。 subgraph Sender["发送端 (Sender NIC)"] Msg["应用提交消息 (Messages)"] --> Chunk["消息切片 (Chunking)"] Chunk --> Spray["多路径喷洒引擎 (带独立熵标签)"] end subgraph Fabric["商用以太网交换网络"] Spray --> P1["路径 1 (独立 RTT / 状态)"] Spray --> P2["路径 2 (独立 RTT / 状态)"] Spray --> P3["路径 3 (独立 RTT / 状态)"] end subgraph Receiver["接收端 (Receiver NIC)"] P1 --> DirectDMA["直接零拷贝写入接收缓冲区"] P2 --> DirectDMA P3 --> DirectDMA DirectDMA --> RecvBuf["接收内存 (无需重排序缓冲区)"] end ``` - **90% 超高网络利用率**：摒弃传统将流静态绑定到单一链路的机制，利用逐包喷洒将以太网链路推向极限利用率。 - **免重排序缓冲区（No Reorder Buffers）**：乱序到达的数据包直接根据内存偏移量放置到最终接收缓冲区，消除硬件端重排序内存开销。 - **故障天然隔离**：路径映射到交换网平面，单个平面或路径发生故障时，流量瞬时无感切换到其他备用平面；每条路径的 RTT 跟踪互不干扰，彻底杜绝故障交叉污染（cross-contamination）。 --- ### [07:14](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=434s) 单路径拥塞控制与消除 QP 膨胀在拥塞控制与可靠性保障方面，MetaRoCE 采用了基于发送端的拥塞控制，但其控制粒度精确收敛到“每路径”（per-path）级别。在传统网络中，一旦遭遇拥塞标记，整条连接都会受到惩罚性降速；而在 MetaRoCE 中，当显式拥塞通知（ECN）触发降速请求时，系统仅调低发生拥塞的那一条特定路径的发送窗口，“一条糟糕的路径绝不会惩罚其他健康的路径”。整个连接的总发送窗口等于所有子路径窗口的总和，这意味着集群可以通过横向增加物理路径来线性扩展网络总容量，而无需过度提高单路径的发包激进程度。此外，接收端通过 ACK 确认包向发送端反向反馈其当前的接收可用能力（readiness/credits），计算接收端所能承载的公平份额，从而在发生多对一入流量突发（Incast）时实现极速收敛。整套高精细度的机制完全构建在最平价的商用以太网之上，Meta 对交换网唯一的要求仅仅是支持标准 ECN 和 ECMP。对于上层应用而言，MetaRoCE 彻底打破了长期困扰 AI 开发者的连接扩展瓶颈。在现行方案中，应用为了跑满高带宽，通常被迫将数据切分后跨多个队列对（Queue Pair, QP）并发传输，而每个 QP 内部都运行着一套相互孤立、缺乏协调的拥塞控制算法，这种机制在大规模集群中极易失控崩溃。MetaRoCE 终结了这种“QP 恶性膨胀”（QP proliferation）的窘境，用单一的多路径连接替代成百上千个独立 QP。在此机制下，独立的应用程序数据流得以高效并发运行，带宽的横向扩展完全交由底层单条连接中协同一致的多路径拥塞控制来驱动。实测对比表明，单条 MetaRoCE 连接的吞吐表现和抗长尾能力，远超传统基于多 QP 拼接的 RDMA 方案，真正展现出细粒度自适应负载均衡的压倒性优势。 --- ### [08:51](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=531s) 开源生态与规范先行：Keysight 带来的一致性测试谈及未来的产业落地，Arvind 坚定地表示，Meta 的目标非常纯粹且明确：“我们希望让 GPU 能够运行在无所不在的普通网络基础设施之上。”Meta 绝不希望行业生态走向碎片化，也坚决反对针对特定工作负载或特定硬件供应商进行深度绑定的专有硬件定制。为此，Meta 正式将 MetaRoCE 传输协议向整个工业界全面开源。该协议对网络交换生态的要求被克制地限定在行业通用的标准功能上——仅仅依赖标准的 ECN 与 ECMP，不存在任何供应商锁死（no vendor lock-in），也没有任何私有专有扩展。然而，开放协议要想真正形成合力，关键在于跨厂商的互操作性（interoperability），必须确保不同芯片和网卡厂商的实现能够丝滑无阻地相互通信。为了建立行业公认的技术基准，Meta 联合全球测试与测量巨头 Keysight（是德科技）共同构建针对 MetaRoCE 的一致性测试套件。Keysight 网络工程师 Kingshuk Mandal 随即登台接棒发言。Kingshuk 深刻地指出，一个全新的协议要想从纸面概念蜕变为成熟的行业生态，其质变的临界点在于“两家不同的芯片厂商在物理线缆上实现逐比特（bit for bit）的完美一致”。在此之前，如果一个协议只能由某一家厂商的实现和自己通信，那它充其量只是一个私有协议；唯有一致性测试（conformance），才是跨不同实现强制执行该契约的唯一手段。 Kingshuk 强调，对于像 MetaRoCE 这样完全从零构筑的全新协议，规范（specification）才是交付给行业的第一款“核心产品”。规范的发布远远早于芯片流片，它构成了所有硬件实现必须对齐的契约。然而白纸黑字的规范文本难免存在歧义，一致性测试的真正价值在于将书面规范转化为一条条可自动验证的确定性断言（testable assertions）。在编写测试用例的过程中，团队能够在极早期暴露并厘清所有规范细节上的模糊地带，而此时修复歧义的成本最为低廉。Kingshuk 一语道破了硬件研发不可逆转的严酷现实：一旦芯片流片封装成硬件实体，任何人都无法将其撤回重来。 - **规范作为首发产品**：在硅片流片前率先定义并发布严格的规范文本，作为全行业软硬件实现的契约准则。 - **一致性驱动生态形成**：唯有两家及以上独立厂商实现线缆上逐比特兼容，新协议才能称之为开放生态，否则只是私有协议。 - **前置消除歧义降低成本**：借助一致性测试将文本转化为可验证测试用例，在芯片物理定型前将歧义扼杀在摇篮阶段，规避流片后不可逆的硬件缺陷。 ### [10:47](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=647s) MetaRoCE 一致性测试套件的严苛门槛与全周期验证连续体硬件修补的代价极其昂贵。Kingshuk Mandal 指出，在芯片设计冻结（silicon freezes）那一刻，规格说明书（spec）中留下的每一个模糊地带，都会直接被固化进流片的物理芯片中。这就是为什么 Meta 和 Keysight 必须从“Day 0”阶段就紧咬协议验证的原因。为此，双方联合打造了一套专门针对 MetaRoCE 1.2 的一致性测试计划（conformance test plan）。该计划具备四大核心特质：首先，它是行业首创，从零针对 MetaRoCE 深度定制；其次，它不仅停留在理论规范上，而是“已被验证”（proven not just specified）——每一项测试用例都切实运行在 MetaRoCE 的软件参考实现上，快速失败的准入门槛完全建立在真实可运行的代码之上；第三，它覆盖协议的全部领域，包括报头完整性、重传逻辑、拥塞控制等全栈规范，绝不只测简单路径；第四，它穷尽测试异常与极端情况，深入检验 NACK（否定应答）、超时、棘手的定时器、重传机制以及各种极易隐藏致命缺陷的恢复分支。 ```mermaid flowchart LR accTitle: MetaRoCE 验证流程连续体 accDescr: 展示从 Day 0 参考实现到流片后物理芯片贯穿不变的一致性测试套件与统一质量门槛 subgraph DesignStages ["跨阶段验证连续体"] A["Day 0: 软件参考实现 (代码修改成本低)"] B["设计成熟期: RTL 硬件仿真 (流片前逻辑验证)"] C["最终阶段: 生产物理芯片 (掩膜/流片后测试)"] end Suite["统一一致性测试套件 (Keysight Anvil 测试仪)"] Suite -. 运行相同测试例.-> A Suite -. 运行相同测试例.-> B Suite -. 运行相同测试例.-> C A ==> B B ==> C ``` 为了确保质量门槛绝不移位，团队构建了一个无缝衔接的验证连续体（continuum）。无论是 Day 0 的软件参考实现，还是随着设计逐步成熟而生成的 RTL 仿真，亦或是最终成型的物理芯片，面对的都是完全同一套协议测试例与同一套严苛的 Pass/Fail 判定标准。Kingshuk 强调，这正是整套体系的核心价值所在：在软件参考实现阶段发现并修复协议缺陷，仅仅是一次代码提交与编辑的成本；可一旦同样的缺陷溜过 RTL、在芯片流片（tape-out）之后才被捕获，就可能导致动辄数百万美元的重新制版（respin）与沉重的时间代价。从架构设计、RTL 编写到硅片交付，测试套件始终如一，彻底杜绝在底层实现方式切换时让 Bug 蒙混过关的漏洞。 - **一致性测试覆盖域**：报头校验完整性、按包多路径重传、精准拥塞控制、NACK 负反馈、复杂超时与故障恢复逻辑。 - **快速失败原则**：每个测试用例均基于真实代码执行而非纸面推演，杜绝歧义。 - **验证跨度**：单套测试用例贯穿软件抽象层、EDA 硬件仿真、流片测试，保证交付全周期质量标准完全零妥协。 --- ### [12:57](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=777s) 案例实测一：混合专家模型（MoE）并发突发下的接收端未就绪（RNR）死锁 Keysight 的测试平台（如 Anvil）之所以至关重要，是因为常规开发与日常 Demo 往往只测试“顺境路径”（happy path），而真正的破坏性缺陷几乎全隐藏在极少被跑到的“逆境路径”（unhappy path）中。Kingshuk 给出了一个源于大语言模型训练的真实测试案例：在混合专家模型（Mixture of Experts, MoE）的典型负载中，庞大的模型参数被分片部署在集群内的众多机器上。在训练的特定计算步，所有节点会突然同时相互通信、爆发密集流量。此时，某个接收端可能会瞬间耗尽接收缓冲区空间——这并不意味着节点损坏，仅仅是它暂时没有准备就绪。在协议层面，接收端会通过发送 RNR（Receiver Not Ready）帧向发送方发起礼貌请求，示意暂时挂起数据发送。Keysight Anvil 工具在此测试中模拟接收端，被测设备（DUT）作为发送端。发送方建立一个 QP（Queue Pair），并使用多个 Tag（标签）通过该 QP 并发发送多条流。当接收端发出 RNR 时，其范围是非常精准且局部的，仅针对出现拥堵的某一个具体 Tag 进行限流，而 QP 内的其他 Tag 必须继续推进传输以保持吞吐。然而，一致性套件在此处抓出了一个隐蔽而致命的 Bug：被测设备在收到该特定 Tag 的 RNR 后确实暂停了其数据发送，但当超时恢复时间过后，该设备却“再也没有恢复传输”，导致该数据流永远陷入死锁挂起状态。诸如 MoE 甚至 All-to-All 等高阶通信原语都会触发这类极端工况，唯有一致性测试仪能够随需按需注入此类故障，让隐蔽缺陷无处遁形。 - **测试拓扑角色**：Keysight Anvil 测试仪作为模拟接收端（Receiver），被测硬件 NIC/芯片作为发送端（Sender）。 - **细粒度多流机制**：单个 QP 承载多个打上独立 Tag 的数据流，RNR 机制被严格限制为外科手术式的局部精准暂停。 - **捕获的协议缺陷**：硬件在针对受影响 Tag 进行特定挂起后，超时到期未能触发恢复状态机，导致工作流在等待中永久冻结。 --- ### [15:06](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=906s) 案例实测二：多路径独立重传定时器（RTO）与单故障路径污染陷阱第二个典型问题则来源于网络本身的复杂性。MetaRoCE 依靠在网卡端将流量喷洒（spray）到多条路径上来实现极高吞吐，因此网卡必须在底层为每条路径独立维持追踪。其中最为关键的一项指标，就是为每个路径单独维护专属的重传定时器（RTO, Retransmission Timeout）。RTO 是发送端判断报文是否丢失并执行快速重传的核心依据：等待时间设得太短，会造成不必要的误重传；等待时间设得太长，又会白白延宕流水线，严重恶化整体作业完成时间（JCT, Job Completion Time）。规格书中极其明确地要求：“每一条路径都必须拥有独立的重传定时器”。 ```mermaid flowchart TD accTitle: MetaRoCE 单路径拥塞污染全局与独立 RTO 对比 accDescr: 对比展示独立路径 RTO 状态与错误共享单个定时器导致全局性能降级到最慢路径的现象 subgraph Independent ["规范设计：独立路径 RTO（测试通过）"] P0["路径 0 (无拥塞)"] --> R0["低 RTO / 快速吞吐"] P1["路径 1 (拥堵)"] --> R1["自适应拉长 RTO / 独立吸收抖动"] end subgraph Shared ["反模式：共享单一 RTO（测试失败）"] SP0["路径 0 (健康)"] -. 共享定时器.-> SR["全局最大 RTO"] SP1["路径 1 (拥堵)"] -. 共享定时器.-> SR SR ==> Degrade["全局 8 端口吞吐降级至单卡拥堵速率 (一条病态路径毒死全局)"] end ``` 然而在硬件设计中，这往往是芯片厂商最容易偷工减料的危险捷径——为整个 QP 仅保留一个统一的重传定时器能够大幅减少状态记录与 SRAM 占用，逻辑确实简单得多。Keysight 与 Meta 在测试中将两台测试节点间的通信人为设置为两条表现迥异的路径：路径 0 作为畅通的超快速路径，路径 1 则故意注入排队时延变成慢速路径，随后实时测量各路径的 RTO 行为。 - **正确规范实现**：健康的网卡为每条路径解耦维护 RTO，路径 0 的定时器迅速回落至底线保持敏捷，路径 1 则独立上调定时器以吸收抖动，彼此互不干扰。 - **共享定时器缺陷**：一旦硬件偷懒共享定时器，发生拥塞的病态路径会瞬间将健康路径的超时时间强行拉长。Kingshuk 形容道："你明明花钱购买了 8 条通道的昂贵带宽，结果全部被迫按照唯一堵塞通道的龟速前行。" - 这种隐患在常规实验室环境下极难暴露，因为实验室网络总是通畅健康、路径绝不分化；可一旦进入万卡规模的生产集群，它就会化作排查耗时数周的“神秘长尾延迟耦合”，彻底废掉多路径架构的高并发价值。 --- ### [17:27](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=1047s) 构建开放生态：彻底告别专有孤岛，迎战百万加速卡互连 Kingshuk 总结道，构建全新传输协议的规范不能等流片之后再去补测，必须在第一天就用高规格手段夯实基础。这为产业界各方带来了三重确定的价值： 1. **对于规范制定者**：规格文本中的每一丝含糊措辞，都在流片前转化为精确的断言逻辑，使规范真正按位（bit-for-bit）严格固化； 2. **对于网络设计人员**：多路径传输能够切实发挥出防御循环、路由隔离以及快速故障转移的威力，全部在严苛的故障注入下得到证实，而非只活在顺风测试里； 3. **对于网络运维人员**：真实训练集群中诸如 All-to-All 突发流量偏斜等引发的诡异硬件异常，在早期环节就被完全阻截。 “一个只有单一实现的传输协议只能被称为私有协议；而只有从 Day 0 就引入开放的一致性测试套件，才能将一项规范催熟为真正的开放工业生态。” Arvind Srinivasan 重新接管讲台进行全篇收尾。他指出，随着集群规模跨越至数百万甚至数十亿加速器节点，网络面临的是处理“数万亿根光纤连接”的超大规模挑战。MetaRoCE 证明了在标准、商用的商用以太网（Commodity Ethernet）上，无需特殊专有交换机、无需被单一厂商锁定，同样可以提供 AI 大模型严苛要求的高吞吐峰值与极高可用性。Meta 正在将 MetaRoCE 全面开源并贡献给产业界，网卡生态伙伴可以根据自身微架构以最大自由度去实现协议规范，因为规范只严格限定线缆上的行为（wire behavior）。不仅如此，Meta 当前的探索早已超越跨节点的 Scale-out，包括机架内的高密 Scale-up 以及跨区域的前端互连网络，都正在经历相同维度的彻底重构。MetaRoCE 只是这场重塑 AI 网络基础架构长征的第一步。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) 可以想象，要将数百万个加速器连接在一起，我们需要数以万亿计的线缆 — Arvind Srinivasan > [00:05](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=5s) 集合通信就像是一支正在行军的军队，整个方阵的前进速度完全取决于走得最慢的那名士兵。 — Arvind Srinivasan > [07:14](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=434s) 一条糟糕的路径绝不会惩罚其他健康的路径 — Arvind Srinivasan > [08:51](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=531s) 我们希望让 GPU 能够运行在无所不在的普通网络基础设施之上。 — Arvind Srinivasan > [08:51](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=531s) 两家不同的芯片厂商在物理线缆上实现逐比特（bit for bit）的完美一致 — Kingshuk Mandal > [15:06](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=906s) 每一条路径都必须拥有独立的重传定时器 — Kingshuk Mandal > [15:06](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=906s) 你明明花钱购买了 8 条通道的昂贵带宽，结果全部被迫按照唯一堵塞通道的龟速前行。 — Kingshuk Mandal > [17:27](https://www.youtube.com/watch?v=SpyWXkWwb1A&t=1047s) 一个只有单一实现的传输协议只能被称为私有协议；而只有从 Day 0 就引入开放的一致性测试套件，才能将一项规范催熟为真正的开放工业生态。 — Kingshuk Mandal ## Mentioned Meta, Keysight, Arvind Srinivasan, Kingshuk Mandal, MetaRoCE, MetaRoCE 1.2, Keysight Anvil, RDMA, RoCE, FPF 拓扑, ECN, ECMP, PFC, QP, MoE, RNR, RTO, RTL, All-to-All ## Video description At Meta's scale — clusters growing to over a million accelerators across regions — the network is in the critical path of every AI training step, and traditional RDMA was never built for it. MetaRoCE is Meta's clean-sheet RDMA transport: a multipath, out-of-order, receiver-driven protocol that treats Ethernet as inherently lossy and pushes all intelligence into the NIC. By shifting intelligence to the edge, the transport decouples from topology — enabling TCO-optimized multiplane FPF topologies with frictionless application adoption on commodity Ethernet. This talk covers MetaRoCE's core mechanisms and Meta's commitment to open standards, including a companion conformance suite developed with Keysight that validates implementations from software reference through hardware emulation to production silicon. Meta is opening MetaRoCE to the industry and inviting broader collaboration to establish it as the ubiquitous RDMA transport for AI-scale networks. Learn more about the @Scale conference here: https://atscaleconference.com/ [Source](https://www.youtube.com/watch?v=SpyWXkWwb1A)

Source quotes

[0:05] Arvind Srinivasan: 可以想象，要将数百万个加速器连接在一起，我们需要数以万亿计的线缆 [0:05] Arvind Srinivasan: 集合通信就像是一支正在行军的军队，整个方阵的前进速度完全取决于走得最慢的那名士兵。 [7:14] Arvind Srinivasan: 一条糟糕的路径绝不会惩罚其他健康的路径 [8:51] Arvind Srinivasan: 我们希望让 GPU 能够运行在无所不在的普通网络基础设施之上。 [8:51] Kingshuk Mandal: 两家不同的芯片厂商在物理线缆上实现逐比特（bit for bit）的完美一致 [15:06] Kingshuk Mandal: 每一条路径都必须拥有独立的重传定时器 [15:06] Kingshuk Mandal: 你明明花钱购买了 8 条通道的昂贵带宽，结果全部被迫按照唯一堵塞通道的龟速前行。 [17:27] Kingshuk Mandal: 一个只有单一实现的传输协议只能被称为私有协议；而只有从 Day 0 就引入开放的一致性测试套件，才能将一项规范催熟为真正的开放工业生态。

Source quotes

[0:05] Arvind Srinivasan: trillions of wires [0:05] Arvind Srinivasan: The modern AI training step is gated entirely by network performance. [2:01] Arvind Srinivasan: only the network edge—the host NIC—has full context. [2:01] Arvind Srinivasan: tiny pipes [4:16] Arvind Srinivasan: Performance at Scale: [4:16] Arvind Srinivasan: Resilience at Scale: [4:16] Arvind Srinivasan: Operations at Scale: [6:27] Arvind Srinivasan: single multipath connection [9:47] Kingshuk Mandal: a brand-new transport protocol becomes a true ecosystem the exact moment two independent vendors agree on the wire format bit-for-bit. [9:47] Kingshuk Mandal: MetaRoCE 1.2 Conformance Test Plan [12:57] Kingshuk Mandal: never resumed transmission [15:06] Kingshuk Mandal: an independent retransmission timer for every single path [18:43] Arvind Srinivasan: trillion wire connectivity across billions of accelerators [18:43] Arvind Srinivasan: every layer of the AI network stack demands the same rethinking, and this is just the beginning.

Original transcript

[0:05] Hello everyone. Today we're going to [0:06] talk about Metaroki Metas RDMA transport [0:09] protocol purpose built for AI networking [0:12] at scale. Hi, my name is Arvin Shinasan. [0:16] I work in metas network infrastructure [0:18] or here's what we plan to cover. AI [0:21] networking scale challenges. Metaria's [0:24] core mechanisms. Kunguk web keyside will [0:27] talk about compliance and [0:28] interoperability and finally we'll [0:30] finish off with our vision for the [0:32] future. So let's get started with this [0:34] topic. At meta we are scaling from [0:36] thousands of GPUs to over million GPUs [0:40] spanning data halls, data centers and [0:42] regions. To keep up with the GPU [0:44] connectivity demand network is also [0:46] going at an exponential rate. One can [0:49] imagine to connect a millions of [0:51] accelerators, we need trillions of [0:53] wires. And when you operate at this [0:55] scale, the interconnect becomes the [0:57] defining challenge. Networking is no [1:00] longer in the background. It determines [1:02] whether the million GPUs can actually [1:04] work together or not. So how is the [1:06] network transforming? Let's see this. [1:09] The AI network is transforming at an [1:11] exponential rate. It is no longer [1:13] auxiliary infrastructure. It is a [1:15] critical part of every training step. To [1:18] keep up with this massive GPU [1:19] connectivity demand, network scale is [1:21] also growing exponentially. As you can [1:23] see in this graph at scale, challenges [1:26] are very real. They require [1:28] fundamentally rethinking of how we build [1:30] a networks. Think of it this way. [1:33] Collectives are like an army marching. [1:35] The entire formation moves at the pace [1:37] of the slowest soldier. Similarly, the [1:39] performance is gated by the slowest node [1:41] in the collective. any extra latency [1:44] that we add that overhead directly [1:46] translates to billions of dollars of [1:48] standard compute. This actually means [1:51] the tail latency and resilience are a [1:53] free economics problem. That's the shift [1:56] we are dealing with when it comes to AI [1:58] networking at scale. So how do we solve [2:01] this? Well, let's start from the [2:04] fundamental observation that we see. [2:06] RDMA has been the workhorse for HPC for [2:09] decades. But what we have observed is as [2:12] a cluster scales from tens of thousands [2:15] of nodes, the traditional model begins [2:17] to break. RDMA was designed to be [2:19] workload agnostic. But in order to make [2:22] it work, we pass hints to the fabric and [2:24] hopes that the switches can figure it [2:26] out what it needs. We ended up with [2:28] persistent congestion, pause and [2:31] constant firefighting of hotspots and [2:33] underutilization. The picture here [2:35] actually shows how in order delivery can [2:38] cause hotspots and underutilization on [2:40] the fabric. Well, the core problem is [2:42] the fabric just doesn't have enough [2:44] information. It only sees packets, only [2:48] the edge and the nick sees the intent. [2:51] So, how do we solve this problem? Can [2:53] you move the intelligence to the edge? [2:55] But what does it look like? When the [2:57] nick has enough information on a per [3:00] path basis, it can manage resilience and [3:03] drive aggregate performance directly. [3:06] Edgdriven load balancing sustains high [3:08] availability without relying on the [3:10] fabric to be smart. That's what the [3:12] middle picture actually is trying to [3:13] show. We have a very well balanced [3:15] network in this case. Now when the edge [3:18] is intelligent enough, we can start [3:20] innovating on the topology itself. that [3:22] is a TCO optimized multiplane fabric [3:25] such as FPF some topology that becomes [3:29] possible when the transport handles the [3:31] complexity that's the fundamental shift [3:33] we are talking about from fabric [3:35] intelligence to the edge intelligence [3:39] now what does an intellent looks like [3:43] traditionally network applications see [3:44] the fabric as a single opaque pipe that [3:47] means the network endpoints do not have [3:50] visibility into what happens inside the [3:52] pipe. It only gets coarse grained [3:55] information. Now when we move from the [3:57] coarse grain view to a fine grain view [4:00] such as exposing many tiny pipes to the [4:03] transport, transport can make fine grain [4:06] decisions. This also means we want to be [4:09] transparent to the applications. This [4:11] shift is what we are calling about an [4:13] intelligent edge. [4:16] Based on this is what we are building [4:17] the metari protocol. Now let's see what [4:20] a protocol actually looks like. Let me [4:22] introduce Metari. [4:24] Metari is a fundamentally redesigned [4:26] RDMA transport for AI workloads running [4:29] on commodity Ethernet. Instead of [4:31] banding traditional RDMA, we built [4:33] something new. Our protocol addresses [4:35] three dimensions of scale. One, [4:38] performance at scale. We target 90% [4:40] network efficiency using per packet [4:43] multipath spraying and load balancing. [4:46] Number two, resilience at scale. [4:48] Resilency matters at the scale. So we [4:51] added PP path reliability to have high [4:54] availability without any fabric level [4:56] intervention for the applications. And [4:59] number three is operations at scale. [5:01] Existing applications need to be adopted [5:04] frictionless and we also want to run out [5:06] commodity Ethernet switches which mean [5:08] no special switch features needed to [5:10] operate. [5:12] Now let's dig deep little bit into the [5:14] protocol what we are trying to do [5:16] specifically parts. Paths are the core [5:19] building block of metari. This enables [5:21] us to do multipaththing and load [5:23] balancing transparently to the [5:25] application. Paths can be considered as [5:27] the foundational piece for metari. Now [5:30] traditional RDMA pins every flow to a [5:33] single path that creates elephant flows [5:35] that results in hotspots and link under [5:38] utilization. Now Metari took a [5:40] completely different approach. The [5:42] picture here actually shows an [5:44] application posting multiple messages to [5:46] the transport. What we do is we take [5:49] these messages and chunk the message [5:51] into multiple packets and spray them [5:53] across the network. Every packet takes a [5:56] different path and is reassembled [5:58] directly into the receive buffer without [6:00] needing any extra reorder buffers. Every [6:03] path is identified by a distinct entropy [6:06] fully managed by the transport and [6:07] transparent to the applications. Every [6:10] path in addition maintains independent [6:12] reliability and a congestion state. This [6:15] helps us scale across non-uniform RTDs. [6:19] This information is fed to the load [6:21] balancer and the load balance selects [6:23] part based on the measured RT and [6:25] available window. [6:27] Now when you talk about what can go [6:30] wrong let's talk about that at a scale [6:32] like this failures would be a natural [6:34] occurrence they are no longer [6:36] exceptional event now in our protocol [6:39] paths maps to the fabric planes on [6:42] failure the traffic shift transm to [6:44] alternate planes because we have enough [6:46] information from the fabric given to the [6:49] transport we do not want applications to [6:51] get involved in this every part tracks [6:54] its own RTD its own condition state [6:56] Therefore, a failure of one path doesn't [6:59] drag others down. There is absolutely no [7:01] crosscontamination between the two. The [7:03] result is resilience is built into the [7:06] transport itself. No fabric level [7:08] recovery, no operator interventions. The [7:10] protocol adapts in real time. Now, let [7:14] me bring congestion control and [7:16] reliability together. Now, Metaroki uses [7:19] a centerbased congestion control, but [7:22] it's on a per path granularity. What [7:24] that means is when EN asks the transport [7:28] to decrease, it only affects the [7:30] congested path. One bad part doesn't [7:32] punish the others. The total window is a [7:35] sum of all the path windows. That means [7:37] we can add paths to give more capacity [7:40] without bringing aggressiveness. In [7:42] addition, the receiver sends radians [7:44] through acknowledgements which computes [7:46] a fair share that the receiver can [7:47] receive and this results in a fast [7:50] convergence of in cost. Now one thing to [7:52] note all of this runs on commodity [7:54] Ethernet switches. We only ask for ECN [7:57] and ECMP order fabric [8:01] at Meta scaling and operating is a key [8:03] priority for us and this is where [8:06] metarogi fundamentally changes the games [8:08] for application. Today application [8:10] scales by chunking messages across [8:12] multiple Q pair each of them run in an [8:15] uncoordinated congestion control. This [8:17] breaks at scale. We replace QP [8:20] proliferation with a single multiath [8:22] connection and by doing this we enable [8:25] independent application streams to work [8:28] and allow pass to scale bandwidth with [8:30] just one coordinated congestion control. [8:33] As you can see in this picture a [8:35] difference between one metari connection [8:37] versus multiple connections out of RDMA [8:40] traditional RDMA we perform much better [8:42] than what exists today. This is the [8:44] power of fine grain load balancing as it [8:47] outperforms traditional RDMA based [8:49] approach. [8:51] Now moving forward, our goal is very [8:53] straightforward. We want to get GPUs [8:55] running on heat resist network [8:57] infrastructure. We do not want [8:59] fragmentation and we do not want [9:01] specialized hardware that needs to be [9:03] customized on a per workload or a per [9:05] vendor basis. Meta is opening Metari [9:08] protocol to the industry. The switch [9:10] ecosystem relies slowly on standard [9:12] features. ECN and ECMP. This means no [9:16] vendor login, no proprietary extensions. [9:19] But now compliance is an interesting [9:21] subject. Openness also means [9:23] interoperability. [9:25] In order for it to interoperate, we want [9:27] different implementations to be able to [9:29] talk to each other in a compliant [9:31] manner. It is very critical that we [9:33] maintain a compliance suite that [9:35] facilitates a communitywide metaroging [9:38] implementation. to that extent I want to [9:41] invite Kingshoot from Kside to talk [9:43] about their offerings and how they plan [9:45] to provide the compliance framework. [9:47] >> Thank you Arvin. Hi I'm Kingship a [9:50] network engineer from Ksite. Arvin [9:53] talked about the transport and here I [9:56] will shift gear and I'll talk about the [9:58] ecosystem. A brand new protocol becomes [10:01] an ecosystem the moment the two vendors [10:04] agrees on the wear bit for bit. Until [10:07] then, one protocol implementation [10:09] talking to itself is a private protocol [10:12] and nothing but confirance enforces that [10:15] agreement across implementation. For [10:17] something brand new like Metari, [10:20] specification is the first product. It [10:22] ships before the silica and it is the [10:25] contract your implementation is bent [10:27] against. [10:29] Now, but a spec is just written written [10:31] words. The confirance is what turns the [10:33] specification into a testable assertion. [10:36] And while writing the test, it exposes [10:38] ambiguity early while it is still cheap [10:41] to fix. Because here is one part that [10:45] none of us can undo. Hardware is [10:47] brutally expensive to patch. Every [10:50] ambiguity left in the spec when the [10:53] silicon freezes ships in the silicon. [10:56] That's why we the protocol from the day [10:58] zero. Now let me show you the test plan [11:02] that we built exactly for for that. [11:07] Now what it is exactly this is the [11:09] conformance test plan for Metari 1.2 and [11:12] four things makes it worth your timing. [11:15] First it is first of it is kind. It is [11:18] developed from the scratch called [11:19] Metari. Second it is proven not just [11:22] specified. Every test is run against a [11:25] metroy reference implementation. So it's [11:27] not just paper, it's real code. The fast [11:30] fail bar is real. Third, it covers the [11:33] entire proto the header integrity, the [11:36] retransmissions, the condition control, [11:38] etc. It covers the whole domain, not [11:40] just the easy one. And then finally, it [11:43] works every part, not just the happy [11:45] one. The knacks, the timeouts, the [11:48] tricky timers, the retransmit, and the [11:50] recovery corners where the tricky bugs [11:53] are hidden. [11:55] Now I will talk about couple of case [11:58] studies but before I go to the go to the [12:00] case studies I want to I want to show [12:02] you one thing to understand how we test [12:05] and it is a continuum the same protocol [12:08] and the same workflow pointed to [12:10] whatever implementation speaks m better [12:14] on day zero it is the reference [12:15] implementation as the design matures it [12:18] is the RTL in liimulation and finally [12:21] the silicon itself the same protocol [12:24] case runs unchanged at every stage with [12:27] one fast spread bar and this is where [12:30] the whole merit is. The quality bar [12:32] never moves. The protocol bug that you [12:34] catch at the reference implementation is [12:37] a code edit and you catch the same bug [12:40] after the tap out. It may cost you a [12:42] risk. So no quality quality compromise [12:45] at the handoff from architecture to RTL [12:48] to silicon greeted by an identical test [12:51] switch. So bugs can't slip in when [12:54] underneath implementation changes. [12:57] So let me show you one real case study [12:59] and the reason this is worth your time [13:02] because our tool like Anwell can drive [13:05] both the happy part and the unhappy [13:07] path. The happy path runs every day [13:09] constantly every demo every bring up [13:12] every day in the lab. So where the bugs [13:15] get caught early and get gets fixed the [13:18] unhy part almost never never runs. [13:21] That's where exactly the nasty bug sits [13:23] and wait for if we now give you a real [13:27] situation. If you train large language [13:30] models, you have probably heard about [13:32] mixture of expert workload. If you this [13:35] in this the model gets split across lots [13:38] of machines and at certain points every [13:41] machine stuck with each other all at [13:43] once and when that happens then some [13:46] receiver may run out of room for [13:48] receiving additional data. They're not [13:51] broken. They're just not ready yet. The [13:54] receiver send backs a polite request to [13:56] the sender. Hold on for some time. I'm [13:59] not ready yet. Top of the picture is [14:02] Ander and this is our trust tool acting [14:04] as the receiver. Bottom is the device [14:07] under test to the sender. The sender [14:09] opens a QP and uses multiple tags to [14:12] send multiple flows through RNR. The [14:15] receiver tells that it is not ready but [14:18] not ready for only one time. The major [14:21] is precise. The device pauses the [14:24] affected tag and the message and it is [14:27] crucial that the QIP keeps making [14:29] progress with all other tags. This [14:32] operation is surgical scope to one tag [14:35] only. What we found that the receiver [14:38] paused the transmissions after the tag [14:41] after for that tag but never resumed [14:44] after the timeout the flow sat there [14:46] forever. [14:48] Now mix of expert is just one one [14:50] trigger the other type like all all to [14:54] all stabil etc can create the same [14:56] situation. With a conformance test tool [14:59] like anvil you can create them anytime [15:02] on demand whenever you want. [15:06] Now the second buff the first one your [15:10] workload cost it and this one the [15:12] network does it to you. Are we sure that [15:15] earlier the metrop is spray traffic [15:18] across many parts at once and it tracks [15:20] them separately. One of the things it [15:23] tracks is per path is retransmission [15:25] timer. In frame terms a retransmission [15:28] timer is used by the sender to make a [15:30] decision if the packet is lost and [15:32] retransmit. If you wet too little then [15:35] you will cut an inappropriate to [15:37] replicates. If your rate transmission [15:39] time is too high then you are going to [15:41] hold the flow for the longer duration [15:43] and that's really impact your job [15:45] completion time. Now speak is very very [15:48] explicit here. Each part must have its [15:51] own retransmission timer. But this is [15:54] the most single most tempting shortcut [15:57] in the pool design. One timer for entire [16:00] QP is less state less memory. It is [16:04] genuinely simpler and it's true. So we [16:08] tested it in the picture. The cast tool [16:10] envelope. [16:15] The two parts between them. We make one [16:17] part and we make them behave differently [16:20] on purpose. Part zero is the fast part [16:22] and the part one is the slow part. We [16:25] rememeasure each part's time out. And [16:27] this is a very common phenomenon in real [16:31] a correct nick keeps the separate RTO [16:33] for each path. Path zero settle downs to [16:36] floor and the part one climbs climbs up [16:38] to absorbs it own table and they move [16:42] independently. That's a pass there is if [16:45] there is one shear timer then the [16:47] congested path drags the head paths time [16:50] out along with it. The moment it it [16:52] happens, you have thrown thrown away [16:55] multipath. A single sick path poisons [16:57] all the good bugs. You have paid for [17:00] eight lens and you are driving all the [17:02] eight at the piece of the one that is [17:04] jet. And here is why this is such a good [17:08] confoundence target in your lab. Every [17:11] path is healthy. So they never diverge. [17:14] It show shows up in production as [17:16] mysterious legency coupling with [17:19] multiple part under load and this is the [17:21] kind of thing that takes a week just to [17:25] see. [17:27] So let me close with this with why this [17:30] is the way forward. This is the way [17:32] forward for all three of you in the [17:34] room. A protocol speak for the brand new [17:36] transport is not something you test at [17:39] the end. It has to be right from the day [17:42] zero because a hardware is brutally [17:44] expensive to pat and the moment the [17:46] silicon freezes every ambiguity you [17:48] leave in the specification ships you the [17:50] d confirming space in three ways at [17:54] once. If you write the spec every [17:57] ambiguity becomes a precise assertion [18:00] where the silicon is still editable and [18:02] the spec hardens bit for bit. If you are [18:05] designing the networks, you get [18:08] multipaths [18:10] that actually performs dressful loops [18:12] with proper separations, failover [18:14] strategies and the recovery all proven [18:17] under the fault injection not just happy [18:20] path. If you run fabrics and the [18:22] failures that only occurs on the real [18:24] workload skew like all to all burst etc [18:28] get caught [18:30] at the end. A transport with one [18:33] implementation is a private protocol. A [18:36] confer page from day zero turns and [18:38] dropped into an ecosystem and it is [18:41] brutally expensive to patch a frozen [18:43] die. So with this I'll hand it back to [18:46] Arvin to talk about what's next and the [18:49] future. Thank you Kshuk. Let me wrap up [18:52] with where we stand and where we are [18:54] headed to. The network scale is [18:56] unprecedented. As several million GPU [18:59] clusters we are solving for trillion [19:01] wire connectivity across billions of [19:02] accelerators that's the problem says we [19:05] are operating in metarogi delivers what [19:07] AI workload demands peak performance and [19:10] high availability on commodity Ethernet [19:13] no special switches no vendor locking [19:16] and we are opening this to the industry [19:18] the nick ecosystem is invited to [19:20] implement metari with full flexibility [19:22] in their approach the spec defines the [19:25] wire behavior the vendors can choose how [19:27] to build it. But we are not done here. [19:30] The challenges go beyond scale out. [19:32] Scale up within the rack, front end [19:35] across the region. We are actively [19:37] working to solve these as well. Every [19:40] layer of the AI network sty demands the [19:42] same rethinking and this is just the [19:44] beginning. Stay tuned for more and thank [19:47] you.