Meta Opens MetaRoCE Protocol to Cut Million-GPU Network Costs

Live from SCCC: MetaRoCE: Meta’s RDMA Transport | Arvind Srinivasan and Kingshuk Mandal — @Scale

@Scale

Meta is releasing its MetaRoCE RDMA transport to open standards to enable multiplane commodity Ethernet clusters and eliminate specialized switch hardware for million-GPU AI networks.

AI-assisted summary of Live from SCCC: MetaRoCE: Meta’s RDMA Transport | Arvind Srinivasan and Kingshuk Mandal — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published by @Scale on YouTube on August 31, 2026, this video features Meta's Arvind Srinivasan and Keysight's Kingshuk Mandal presenting live from the Santa Clara Convention Center. The page description outlines how MetaRoCE was created as a clean-sheet, multipath, receiver-driven RDMA transport to solve the massive networking bottlenecks of AI clusters scaling to over a million GPUs, while highlighting Meta's open-source commitment and Keysight's companion conformance suite.

## TL;DR MetaRoCE decouples transport intelligence from switch topology by pushing congestion control, dynamic packet spraying, and receiver-driven incast management directly onto endpoint NICs. Designed to achieve up to 90% fabric efficiency on commodity Ethernet without switch-level customization, it eliminates traditional Queue Pair (QP) proliferation while maintaining per-path RTT tracking and isolation. Keysight's KAI Conformance suite complements the protocol by executing day-zero validation across software, RTL emulation, and production silicon to eliminate hardware bugs like Receiver Not Ready (RNR) freezes and shared RTO shortcuts before tap-out. Ultimately, Meta is opening MetaRoCE to the broader hardware ecosystem to establish a multi-vendor, standardized transport for next-generation scale-out, scale-up, and regional AI networks.

## Atmosphere & style The presentation carries an authoritative, pragmatic, and highly technical tone tailored for an audience of network architects and infrastructure engineers. The narrative flows smoothly between Arvind Srinivasan's visionary perspective on macro-economic AI bottlenecks and Kingshuk Mandal's sharp, hands-on focus on silicon validation realities. The session feels collaborative and forward-looking, balancing high-stakes engineering challenges with a clear passion for open-source industry standards.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=0s) Meta introduced MetaRoCE, a clean-sheet RDMA transport protocol built to solve severe networking bottlenecks as AI clusters scale to over one million GPUs across regions. - [00:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=0s) Collective operations in AI workloads operate like an army marching at the speed of the slowest worker, turning network tail latency and packet drops into macroeconomic bottlenecks that risk stranding billions of dollars in compute. - [02:13](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=133s) Traditional RDMA relies on switch fabric signals (e.g., ECN) for congestion control, but switches only see raw packets whereas endpoints see workload intent. - [02:13](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=133s) MetaRoCE shifts all intelligence, load balancing, and failure management to endpoint NICs, allowing network switches to remain simple, commodity Ethernet units. - [04:09](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=249s) MetaRoCE exposes large network pipes as sets of multiple sub-paths, splitting application messages into individual packets sprayed out-of-order across distinct paths. - [04:09](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=249s) Each path maintains its own distinct entropy, congestion control state, and RTT tracking, isolating link drops or timeouts to single paths without dragging down overall performance. - [08:33](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=513s) Sender congestion control operates per-path while receiver NICs calculate fair-share bandwidth hints during incast patterns, preventing heavy all-to-all incast congestion. - [08:33](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=513s) MetaRoCE eliminates Queue Pair (QP) proliferation by handling high bandwidth through internal multi-pathing under single coordinated QPs rather than running hundreds of uncoordinated QPs. - [11:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=660s) To convert written specifications into interoperable hardware, Meta partnered with Keysight to build the Keysight AI Conformance (KAI Conformance) test suite for MetaRoCE v1.0. - [12:54](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=774s) Keysight's testing continuum executes identical conformance assertions across software reference models, RTL emulation platforms, and production silicon to catch logic bugs before expensive tap-outs. - [12:54](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=774s) A real-world case study on Mixture of Experts (MoE) workloads showed how automated conformance testing identified a DUT NIC bug where a flow failed to resume transmission after a Receiver Not Ready (RNR) timeout. - [17:10](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1030s) Sharing a single RTO timer across a Queue Pair is a tempting implementation shortcut that invalidates multipathing; per-path RTO isolation is strictly required so one congested path does not poison healthy ones. - [19:34](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1174s) Day-zero protocol conformance prevents hardware bugs from freezing into physical silicon and transforms vendor-specific private implementations into an open multi-vendor ecosystem. - [20:50](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1250s) Meta is open-sourcing MetaRoCE to the industry and expanding its transport principles beyond scale-out networks into intra-rack scale-up domains and cross-region networks.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=0s) Rethinking RDMA for the AI Era: Scaling to Millions of GPUs

Arvind Srinivasan from Meta opens the presentation live at the Santa Clara Convention Center by introducing **MetaRoCE**, Meta's clean-sheet RDMA transport protocol purpose-built for AI networking at scale. Setting the stage after an introduction by Ju, Arvind outlines the talk's roadmap: breaking down the scaling challenges faced in modern AI clusters, exploring MetaRoCE’s core mechanisms, demonstrating compliance and interoperability with Keysight's Kingshuk Mandal, and presenting Meta’s vision for an open industry ecosystem.

As Meta scales its infrastructure from thousands of GPUs to clusters exceeding one million accelerators spanning data halls, data centers, and entire regions, the underlying network infrastructure must expand at an identical exponential pace. Connecting a million accelerators translates to needing "trillions of wires," shifting the network from an auxiliary background component into the primary bottleneck that determines whether a million GPUs can actually function together.

In AI workloads, network performance is directly tied to job execution speed. Arvind highlights a critical physical reality: collective operations act "like an army marching," where the entire formation moves strictly at the pace of the slowest soldier. Any microsecond of tail latency or packet drop directly translates to billions of dollars in stranded compute time. Resiliency and tail latency cease to be purely technical networking problems—they become macro-economic imperatives for AI infrastructure.

- **Scale Target:** Expanding beyond 1,000,000 GPUs across data halls, data centers, and multi-region deployments. - **Interconnect Demand:** Exponential 10x growth in connected accelerators over short time horizons. - **Economic Impact:** Gated performance on collective operations directly risks stranding billions of dollars of GPU compute.

---

[02:13](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=133s) Shifting Intelligence from Fabric to Edge

Taking a historical step back, traditional RDMA (RoCE) has been the workhorse of high-performance computing (HPC) for decades. However, as cluster sizes scaled into tens of thousands of nodes, traditional RDMA transport models began to break down, creating persistent hotspots, burst storms, and severe fabric underutilization. Up until now, the industry worked around these limits by passing workload "hints" down into the network switches, hoping the switch fabric could resolve congestion. This resulted in constant operational firefighting and fragile tuning.

Arvind pinpoints the fundamental structural flaw of traditional architectures: *the network fabric only sees raw packets, whereas only the NIC and endpoints see the actual workload intent*. By moving intelligence out of the fabric and pushing it entirely to the endpoint NICs, the edge can perform intelligent load balancing directly across the network. If the NIC possesses granular per-path visibility, it can drive aggregate performance, manage resiliency, and handle load balancing without requiring the underlying network switches to be smart.

This edge-driven approach unlocks cost-optimized network topologies, such as multi-plane, free-pass-fabric (FPF) architectures. By decoupling transport intelligence from fabric topology, Meta shifts the burden of congestion and failure management entirely to smart edge nodes, allowing the network fabric to remain simple, commodity, and scalable.

- **Traditional Fallback:** Over-reliance on switch fabric intelligence via coarse-grained signals (e.g., ECN, CNP) leads to persistent congestion and burst storms. - **Architectural Shift:** Pushing flow intent and load balancing to the endpoint NIC decouples transport logic from switch topology. - **Topology Benefit:** Enables cost-effective multi-plane Ethernet topologies without requiring specialized switch hardware.

---

[04:09](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=249s) MetaRoCE Core Architecture: Multipath and Per-Path Intelligence

Traditionally, applications view the network as a single, opaque, "big fat pipe" with zero visibility into internal network dynamics. MetaRoCE replaces this model by exposing the large network fabric to the transport layer as a set of multiple "tiny pipes," enabling fine-grained, intelligent routing decisions that remain completely transparent to the user application. MetaRoCE is designed around three primary dimensions of scale:

1. **Performance at Scale:** Achieves up to 90% fabric efficiency using per-packet load balancing and dynamic multipathing. 2. **Resiliency:** Provides per-path reliability and fast fault recovery without requiring any switch-level intervention. 3. **Operations at Scale:** Built on standard commodity Ethernet with full Verbs API compatibility, ensuring frictionless adoption for existing AI applications without needing specialized switches.

At the heart of MetaRoCE is the abstraction of a **path**. Rather than pinning a Queue Pair (QP) to a single network flow—which causes massive elephant flows and fabric hot-spotting—MetaRoCE splits application messages into individual packets and sprays them across multiple independent paths. At the receiving end, packets are reassembled directly into the receiver's memory buffer regardless of the underlying RDMA operation semantics.

Each path is assigned a distinct entropy value managed automatically by the transport. Crucially, every individual path maintains its own independent congestion control state and round-trip time (RTT) tracking. This design allows MetaRoCE to scale naturally across non-uniform RTT topologies. Load balancing decisions adapt dynamically based on real-time RTT measurements and available window sizes, allowing the NIC to bypass network hotspots. If a path times out or experiences link flaps, MetaRoCE automatically rotates path entropies to find an unmanaged route, isolating failures so one degraded path never drags down the performance of others.

---

[08:33](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=513s) Sender Congestion Control, Receiver Incast Control, and Avoiding QP Proliferation

MetaRoCE features a unique dual-ended congestion control scheme designed to handle heavy AI workloads without switch customization:

- **Sender-Driven Per-Path Congestion Control:** Congestion response (such as Additive Increase/Multiplicative Decrease) is evaluated strictly on a per-path basis. If an Explicit Congestion Notification (ECN) or loss occurs on one specific path, only that single path throttles its rate. The total window available to the Queue Pair is the aggregate sum of all active path windows. Adding more paths increases available bandwidth capacity without increasing transport aggressiveness. - **Receiver-Driven Rate Hints for Incast:** To prevent many-to-one incast bottlenecks (common in all-to-all collective patterns), the receiving NIC calculates fair-share bandwidth limits based on the number of active incoming senders. The receiver transmits rate hints back to senders inside ACK packets, allowing endpoints to manage incast bandwidth distribution dynamically without relying on switch buffer management. - **Elimination of QP Proliferation:** Traditional implementations attempt to scale bandwidth by multiplying the number of Queue Pairs (QPs) per host. At scale, hundreds of uncoordinated QPs run competing congestion algorithms, stepping on each other and degrading aggregate throughput. MetaRoCE solves this by maintaining a low QP count and scaling bandwidth through internal multi-pathing managed by a single coordinated transport engine.

Meta's core goal is to run high-density GPU workloads across heterogeneous, commodity Ethernet networks using standard ECN and ECMP mechanisms—avoiding proprietary vendor lock-in or specialized hardware extensions.

- **Efficiency Metric:** Targets 90% network utilization via packet spraying and per-path dynamic routing. - **Commodity Support:** Requires only basic ECN (Explicit Congestion Notification) and ECMP (Equal-Cost Multi-Path) from underlying switches. - **QP Management:** Replaces uncoordinated multi-QP bandwidth scaling with coordinated multi-pathing under single QPs.

---

[11:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=660s) Open Standards and Keysight Compliance Framework

To ensure MetaRoCE becomes a ubiquitous industry transport rather than an isolated proprietary technology, Meta is opening the protocol specification to the broader AI ecosystem. Arvind emphasizes that "interoperability is just a spec" until validated by a robust compliance suite. To turn written specifications into interoperable hardware, Meta partnered with Keysight Technologies.

Kingshuk Mandal from Keysight takes the stage to explain how protocol compliance bridges the gap between written standards and real-world silicon:

> "A protocol specification becomes an interoperable system when multiple vendors agree on the wire. Until then, one implementation talking to itself is a private implementation... Specification is written words, and conformance converts these written words into testable assertions."

Kingshuk underscores the harsh economic reality of hardware design: **silicon is brutally expensive to patch**. Any ambiguity left in a transport specification that slips past tap-out gets permanently frozen into the physical chip, costing millions in respin expenses. To avoid post-mortems, Keysight developed the **Keysight AI Conformance (KAI Conformance)** test suite for MetaRoCE v1.0, executing test plans from day one against reference software models, RTL emulation platforms, and final production silicon.

---

[12:54](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=774s) Testing in Continuum and Real-World Case Study: Receiver Not Ready (RNR)

Keysight’s testing framework operates as a continuous pipeline. The exact same protocol test cases, workflow, and pass/fail quality bars are applied across every stage of the hardware lifecycle:

``` [ Software Reference Implementation ] ➔ [ RTL Emulation ] ➔ [ Production Silicon ] ```

Because the pass/fail bar never moves, bugs discovered during architecture prototyping are resolved via simple code updates rather than multi-million-dollar post-tapout silicon respins. The test suite aggressively probes corner-case paths, including negative acknowledgements (NACKs), timer roll-overs, and recovery states where production bugs typically hide.

Kingshuk walks through a real-world case study involving **Mixture of Experts (MoE)** workloads. In MoE model training, thousands of GPUs engage in massive all-to-all communication phases. Occasionally, a target machine temporarily runs out of memory buffer space to accept incoming transfers. This non-error condition is signaled via an **RNR (Receiver Not Ready)** frame, where the receiver politely requests the sender to pause and retry after a backoff duration.

In MetaRoCE's multipath architecture, a single Queue Pair may handle multiple active flows using distinct packet tags (entropies). The MetaRoCE specification explicitly mandates that upon receiving an RNR for a specific tag, the sender NIC **must pause only the impacted tag and its associated message**, while allowing all other tags and flows on that same QP to continue transmitting normally.

During early conformance testing on a Device Under Test (DUT) NIC acting as a sender: - The Keysight test tool emulated an RNR response targeting a single specific tag. - **The Bug:** The DUT NIC correctly paused the impacted tag, but due to a logic flaw, **forgot to wake up after the timeout expired**, leaving that specific flow permanently frozen. - **The Takeaway:** Conformance tools allow vendors to deterministically trigger and validate subtle recovery bugs on demand before freezing RTL design files.

- **Conformance Suite:** Keysight AI Conformance (KAI Conformance) suite built specifically for MetaRoCE 1.0. - **Coverage Scope:** Full domain coverage including hardware header integrity, per-path congestion, reliability timeouts, NACK handling, and edge recovery. - **Target Workloads:** Validates dynamic behavior under heavy MoE (Mixture of Experts) and all-to-all traffic stress patterns.

[17:10](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1030s) Validating Independent Path RTO: Testing the Single Most Tempting Shortcut

Kingshuk Mandal demonstrates how Keysight tests device-under-test (DUT) path isolation in MetaRoCE. Because MetaRoCE sprays packets across multiple paths and tracks each path individually, maintaining an independent Retransmission Timeout (RTO) for every single path is critical. In simple terms, RTO is the timer used by the sender to determine when a packet has been dropped so it can retransmit. Managing RTO involves a delicate balance: setting it too small floods the network with redundant duplicates and collapses goodput, while setting it extra large causes flows to stall excessively, degrading Job Completion Time (JCT).

The MetaRoCE specification explicitly mandates that each path maintain its own independent RTO. However, Kingshuk points out that adapting a single, shared RTO for the entire Queue Pair (QP) is "the single most tempting shortcut in the whole design." While sharing a single RTO requires much less memory, fewer lines of code, and is genuinely simpler to implement, it is fundamentally wrong. To catch this flaw, Keysight configures a test topology featuring multiple paths between the DUT and the emulation tool: * **Path 0:** Deliberately maintained as a fast, healthy, high-speed path expected to achieve a very low RTO floor. * **Path 1:** Intentionally injected with heavy congestion to force a high RTO.

In a correct NIC implementation, Path 0 settles down to its fast baseline, while Path 1 scales up its RTO to absorb local delays—yielding a pass. But if an implementation takes the shortcut of a shared QP RTO, the single congested path drags down all healthy ones. "One unhealthy path is going to poison the entire good set," Kingshuk warns. "The moment that happened, you have already thrown away multipath. You have paid for eight individual lanes, but you are driving all of them at the speed of the one that is jammed." While every path stays healthy in a sanitized lab environment, this flaw manifests in real-world deployments as "a mysterious latency coupling multiple paths under load"—a nightmare bug that can take weeks to debug.

[19:34](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1174s) Why Day-Zero Conformance Turns a Draft into a Vibrant Ecosystem

Kingshuk shifts focus to the strategic necessity of validating transport protocol specifications early in the development lifecycle. A brand-new transport technology like MetaRoCE cannot be treated as something tested at the tail end of hardware development. Ambiguity left in the specification while silicon freezes gets permanently etched into the die, and "hardware is brutally expensive to patch."

Day-zero conformance pays off for all three major industry stakeholders simultaneously: * **Protocol & ASIC Designers:** Catch specification ambiguities early while silicon designs are still editable, hardening the protocol bit for bit. * **Network Architects:** Guarantee a true multipath network that handles degradation gracefully with proper isolation and recovery, verified under stress and fault injection rather than just the happy path. * **Fabric Operators:** Uncover subtle edge-case bugs that typically surface only under live production workloads long before deployment.

Kingshuk emphasizes that a protocol implemented by a single vendor remains nothing more than a private implementation. Rigorous, open conformance from day zero turns a specification draft into a vibrant, multi-vendor ecosystem.

[20:50](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1250s) Expanding MetaRoCE: Million-GPU Fabrics and Open Collaboration

Arvind Srinivasan returns to conclude the presentation with Meta's vision for future AI scale. Tomorrow's network demands are unprecedented, as Meta builds million-GPU clusters designed to solve "trillion-wire connectivity across millions of ASICs." MetaRoCE achieves peak AI workload performance and high availability on commodity Ethernet without requiring specialized switches or locking operators into proprietary hardware ecosystems.

Meta is opening MetaRoCE to the broader industry and empowering the NIC vendor ecosystem to build compliant silicon. Later in the conference schedule, representatives from AMD (Sep and Bali) present their own native implementation of MetaRoCE on AMD hardware platforms. Arvind concludes by stressing that Meta's networking team is far from done: * **Current Expansion Focus:** Extending MetaRoCE principles beyond scale-out fabrics to tackle intra-rack scale-up domains as well as cross-region front-end networks. * **Call to Action:** Every layer of the AI network stack demands a fundamental rethinking, inviting industry partners to collaborate on open transport standards.

---

<ElicitationsGroup message="Explore related aspects of MetaRoCE architecture and test methodologies:"> <Elicitation label="Deep dive into MetaRoCE multipath & packet-spraying design" query="Can you explain the detailed packet-spraying mechanisms and out-of-order handling used by MetaRoCE?"/> <Elicitation label="Overview of Keysight's network emulation suite" query="How does Keysight's validation and emulation suite test RDMA transport protocols under lossy network conditions?"/> </ElicitationsGroup>

## Notable quotes > [00:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=0s) like an army marching — Arvind Srinivasan

> [02:13](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=133s) the network fabric only sees raw packets, whereas only the NIC and endpoints see the actual workload intent — Arvind Srinivasan

> [11:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=660s) A protocol specification becomes an interoperable system when multiple vendors agree on the wire. Until then, one implementation talking to itself is a private implementation... Specification is written words, and conformance converts these written words into testable assertions. — Kingshuk Mandal

> [11:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=660s) silicon is brutally expensive to patch — Kingshuk Mandal

> [12:54](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=774s) must pause only the impacted tag and its associated message

> [12:54](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=774s) forgot to wake up after the timeout expired

> [17:10](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1030s) the single most tempting shortcut in the whole design — Kingshuk Mandal

> [17:10](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1030s) One unhealthy path is going to poison the entire good set — Kingshuk Mandal

> [17:10](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1030s) You have paid for eight individual lanes, but you are driving all of them at the speed of the one that is jammed. — Kingshuk Mandal

> [17:10](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1030s) a mysterious latency coupling multiple paths under load — Kingshuk Mandal

> [19:34](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1174s) hardware is brutally expensive to patch — Kingshuk Mandal

> [20:50](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1250s) trillion-wire connectivity across millions of ASICs — Arvind Srinivasan

## Mentioned Arvind Srinivasan, Kingshuk Mandal, Ju, Sep, Bali, Meta, Keysight Technologies, AMD, @Scale, Santa Clara Convention Center, MetaRoCE, RDMA, RoCE, Ethernet, Keysight AI Conformance (KAI Conformance), Verbs API

[Source](https://www.youtube.com/watch?v=QX8C-8uxXKc)

中文简报

Meta开放MetaRoCE协议以降低百万GPU网络成本 Meta正将其MetaRoCE RDMA传输协议推向开放标准，以支持多平面商用以太网集群，并消除百万GPU级AI网络所需的专用交换机硬件。 MetaRoCE通过将拥塞控制、动态报文喷洒和接收端驱动的incast管理直接移至终端NIC，将传输层智能与交换机拓扑解耦。该协议旨在无需交换机层面定制的前提下，在商用以太网上实现高达90%的互联网络效率，消除传统队列对（QP）激增问题，同时维持每条路径的RTT跟踪与隔离。Keysight的KAI一致性套件进一步补充了该协议，通过跨软件、RTL仿真和量产硅片执行零日验证，在流片前消除接收端未就绪（RNR）卡死和共享RTO捷径等硬件缺陷。最终，Meta正向更广泛的硬件生态系统开放MetaRoCE，以构建面向下一代scale-out、scale-up和区域AI网络的多厂商标准化传输体系。 Meta 发布专为百万级加速卡 AI 集群全新设计的 RDMA 传输协议 MetaRoCE，旨在消除集合通信中的网络长尾延迟与丢包，避免数十亿美元的 GPU 算力闲置。 (0:00) MetaRoCE 将网络智能与负载均衡从交换机转移至端点 NIC，在通用以太网上将消息拆分为独立数据包跨多路径无序散列发送，实现高达 90% 的网络利用率并支持解耦的多平面拓扑。 (4:09) 协议引入每路径独立的发送端拥塞控制与接收端驱动的 Incast 速率提示，并通过单一 QP 内部的多路径调度规避了传统架构中队列对（QP）剧增引发的性能退化。 (8:33) Meta 携手是德科技（Keysight）推出贯穿软件、RTL 仿真及量产芯片的 KAI 一致性测试套件，严格验证独立路径 RTO 隔离与 RNR 恢复等边界场景，以推动多厂商开放标准的落地与生态繁荣。 (11:00)

Reading summary

Meta has introduced MetaRoCE, a clean-sheet RDMA transport designed to scale AI clusters to over one million GPUs by shifting transport intelligence from switch fabrics to endpoint NICs. To establish an open multi-vendor ecosystem, Meta partnered with Keysight Technologies to implement hardware conformance testing from day zero. The AI Networking Bottleneck and Edge Intelligence (0:00) As AI clusters expand toward one million accelerators, Arvind Srinivasan from Meta argues that the network becomes the primary computational bottleneck. In distributed AI training, collective operations progress at the pace of the slowest node, meaning microsecond tail latencies or packet drops risk stranding billions of dollars in GPU compute time. Traditional RoCE architectures rely on switch fabrics to resolve congestion using coarse signals like ECN, leading to fabric underutilization and hotspots. Meta addresses this structural flaw by moving flow management and load-balancing intelligence entirely to endpoint NICs, which have full visibility into application intent. This shift allows Meta to deploy standard, commodity Ethernet topologies without requiring specialized, intelligent switch hardware. Multipath Architecture and Dynamic Congestion Control (4:09) MetaRoCE replaces the conventional single-flow Queue Pair (QP) abstraction by splitting messages into individual packets and spraying them across multiple independent paths. Each path maintains its own entropy value, round-trip time (RTT) tracking, and congestion control state. This design targets up to 90% network utilization, isolates link degradation to single paths without stalling an entire QP, and avoids the performance penalties caused by QP proliferation. The protocol implements dual-ended congestion control: senders adjust rates independently on a per-path basis when congestion occurs, while receivers compute fair-share bandwidth allocations during many-to-one incast events and return rate hints via ACK packets. Because reassembly is handled in endpoint memory, the protocol preserves Verbs API compatibility while relying only on standard switch features like Equal-Cost Multi-Path (ECMP) and Explicit Congestion Notification (ECN). Hardware Conformance Across the Silicon Lifecycle (11:00) To prevent MetaRoCE from remaining a single-vendor proprietary transport, Meta teamed with Keysight to produce the Keysight AI Conformance (KAI Conformance) suite for MetaRoCE v1.0. Keysight's Kingshuk Mandal emphasizes that transport bugs frozen past silicon tap-out incur multi-million-dollar respin penalties. The test suite applies identical validation checks continuously across reference software, RTL emulation, and production silicon. Mandal demonstrated this approach through an emulated Mixture of Experts (MoE) workload using Receiver Not Ready (RNR) flow pauses. While testing a device under test (DUT), the tool revealed a logic error where the sender paused an impacted traffic tag as mandated by the specification, but permanently hung because it failed to wake up after the backoff timer expired. Testing Path Isolation Against Implementation Shortcuts (17:10) A critical verification focus for Keysight is preventing the common shortcut of sharing a single Retransmission Timeout (RTO) across an entire Queue Pair. While a unified RTO simplifies implementation and saves memory, Mandal warns it causes a single degraded path to increase the RTO for all healthy paths, effectively collapsing multipath throughput to the speed of the slowest lane. Keysight detects this failure by configuring dual-path test topologies with one clear, fast path and one artificially congested path. Conforming silicon must decouple these timers so that the healthy path preserves its low latency floor while the congested path scales its RTO independently, preventing cross-path latency contamination. Meta plans to extend MetaRoCE architectures into intra-rack scale-up domains and regional front-end fabrics, with third-party vendors such as AMD developing native hardware implementations. The overarching challenge remains whether the broader industry will adopt these open transport and testing standards across heterogeneous hardware.

精炼阅读摘要

Meta 联合 Keysight 推出面向超大规模 AI 集群的自研 RDMA 传输协议 MetaRoCE，旨在通过将网络智能从交换机 Fabric 迁移至终端网卡，在通用以太网上解决百万级 GPU 互联下的尾部延迟与拥塞难题。 百万级 GPU 扩展瓶颈与智能边缘化 (0:00) 随着 Meta 将 AI 算力集群扩展至跨数据中心甚至跨地域的百万 GPU 规模，互联线缆需求达到数万亿条，底层网络已从辅助设施转变为决定整体算力利用率的直接瓶颈。在 AI 集群中，集合通信操作类似步调整齐的军队行进，整体进度受制于最慢的节点。微秒级的尾部延迟或丢包都可能导致巨额算力资本闲置，因此网络韧性已成为关键的宏观经济指标。 Arvind Srinivasan 指出，传统 RoCE 依赖交换机通过粗粒度信号（如 ECN 和 CNP）感知拥塞并做出调整，但交换机仅能看见原始数据包，只有终端网卡（NIC）才掌握完整的工作负载意图。MetaRoCE 改变了这一架构，将流意图、动态负载均衡以及拥塞和故障恢复逻辑全部推向终端边缘网卡。这种解耦设计使得交换机可以保持简单的通用商用以太网硬件，支持 Free-Pass-Fabric（FPF）等多平面拓扑，避免了依赖复杂私有交换机带来的高昂成本与运维陷阱。 多路径数据包喷洒与独立路径拥塞控制 (4:09) 传统传输协议将 Queue Pair（QP）绑定到单一网络流上，容易在 AI 流量突发时引发严重的大流（elephant flow）热点。MetaRoCE 摒弃了将网络视为单一粗管道的黑盒抽象，将数据包喷洒（packet spraying）到多条独立路径上，并在接收端直接重组至内存缓冲区，实现约 90% 的网络利用率，同时对上层应用保持 Verbs API 兼容。 在 MetaRoCE 中，传输引擎为每条路径分配独立的熵值，并各自维护独立的拥塞控制状态与往返时间（RTT）跟踪。发送端在单条路径遭遇丢包或拥塞标记时，仅对该路径实施降速，QP 的可用总窗口则是所有健康路径窗口的聚合；当某条路径发生超时或链路抖动时，协议自动轮换路径熵值进行隔离。此外，针对全对全（all-to-all）场景下的多对一 Incast 瓶颈，接收端网卡会计算各发送方的公平均衡速率，并通过 ACK 数据包返回速率提示（rate hints），从接收端协同抑制拥塞，彻底避免了传统架构通过泛滥 QP 争夺带宽而引发的拥塞雪崩。 硬件流片风险与一致性测试流水线 (11:00) 为了将书面规范转化为行业开放标准并防止生态分裂，Keysight 与 Meta 合作开发了面向 MetaRoCE 1.0 的 Keysight AI Conformance（KAI Conformance）测试套件。Keysight 的 Kingshuk Mandal 强调，硬件芯片流片（tap-out）成本极其昂贵，规范中遗留的任何模糊定义一旦固化至物理芯片，将带来数百万美元的重新流片（respin）损失；若缺乏多厂商一致性验证，单个厂商的实现充其量只是私有协议。 Keysight 建立了贯穿软件参考模型、RTL 硬件仿真平台直到量产芯片的全生命周期连续测试流水线，并在各个阶段维持严苛且统一的测试准则。该套件重点覆盖负向应答（NACK）、计时器溢出及边界恢复等边缘情况。例如在 Mixture of Experts（MoE）全对全突发场景测试中，接收端因缓冲区不足发送 Receiver Not Ready（RNR）帧时，规范要求发送端仅暂停受影响标签的流，但被测网卡在早期仿真中出现了超时后无法唤醒该流的逻辑缺陷，此缺陷正是通过一致性断言测试在前端设计阶段被及时拦截。 独立路径 RTO 验证与反投机设计 (17:10) 在多路径传输设计中，每个路径维持独立的重传超时计时器（RTO）对性能隔离至关重要。RTO 过短会导致大量冗余报文淹没链路导致有效吞吐崩溃，RTO 过长则会导致作业完成时间（JCT）出现漫长停滞。尽管 MetaRoCE 严格规定每个路径必须独立维护 RTO，但开发团队极易在硬件实现时选择投机取巧，让整个 QP 共享一个全局 RTO，以此节省片上内存和代码逻辑复杂度。 Keysight 设计了针对性的双路径对照测试：对 Path 0 维持健康低时延路径，同时对 Path 1 注入重度网络拥塞。合规的网卡实现能够让 Path 0 维持低 RTO 基线，而 Path 1 独立放宽 RTO；如果实现采用了共享全局 RTO 的妥协方案，单条拥塞路径就会拖垮所有健康路径的计时器基准，导致多路径特性名存实亡。这一测试杜绝了在实验室理想环境下掩盖、而在实际生产高压下才会暴露的跨路径时延耦合隐患。 Meta 计划将 MetaRoCE 规范全面向开源生态开放，除 AMD 展示了其硬件平台的原生实现外，该协议的核心机制正进一步向机架内纵向扩展（scale-up）与跨区域前端网络推进。不过，在更广泛的异构多厂商芯片与大规模复杂拓扑中，能否在不牺牲性能的前提下完全消除实现偏差，仍有赖于全行业对早期一致性测试标准的长久遵守。

Detailed notes — zh

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布，记录了在圣克拉拉会议中心举办的现场演讲。视频中，来自 Meta 的 Arvind Srinivasan 与来自 Keysight 的 Kingshuk Mandal 共同介绍了 Meta 专为百万级 GPU 超大规模 AI 集群打造的自研 RDMA 传输协议 MetaRoCE，并展示了其开源规范与联合开发的一致性验证套件。 ## TL;DR 随着 AI 集群扩展至百万加速卡级别，传统 RDMA 协议在长尾延迟和网络利用率上面临严重瓶颈。MetaRoCE 彻底重构了传输层逻辑，将调度智能从交换网络转移至边缘智能网卡（NIC），通过多路径逐数据包喷淋、单路径独立拥塞控制及接收端 Incast 协调，在纯商用以太网上实现了超 90% 的网络有效利用率与无感容灾。为了打破专有厂商锁定并构建开放生态，Keysight 携手 Meta 建立了贯穿软件参考实现、RTL 硬件仿真到物理硅片的一致性测试套件，在流片前严密拦截诸如 MoE 场景下的 RNR 唤醒死锁以及共享 RTO 引发的全网延迟投毒等恶性缺陷。 ## Atmosphere & style 整场演讲呈现出硬核务实、高密度的顶级系统架构师工程对话风格。Arvind 开篇以“军队行军由最慢士兵决定速度”的生动隐喻将尾部延迟升格为百亿美元级的资本经济学命题，展现出宏大架构视野；随后 Kingshuk 登台以“硬件补丁极其昂贵”的冷峻现实，抽丝剥茧地展开底层协议漏洞排查案例，叙事严密而扣人心弦。现场节奏紧凑、观点犀利，弥漫着大厂在超大规模基础设施前沿深水区披荆斩棘的技术自信与推动行业开放标准的号召力。 ## Bulletin - [00:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=0s) 面对扩展至百万级 GPU 的超大规模 AI 集群，Meta 正式发布全新从零设计的 RDMA 传输协议 MetaRoCE，将网络智能与负载均衡完全从交换 Fabric 下沉至边缘网卡（NIC）。 - [06:06](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=366s) MetaRoCE 采用多路径逐包喷淋与单路径独立拥塞控制架构，并在接收端引入集中调度机制化解 Incast 危机，摆脱传统对智能交换芯片和队列扩散（QP Proliferation）的强依赖。 - [13:40](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=820s) Keysight 联合 Meta 推出贯穿软件参考模型、RTL 仿真及量产硅片的专属一致性测试套件，在硬件固化流片前严格验证 RNR 超时死锁与单路径独立 RTO 状态等关键边界工况。 - [20:50](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1250s) Meta 宣布将 MetaRoCE 完全开源以推动通用以太网生态建设，并指出除当前的横向扩展（Scale-Out）外，未来还将深入重构机架内 Scale-Up 及跨区域 Front-End 网络栈。 ## Key takeaways - [01:14](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=74s) 在大规模分布式大模型训练中，网络已直接嵌入每个训练步的关键路径；集合通信受制于“最慢士兵”的木桶效应，尾部延迟已转化为直接关乎数十亿美元算力闲置的宏观经济学问题。 - [02:13](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=133s) 传统 RDMA 试图通过底层 Fabric 交换机感知拥塞并下发提示，但 Fabric 仅能看到孤立数据包；Meta 认为通信意图唯有位于边缘的端点网卡（NIC）才能洞察，应将调度控制权彻底转移至边缘。 - [04:09](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=249s) MetaRoCE 将传统粗水管解耦为大量对应用透明的“微型管道（路径）”，基于通用商用以太网交换芯片实现 90% 以上的网络有效利用率与单路径自主快速重传容灾。 - [06:06](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=366s) MetaRoCE 打破 QP 与单条网络流的死板绑定，将应用消息拆包喷淋至多条物理路径并在接收端乱序重组，原生支持 Write、Read 和 Send 等 Verbs 语义。 - [08:30](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=510s) 协议在单路径粒度上运行独立的拥塞控制与 RTO 计时器，单条路径劣化或触发降速时仅惩罚自身，绝不拖垮同一 QP 下的其他健康并行路径。 - [08:30](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=510s) 针对大规模集合通信中的 Incast 瓶颈，MetaRoCE 由天然掌握全局流并发视野的接收端动态计算公平成分（Fair Share），并通过 ACK 回传速率提示反向平息拥塞。 - [10:08](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=608s) MetaRoCE 摒弃了为榨干带宽而被迫创建海量 QP 的“队列扩散”旧模式，改由单一套协调拥塞控制统管多条物理路径，从根本上消除了 QP 间的恶性抢占。 - [11:55](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=715s) 芯片一旦固化流片（Respin），硬件打补丁的代价极其昂贵；Keysight 与 Meta 针对 MetaRoCE 1.0 打造的一致性测试套件在硬件设计之初便全面覆盖异常边界用例。 - [13:40](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=820s) 测试套件构建了贯穿“软件参考实现-硬件 RTL 仿真-物理硅片”的连续体验证流程，始终坚持统一的通过/失败基准，防止隐患在阶段交接的缝隙中溜走。 - [15:01](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=901s) 在 MoE 典型的 All-to-All 高并发 RNR（接收端未就绪）测试中，一致性工具成功抓获了 DUT 网卡在超时后未能按时唤醒特定 Tag 进而导致流无限期死锁的逻辑缺陷。 - [17:01](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1021s) 测试拓扑证实多路径下共享 QP 级 RTO 会带来“最慢路径投毒”陷阱：某一条堵塞路径的漫长等待将强行拖慢整套健康网络，硬件实现必须严格维持路径级独立 RTO 状态。 - [20:50](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1250s) MetaRoCE 坚持完全基于标准以太网（仅需标准 ECN 和 ECMP），Meta 现已全面开源该协议规范并联合生态厂商（如 AMD）落地落地，避免专用网络架构的技术碎片化与厂商锁定。 - [21:26](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1286s) 面向百万 GPU 超级集群，网络协议重构已超越 Scale-Out 范畴，机架内 Scale-Up 互联与跨数据中心 Front-End 架构的深度变革将是下一步攻关方向。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=0s) 欢迎致辞与议程：为 AI 时代重构 RDMA 演讲由来自 Meta 的 Arvind Srinivasan 拉开序幕。在致谢前序环节的引言后，Arvind 将话题切入核心主题——MetaRoCE。MetaRoCE 是 Meta 专为大规模 AI 网络全新设计打造的 RDMA 传输协议。传统 RDMA 协议在过去几十年中虽然一直扮演着高性能计算的中流砥柱，但面对如今爆发式的 AI 基础设施需求，Meta 正在对其底层逻辑进行根本性的重构。 本次演讲的脉络清晰明确：首先深入剖析超大规模集群所带来的网络挑战；接着详细拆解 MetaRoCE 的核心设计机制；随后由来自 Keysight 的 Kingshuk Mandal 介绍一致性测试与互操作性验证体系；最后展望 Meta 对下一代开放网络标准的愿景。正如 Arvind 指出的，Meta 当前的算力底座正从数千张 GPU 飞跃到横跨多个数据机房、数据中心及区域的“百万 GPU 级”超级集群。在这种史无前例的体量下，若要连接数百万个计算加速器，所需光纤和线缆规模将达到数万亿根。网络早已不再是偏安后台的辅助设施，互联通信成为了决定百万 GPU 能否高效协同运转的生死命题。 --- ### [01:14](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=74s) 规模化挑战：“行军”隐喻与尾部延迟的经济代价 AI 网络的基础架构正在以指数级速度演进，连接加速器的规模已实现 10 倍增长。在传统数据中心内，网络或许只是辅助性的管道，但在超大规模分布式大模型训练中，它直接嵌在每一个训练步（Training Step）的关键路径上。 Arvind 用了一个极为生动的比喻来阐释集体通信（Collectives）的运行本质： > “集体通信就像是一支正在行进的军队，整个方阵的前进速度完全取决于走得最慢的那名士兵。”（*Collectives are like an army marching. The entire formation moves at the pace of the slowest soldier.*） 在 AllReduce 等分布式集合通信模式下，整体性能直接受制于集群中最慢的木桶短板。哪怕网络仅引入了微小的长尾延迟或瞬态抖动，都会导致海量加速芯片陷入阻塞等待，瞬间转化为价值数十亿美元的闲置搁浅算力（Stranded Compute）。因此在 AI 时代，尾部延迟（Tail Latency）与网络弹性（Resiliency）已不仅仅是纯粹的工程调优指标，而是一个直接与资本开支挂钩的宏观经济学问题。 --- ### [02:13](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=133s) 传统架构的瓶颈：将智能从网络 Fabric 转移到边缘 NIC 面对规模暴增引发的通信墙，Arvind 带领大家退后一步，审视传统 RDMA 的底层缺陷。过去数十年间，RDMA 一直是高性能计算（HPC）的默认选择，但当集群节点扩展至数万台时，经典架构立刻捉襟见肘，热点拥塞和链路欠载（Underutilization）层出不穷。业内传统的修补方案通常是将传输层视为与工作负载无关的黑盒，试图通过向底层交换 Fabric 发送各种拥塞提示（Hints），寄希望于让网络 Fabric 自行平息拥塞。然而现实却是按下葫芦浮起瓢：网络中频繁出现持续性拥塞和 PFC 暂停帧风暴（Pause Storms），工程师陷入无休止的手工参数调优和故障救火中。 问题的症结在于职责划分的根本错位：**底层 Fabric 只能看到一个个孤立的数据包（Packets），而真正的通信意图（Intent）唯有位于边缘的端点（Endpoints，即智能网卡 NIC）才能洞察。** Meta 的破局思路是彻底颠覆传统范式：既然网卡掌握着完整的语义和意图，那就应将负载均衡的控制权完全转移给网卡端点。让端点直接基于流意图在全网做负载分发，热点与不均衡便会迎刃而解。只要赋予网卡细粒度的逐路径（Per-path）感知能力，端点即可自主驱动网络弹性与聚合吞吐。这种“边缘驱动的负载均衡”彻底解耦了传输层对交换机内部复杂特性的依赖，无需依赖聪明的交换机，即可利用多平面（Multiplane）光纤拓扑构建出 TCO（总体拥有成本）高度优化的网络。 ```mermaid flowchart TD accTitle: 从智能网络交换向智能边缘网卡的范式转移 accDescr: 对比传统架构中网络 Fabric 负责拥塞调度的黑盒模式，与 MetaRoCE 将感知和调度下沉至智能网卡的新模式。 subgraph Traditional["传统 RDMA 模式 (智能在交换网)"] direction TB App1["应用 / 工作负载"] --> NIC1["传统哑网卡 (粗粒度 ECN/CNP)"] NIC1 --> Fabric1["复杂 Fabric 交换机 (试图猜测意图 / 频发 PFC 风暴)"] end subgraph MetaRoCE["MetaRoCE 模式 (智能移至边缘)"] direction TB App2["应用 / 集合通信意图"] --> NIC2["智能网卡 (掌握通信意图 / 动态路径感知)"] NIC2 -->|"逐数据包喷淋 (Per-packet Spray)"| Fabric2["通用商用以太网 (仅依赖 ECMP / 无需特殊交换机)"] end ``` --- ### [04:09](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=249s) 智能边缘的构想与 MetaRoCE 的三大规模维度 究竟什么才是真正的“智能边缘”？传统上，上层应用将底层网络视为一根不透明的单一粗水管（Single Opaque Pipe），端点缺乏内部拓扑能见度，只能依赖 ECN 或 CNP 等粗粒度回馈信号进行被动反应。而 Meta 所倡导的智能边缘，则是将这根粗水管分解暴露为大量精细的“微型管道（Tiny Pipes）”，向传输层实时输出丰富的路径细节，从而让网卡在微秒级时间内做出最优决策。更关键的是，这一复杂的精细化控制必须完全对上层应用透明。 基于这一核心原则，Meta 抛弃了给传统 RDMA 缝缝补补的“打补丁”做法，推出了全新设计的 MetaRoCE 协议，从三个维度全面破解超大规模网络困局： - **极致性能（Performance at scale）：** 通过基于逐数据包的智能动态负载均衡与多路径分发，直接锚定 90% 以上的网络有效利用率。 - **高弹性容灾（Resiliency）：** 在单路径（Per-path）粒度上实现自主可靠传输与快速重传，在完全不需要网络 Fabric 交换机参与干预的前提下，达成极致的高可用性。 - **超大规模运维（Operations at scale）：** 完全基于通用商用以太网（Commodity Ethernet）交换芯片运行，无需昂贵的专有交换架构；同时向下保持完整的 Verbs API 级别兼容性，应用软件无需任何修改即可实现无摩擦落地迁移。 --- ### [06:06](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=366s) 核心机制：“路径（Paths）”、多路径喷淋与无感容灾 深入到 MetaRoCE 协议的技术内部，其最为核心的基础抽象即是“路径（Path）”，对应于前文提及的微型管道概念。传统 RDMA 将一对通信节点（QP）死死绑定在单一哈希网络流上，极易产生大象流碰撞并导致链路闲置；而 MetaRoCE 彻底打破了这种一对一绑定。 在 MetaRoCE 的运行机制下，上层应用程序依然按常规方式提交消息（Messages），传输层则立即将该大块消息拆切为众多数据包，并将它们喷淋打散（Spray）到多条并行路径上并发传输。在接收端网卡，协议直接将乱序到达的数据包原位重组写入应用程序的接收缓冲区中。这一处理全程独立于特定的 RDMA 语义，无论是 Write、Read 还是 Send 均可原生支持。 每一个 MetaRoCE 路径都具备以下关键工程特性： - **独立的熵标识：** 每条路径都由传输层分配并全权管理独立的熵值（Entropy），以此利用底层 ECMP 机制遍历不同路由，对上层应用完全透明。 - **独立的可靠性与拥塞状态：** 每条路径维持自己独立的重传控制与拥塞窗口，这一解耦特性使其能够天然适应具有非均匀 RTT（Non-uniform RTTs）的大型复杂拓扑。 - **基于实时 RTT 与可用窗口的动态负载均衡：** 发送端根据每条路径当前实测的 RTT 与空闲发送窗口，智能调度数据包，主动避让突发热点和慢速节点。 - **超时熵值自动轮转（Entropy Rotation）：** 一旦某条路径遭遇超时或黑洞丢包，网卡自动切换该路径的熵值，瞬间重定向至健康的网络链路。 在大规模集群环境中，光纤抖动、链路 Flap 和交换节点偶发故障属于家常便饭。MetaRoCE 将抽象出来的逻辑路径映射至物理网络的多套平面（Fabric Planes）。一旦某条物理平面突发断链，流量会自动且透明地瞬移至备用平面，上层 AI 任务毫无感知。由于各路径独立追踪自身 RTT 与拥塞，单条路径的劣化或中断绝不会“拖垮”其他正常路径，从而构建起高度隔离的容灾屏障。 --- ### [08:30](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=510s) 单路径粒度的拥塞控制与接收端驱动的 Incast 管理 在拥塞控制与可靠性的协同设计上，MetaRoCE 展现出独特的架构思考。MetaRoCE 采用了在**单路径粒度（Per-path Granularity）**上运行的发送端驱动拥塞控制机制，这一设计显著区别于业内现有绝大多数 RDMA 变体方案。 这种粒度带来的核心优势是精准的故障域隔离：当某条路径检测到 ECN 标记需要触发降速回退时，受影响的仅仅是该条拥塞路径本身，其他健康路径的传输速率完全不受牵连。Arvind 强调：“一条路径犯了错，只会惩罚它自己，而不会株连其他任何人。”在顶层暴露给应用或 QP 的总发送窗口，实质上是底层所有活跃路径窗口的代数和。因此，系统可以通过并发增加路径数量来线性扩展通信容量，而绝不会增加单路径上的激进碰撞概率。 ```mermaid flowchart LR accTitle: MetaRoCE 单路径拥塞控制与接收端 Incast 反馈闭环 accDescr: 发送端多路径喷淋数据包至接收端缓冲区，接收端统计发送端数量并通过 ACK 回传速率提示，实现硬件级公平带宽调度。 subgraph Sender["发送端 NIC"] P1["路径 1 (独占 RTT/拥塞窗口)"] P2["路径 2 (ECN 降速 / 仅惩罚自身)"] P3["路径 3 (独占 RTT/拥塞窗口)"] end subgraph Fabric["商用以太网 Fabric"] F["标准 ECMP / ECN 转发<br/>(无专有交换协议)"] end subgraph Receiver["接收端 NIC"] Buf["接收缓冲区 (直接重组落盘)"] Calc["接收端调度引擎<br/>(计算并发 Sender 数 & Fair Share)"] end P1 -->|"Packet 1"| F P2 -->|"Packet 2"| F P3 -->|"Packet 3"| F F --> Buf Buf --> Calc Calc -.->|"ACK 携带速率提示 (Rate Hints)"| Sender ``` 面对大规模集合通信中最棘手的 Incast（多对一突发涌入）问题，MetaRoCE 引入了创新性的接收端协同机制。接收端网卡凭借端点优势，天然具有对并发工作负载的全局视野。接收端在任意时刻都能精确掌握有多少个发送端正在向自身发起传输，据此动态计算出每个流的公平成分（Fair Share），并通过 ACK 确认报文直接将速率提示（Rate Hints）回传给各发送端。发送端依据反馈主动调整注入速率，在交换机缓冲区被击穿前就地化解 Incast 危机，完全无需特殊的智能交换机介入。 在谈到规模化运营时，Arvind 指出，传统上应用程序为了榨干网络带宽，往往不得不强行创建数十甚至上百个 QP（Queue Pairs），并在软件层痛苦地将大消息切片分发。而 MetaRoCE 的诞生，正从根本上重塑这一过时的软件栈交互范式。 - **通用硬件解耦：** 整个协议设计不依赖专有以太网交换特性，仅需标准交换芯片支持标准的 ECN 标记与 ECMP 散列转发即可平滑运作。 - **Incast 智能移至接收端：** 由感知通信上下文的接收端 NIC 统筹调度各发送节点速率，根除对网络层深缓冲区或复杂 PFC 门限设置的强依赖。 - **单路径容灾与性能隔离：** 单一路径降速或故障互不干扰，总可用窗口为多路径动态求和，兼顾高吞吐与稳态控制。 ### [10:08](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=608s) 摆脱队列扩散：多路径与单一拥塞控制的演进 在传统 RDMA 架构中，系统扩展带宽的标准做法是将流量切分并分配给越来越多的队列对（Queue Pair，简称 QP）。然而，随着 QP 数量不断膨胀，网络将面临严重灾难：每个 QP 都在运行各自完全独立、缺乏协调的拥塞控制算法。其直接后果就是各个 QP 相互竞争、相互踩踏，最终在规模化扩展时彻底崩溃。MetaRoCE 彻底摒弃了这种“队列扩散”（Queue Proliferation）的旧思路。当系统需要更大带宽时，MetaRoCE 不去增加 QP，而是引入多路径（Multipath）架构——通过增加并发网络路径来扩展带宽，同时让所有流量统一受控于“单一且协同的拥塞控制机制”，从而将整体流量特征彻底抚平。 Meta 的核心诉求非常直接：让海量 GPU 能够平稳运行在异构网络基础设施之上，拒绝技术碎片化，更不希望针对每一种计算负载或硬件厂商去专门定制专属硬件。为此，Meta 决定将 MetaRoCE 协议面向全行业彻底开源。在交换机层面，该协议完全依赖行业通用标准特性，仅需交换机支持显式拥塞通知（ECN）和等价多路径路由（ECMP），不包含任何供应商绑定或专有私有扩展。然而，正如演讲者所言，“缺乏互操作性的开放仅仅是一纸空文”（"openness without interoperability is just a spec"）。为了让不同厂商的芯片在物理链路上能够“逐比特准确对接”，必须建立一套严格的一致性认证体系。随后，来自 Keysight（是德科技）的 Kingshuk Mandal 受邀登台，详细拆解了一致性验证与互操作性的实现细节。 - **传统扩展模式的致命伤**：QP 线性增加导致非协同拥塞控制相互干扰，引发严重的网络踩踏与拥塞崩溃。 - **MetaRoCE 的解法**：保持 QP 简洁，利用多路径扩展物理带宽，依托全局统一协调的拥塞控制平滑流量峰值。 - **无专有硬件绑定**：整网基于通用以太网构建，交换机侧仅依赖标准 ECN 与 ECMP 特性。 ### [11:55](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=715s) 规范即产品：Keysight 一致性测试套件的诞生 Kingshuk Mandal 指出，一份协议规范只有在多方硬件厂商于物理链路层面达成完全一致时，才能真正演化为具备互操作性的工业系统；在此之前，“单个实现同自身的对话，不过是纯粹的私有实现”。一致性测试（Conformance Testing）的本质，就是将写在纸面上的规范文本转化为严谨、可执行的测试断言。在协议制定之初就编写测试用例具有关键的战略意义，因为及早暴露模糊地带的修复成本最低。现场响起了一句极具共鸣的行业警醒：“硬件的补丁是极其昂贵的”（"Hardware is brutally expensive to patch"）。一旦芯片设计被冻结，规范中残留的任何歧义都将随着硅片一同被固化流片，无可挽回。 为此，Keysight 携手 Meta 针对 MetaRoCE 1.0 规范专门开发了业内首创的一致性测试计划。该测试计划具备四大核心特质： 1. **纯自研首创**：完全针对 MetaRoCE 协议特性从零构建； 2. **代码实证而非纸上谈兵**：每一项测试用例都在真实的软件参考实现上经过了严格验证，“通过/失败门槛真实存在”； 3. **覆盖全面**：横跨硬件报头完整性、链路可靠性、拥塞控制等协议全生命周期领域； 4. **穷尽异常边界**：不仅覆盖一切正常顺畅的“顺利路径”（Happy Path），更着重探索各类否定应答（NAK）、计时器超时、异常恢复等极度容易潜藏恶性 Bug 的边缘场景。 ### [13:40](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=820s) 贯穿始终的测试连续体：从软件参考到芯片硅片 在实际工程落地中，测试并非芯片流片之后才开始的收尾工序，而是一个不可割裂的“测试连续体”（Testing Continuum）。无论被测对象处于研发周期的哪一阶段，Keysight 的测试套件、测试用例以及验证工作流都保持完全一致，并以完全相同的“通过/失败”（Pass/Fail）质量门槛进行衡量。这一连续统一的基准线在整个硬件迭代周期中绝不动摇。 ```mermaid flowchart LR accTitle: MetaRoCE 一致性测试连续体流程 accDescr: 展示从软件参考实现到硬件 RTL 仿真再到最终量产物理硅片，统一测试套件与不可动摇的通过标准贯穿始终。 subgraph TestingEngine ["一致性验证核心"] T["Keysight 一致性测试套件<br/>(固定不变的 Pass/Fail 门槛)"] end subgraph ContinuumPhases ["硬件演进三阶段"] A["阶段一：软件参考实现<br/>(修复成本：修改代码)"] B["阶段二：RTL 硬件仿真<br/>(修复成本：逻辑迭代)"] C["阶段三：量产物理硅片<br/>(修复成本：重新流片 Respin)"] end T -.->|"协议级测试用例"| A T -.->|"无缝迁移执行"| B T -.->|"硬件实测验收"| C ``` 在架构定义初期，测试套件直连软件参考实现；若此时发现 Bug，修复仅仅是“修改几行代码”。随着架构逐渐成熟，相同的测试集直接无缝切换至 RTL 级硬件仿真环境；而当芯片最终流片下线，物理硅片仍需经受这套完全相同的协议测试检验。如果在流片后才撞上同样的逻辑漏洞，代价将是一次极其昂贵的重新流片（Respin）。Meta 在项目第一天就决定引入严苛的一致性测试，正是为了避免在产品交付后撰写惨痛的“事后复盘报告”（Postmortem Report）。 - **统一质量门槛**：测试标准在软硬件交接、架构演进过程中不发生任何妥协降级，彻底杜绝隐患在交接缝隙中溜走。 - **Keysight AI Conformance 工具**：支持同时模拟发送端与接收端角色，具备随时注入故障以探索非正常执行路径的测试能力。 ### [15:01](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=901s) 案例一：MoE 场景下的 RNR（接收端未就绪）与标签级暂停 在大语言模型（LLM）分布式训练中，混合专家模型（MoE）是极具代表性的高并发工作负载。计算任务被分散在数以万计的算力节点上，在特定阶段，所有节点必须同时与其他所有节点进行高密度的“全对全”（All-to-All）通信。在瞬时海量并发冲击下，某些节点的主存或缓冲区可能会临时耗尽，无法再容纳新传入的数据。Kingshuk 强调，这在分布式系统中根本不是系统错误或硬件故障，只是机器在那一瞬间尚未准备好，行业标准术语称之为 RNR（Receiver Not Ready，接收端未就绪）。 面对 RNR，接收端需要礼貌地通知发送方稍后重试。在 Keysight 搭建的测试拓扑中，被测网卡（DUT）作为发送端打开了一个 QP，并利用多个不同的 Tag（标签）向扮演接收端的 Keysight 验证工具并发发起多个数据流。当接收端对其中某一个特定 Tag 触发 RNR 响应时，MetaRoCE 规范有着极其明确严格的界定：**网卡必须且只能暂停受影响的那一个 Tag 及其对应的消息传输，而该 QP 上的其余所有正常 Tag 必须维持满速运转**。 ``` [单个 QP 队列] ────┬── Tag 0: 正常传输 ───────────────> [接收端] ├── Tag 1: 收到 RNR 暂停 ──[等待超时唤醒]──> [接收端缓冲区满] └── Tag 2: 正常传输 ───────────────> [接收端] ``` 测试执行时，工具精准抓获了一个关键逻辑缺陷：DUT 网卡确实如期挂起了受阻的 Tag，但在预设的超时时间到达后，网卡竟然“彻底忘记了将其唤醒”，导致该特定数据流无限期锁死停滞。MoE 只是触发 RNR 的典型场景之一，借助该一致性工具，开发团队能够随时随地复现这一恶劣边界工况，在硬件固化前将死锁 Bug 扼杀。 ### [17:01](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1021s) 案例二：多路径下的独立 RTO 与“最慢路径投毒”陷阱 如果说 RNR 是业务负载引发的边界挑战，那么重传超时（RTO）问题则是纯粹由网络拓扑与流量动态诱发的系统陷阱。MetaRoCE 采用数据包逐包喷洒（Packet Spraying）跨越多条网络路径传输，且必须对每条路径的状态进行独立追踪。RTO 是发送端判断报文是否丢失并触发重传的关键计时器： - **RTO 设得过小**：网络中将充斥大量不必要的虚假重复报文，直接冲垮有效吞吐率（Goodput）； - **RTO 设得过大**：丢包后数据流将陷入漫长等待，直接拉垮整体作业完成时间（JCT）。 MetaRoCE 规范强制要求：**每一条物理路径都必须维护其专属且相互独立的 RTO**。然而，这恰恰是硬件设计中最具诱惑力的“工程偷懒捷径”——许多硬件团队倾向于让整个 QP 共享一个全局 RTO 状态。共享 RTO 能大幅节省芯片片上 SRAM 内存占用，显著减少控制逻辑代码量，实现起来看似极其优雅，但在多路径环境下这是一种致命错误。 为了验证该缺陷，Keysight 构建了差异化路径拓扑： - **路径 0（Path 0）**：维持为绝对高速、通畅的健康路径，预期收敛至极低的 RTO 下限值； - **路径 1（Path 1）**：人为注入持续拥塞，预期维持极高的 RTO 以吸收网络延迟。 在正确的实现下，Path 0 保持低延迟快速重传，Path 1 单独拉长重传周期，系统顺利通过测试。但倘若网卡采用了共享 RTO 机制，最慢的那条拥塞路径（Path 1）就会彻底拖垮整组健康路径，“单条不健康的路径将污染整套优质路径网络”。此时用户虽然为 8 条高速通道付出了高昂硬件成本，但所有路径却全都被迫降速到那条堵死路径的爬行速度。更隐蔽的是，这种设计缺陷在链路均等的纯净实验室环境中极难被捕获，只有在真实生产环境的重载高压下，才会表现为令人匪夷所思的“多路径延迟强行耦合”，排查往往耗时数周之久。 ### [19:34](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1174s) 一致性测试的行业价值：从私有实现到繁荣生态 Kingshuk 总结道，对于新一代 MetaRoCE 传输协议而言，从首日起推行前置一致性测试是整个产业向前发展的唯一正确路径。无论在场工程师扮演何种行业角色，都能从中获取三重深远收益： - **协议设计与实现者**：在芯片硅片尚未定型的可编辑阶段尽早澄清规范歧义，使规格标准在比特级实现上持续加固； - **网络构建与设备集成者**：获得真正具备鲁棒性的多路径传输机制，该机制在主动故障注入下充分验证了流量隔离与故障恢复能力，绝非仅仅跑通顺利路径； - **网络集群运维者**：能够在系统上线前提前扫除仅在真实复杂业务流下才会爆发的深层隐患。 单打独斗的技术实现终究只是封闭的私有协议。只有依靠从“Day 0”起步的全套严格一致性测试，才能将纸面上的草案转化为跨厂商自由互通、蓬勃发展的行业开放生态。 ### [20:39](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1239s) 硬件修复的高昂代价与一致性测试套件的交接 来自 Keysight 的 Kingshuk Mandal 在结束关于 MetaRoCE 一致性测试套件（Compliance Suite）的阐述时强调，构建严格的验证生态至关重要，因为“给已经冻结流片的硅晶圆打补丁是极其昂贵的”（"it is brutally expensive to patch a frozen die"）。硬件一旦进入硅片固化阶段，任何设计缺陷导致的重做都将带来无法承受的成本与工期延误。基于此，确保从软件参考模型、硬件仿真到最终量产芯片整个链条的合规性是项目落地的基石。随后，Kingshuk 将发言权交还给 Meta 的 Arvind Srinivasan，由他展望 MetaRoCE 的未来演进与下一步规划。 ### [20:50](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1250s) 迈向百万 GPU 时代：MetaRoCE 的定位与开源愿景 Arvind 接过麦克风，对 Kingshuk 介绍的合规套件表示感谢，并总结了 Meta 当前的网络现状与未来方向。他指出，明天即将面对的网络规模是前所未有的：Meta 目前正在着手构建百万级 GPU 的超大规模集群，必须真正解决横跨数百万计算接入点（Access Points）之间的“万亿级连线连接”（"trillion wire connectivity"）挑战。在如此庞大的问题域中，MetaRoCE 展现出了核心价值——它不仅能充分满足 AI 工作负载所要求的峰值性能与高可用性，更关键的是完全基于通用商用以太网（Commodity Ethernet）构建，不需要任何特殊的交换机，彻底消除了厂商锁定（Vendor Lock-in）的隐患。 为了推动整个产业共同进步，Meta 正在向全行业开放 MetaRoCE，积极赋能 NIC（网卡）生态系统跟进实现这一传输协议规范。Arvind 提到，当天晚些时候，来自合作伙伴团队的成员还将登台分享他们如何在 AMD 平台上具体实现与落地 MetaRoCE。 ```mermaid flowchart TD accTitle: MetaRoCE 规模化设计与挑战演进 accDescr: 展示 MetaRoCE 支撑百万 GPU 集群的核心优势及后续跨层级网络挑战的扩展方向。 subgraph ScaleOut ["Scale-Out 网络现状"] A["百万级 GPU 集群<br/>万亿级连线连接"] --> B["MetaRoCE 协议"] B --> C["通用以太网基础设施<br/>(无特殊交换机 / 无厂商锁定)"] B --> D["极高吞吐与高可用性"] end subgraph Ecosystem ["产业生态共建"] E["开放标准与开源推进"] --> F["赋能网卡 (NIC) 生态"] F --> G["AMD 平台落地实现"] end subgraph NextHorizons ["跨层级全栈重构挑战"] H["机架内 Scale-Up 互联"] I["跨区域 Front-End 网络"] end B -.-> E B -.-> NextHorizons ``` ### [21:26](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1286s) 超越 Scale-Out：AI 网络全栈的深度重构 在演讲的尾声，Arvind 特别提醒大家“我们还远未大功告成”（"we are not done yet"）。AI 带来的网络挑战远远超出了横向扩展（Scale-Out）网络的范畴： - **机架内 Scale-Up**：机架内部更高密度、极低时延的扩展网络正在面临全新的物理与协议瓶颈。 - **跨区域 Front-End 网络**：跨地域数据中心之间的前端通信同样是团队正在积极攻关的核心场景。 Arvind 总结道，AI 基础设施网络协议栈的每一个层级都迫切需要经历类似 MetaRoCE 这样的根本性重构与重新审视。他呼吁行业持续关注并携手探索，随后向全场观众致谢完结本次分享。 ## Notable quotes > [01:14](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=74s) Collectives are like an army marching. The entire formation moves at the pace of the slowest soldier. — Arvind Srinivasan > [02:13](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=133s) 底层 Fabric 只能看到一个个孤立的数据包（Packets），而真正的通信意图（Intent）唯有位于边缘的端点（Endpoints，即智能网卡 NIC）才能洞察。 — Arvind Srinivasan > [08:30](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=510s) 一条路径犯了错，只会惩罚它自己，而不会株连其他任何人。 — Arvind Srinivasan > [10:08](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=608s) openness without interoperability is just a spec — Arvind Srinivasan > [11:55](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=715s) 单个实现同自身的对话，不过是纯粹的私有实现 — Kingshuk Mandal > [11:55](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=715s) Hardware is brutally expensive to patch — Kingshuk Mandal > [15:01](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=901s) 网卡必须且只能暂停受影响的那一个 Tag 及其对应的消息传输，而该 QP 上的其余所有正常 Tag 必须维持满速运转 — Kingshuk Mandal > [17:01](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1021s) 每一条物理路径都必须维护其专属且相互独立的 RTO — Kingshuk Mandal > [17:01](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1021s) 单条不健康的路径将污染整套优质路径网络 — Kingshuk Mandal > [20:39](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1239s) it is brutally expensive to patch a frozen die — Kingshuk Mandal > [20:50](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1250s) trillion wire connectivity — Arvind Srinivasan > [21:26](https://www.youtube.com/watch?v=QX8C-8uxXKc&t=1286s) we are not done yet — Arvind Srinivasan ## Mentioned Arvind Srinivasan, Kingshuk Mandal, Meta, Keysight, AMD, MetaRoCE, RDMA, RoCE, Verbs API, ECN, CNP, PFC, ECMP, QP (Queue Pair), RNR (Receiver Not Ready), RTO (Retransmission Timeout), MoE (Mixture of Experts), Santa Clara Convention Center, @Scale ## Video description Arvind Srinivasan from Meta and Kingshuk Mandal from Keysight present "MetaRoCE: Meta’s RDMA Transport" live at the Santa Clara Convention Center. At Meta's scale — clusters growing to over a million accelerators across regions — the network is in the critical path of every AI training step, and traditional RDMA was never built for it. MetaRoCE is Meta's clean-sheet RDMA transport: a multipath, out-of-order, receiver-driven protocol that treats Ethernet as inherently lossy and pushes all intelligence into the NIC. By shifting intelligence to the edge, the transport decouples from topology — enabling TCO-optimized multiplane FPF topologies with frictionless application adoption on commodity Ethernet. This talk covers MetaRoCE's core mechanisms and Meta's commitment to open standards, including a companion conformance suite developed with Keysight that validates implementations from software reference through hardware emulation to production silicon. Meta is opening MetaRoCE to the industry and inviting broader collaboration to establish it as the ubiquitous RDMA transport for AI-scale networks. Learn more about @Scale here: atscaleconference.com [Source](https://www.youtube.com/watch?v=QX8C-8uxXKc)

Source quotes

[1:14] Arvind Srinivasan: Collectives are like an army marching. The entire formation moves at the pace of the slowest soldier. [2:13] Arvind Srinivasan: 底层 Fabric 只能看到一个个孤立的数据包（Packets），而真正的通信意图（Intent）唯有位于边缘的端点（Endpoints，即智能网卡 NIC）才能洞察。 [8:30] Arvind Srinivasan: 一条路径犯了错，只会惩罚它自己，而不会株连其他任何人。 [10:08] Arvind Srinivasan: openness without interoperability is just a spec [11:55] Kingshuk Mandal: 单个实现同自身的对话，不过是纯粹的私有实现 [11:55] Kingshuk Mandal: Hardware is brutally expensive to patch [15:01] Kingshuk Mandal: 网卡必须且只能暂停受影响的那一个 Tag 及其对应的消息传输，而该 QP 上的其余所有正常 Tag 必须维持满速运转 [17:01] Kingshuk Mandal: 每一条物理路径都必须维护其专属且相互独立的 RTO [17:01] Kingshuk Mandal: 单条不健康的路径将污染整套优质路径网络 [20:39] Kingshuk Mandal: it is brutally expensive to patch a frozen die [20:50] Arvind Srinivasan: trillion wire connectivity [21:26] Arvind Srinivasan: we are not done yet

Source quotes

[0:00] Arvind Srinivasan: like an army marching [2:13] Arvind Srinivasan: the network fabric only sees raw packets, whereas only the NIC and endpoints see the actual workload intent [11:00] Kingshuk Mandal: A protocol specification becomes an interoperable system when multiple vendors agree on the wire. Until then, one implementation talking to itself is a private implementation... Specification is written words, and conformance converts these written words into testable assertions. [11:00] Kingshuk Mandal: silicon is brutally expensive to patch [12:54] must pause only the impacted tag and its associated message [12:54] forgot to wake up after the timeout expired [17:10] Kingshuk Mandal: the single most tempting shortcut in the whole design [17:10] Kingshuk Mandal: One unhealthy path is going to poison the entire good set [17:10] Kingshuk Mandal: You have paid for eight individual lanes, but you are driving all of them at the speed of the one that is jammed. [17:10] Kingshuk Mandal: a mysterious latency coupling multiple paths under load [19:34] Kingshuk Mandal: hardware is brutally expensive to patch [20:50] Arvind Srinivasan: trillion-wire connectivity across millions of ASICs

Original transcript

[0:00] Morning [0:04] everyone and um thanks a lot Ju for a [0:07] great introduction on MRC. Now let's [0:09] switch gears and talk about Metari. [0:12] Metari is Meta's RDMA transfer protocol [0:14] purpose-built for AI networking at [0:16] scale. So this is what we plan to cover [0:19] today. We are fundamentally rethinking [0:21] RDMA for the AI era. What we plan to [0:24] cover is the scale challenges, [0:26] metaroki's core mechanisms, compliance [0:28] and interoperability. King shook from [0:30] keyside will talk through that and our [0:32] vision for the future. So let's get [0:34] started. [0:36] So as Omar mentioned earlier, we are at [0:39] Meta, we are scaling from thousands of [0:41] GPUs to over million GPU that spans data [0:44] halls, data centers and regions. Now to [0:48] keep up with the demand of GPU [0:49] connectivity, network also grows at the [0:52] same rate. Now one can imagine if you [0:54] start connecting millions of [0:56] accelerators we probably need trillions [0:58] of wires and when you start operating at [1:00] this scale the interconnect becomes a [1:03] defining challenge. The network is no [1:06] longer in the background and it [1:07] determines whether the million GPUs can [1:09] actually work together or not. So let's [1:12] see how is the network transforming. [1:14] So the AI networking is also [1:16] transforming at an exponential rate. It [1:18] is no longer an auxilary infrastructure [1:20] and it is a critical part of every [1:22] training step. So what the graph shows [1:25] here is the way the number of connector [1:27] accelerators are going over a period of [1:30] time and we see it's going 10x. Now at [1:34] this scale the challenges are very real. [1:38] They require fundamentally rethinking [1:40] how we build networks. So think about [1:43] this way. Collectives are like an army [1:45] marching. The entire formation moves at [1:48] the pace of the slowest soldier. [1:51] Similarly, the performance is gated by [1:53] the slowest node in the collector. So [1:56] what that translates is any extra [1:59] latency that you add directly translates [2:01] to billions of dollars of standard [2:03] compute. And that's a problem for us [2:05] because tail latency and resiliency [2:07] becomes a free economics problem. That's [2:09] the shift we are talking about here with [2:11] the AI network. [2:13] So well let's see how do we solve this [2:15] problem. So I want to take a step back [2:17] and go through a fundamental [2:18] observation. RDMA has been the workhorse [2:21] for high performance computing for [2:23] decades. But as the cluster scale from [2:25] to tens of thousand of nodes the [2:28] traditional model started to break. G2 [2:30] talked about it in a few minutes ago how [2:32] you start to see hotspots and [2:34] underutilization. That's what happens [2:36] today. But how do we how are we working [2:38] around this problem? Well, the transport [2:40] is supposed to be workload agnostic, but [2:42] we end up passing hints to the fabric [2:44] and hoping that the fabric can resolve [2:46] this problem. But yet what happens is we [2:49] start to see persistent congestion boss [2:51] storms and it's a constant firefighting [2:53] of tuning and to see if we can get rid [2:55] of the hot spot under utilization. Well, [2:59] let's see what the real core problem is. [3:01] Actually, the core problem is the fabric [3:03] only knows the packets. the nick or the [3:06] endpoints sees the intent. That's what [3:09] we are talking about. So now if I if I [3:11] kind of move the problem to the place [3:12] where we actually know the intent which [3:14] is essentially the end point and [3:16] endpoint starts to load balance on the [3:18] fabric you probably see a much better [3:20] picture here because there are no more [3:21] hotspot no moreization. [3:24] Now if the nick has enough information [3:26] which is such as a perp [3:29] it can manage resiliency and drive the [3:31] aggregate performance directly. So what [3:33] that effectively means is can we do an [3:36] eddriven load balancing and sustain high [3:38] availability without relying on the [3:40] fabric to be smart. Now as you kind of [3:43] go down this way it also leads to [3:45] another picture. Well what if I can make [3:48] a TCO optimized topology such as a [3:50] multiplane fabric. Now in order to make [3:53] this work the edges needs to be smart [3:56] and the the transport need to be aware [4:00] of what's happening in the in the thing. [4:02] So that's the fundamental shift we are [4:04] talking about saying let's move the [4:05] intelligence from the fabric to the [4:07] edge. [4:09] Okay. So let's talk about what is an [4:11] intelligence edge actually looks like. [4:15] So traditionally network applications [4:18] sees the fabric as a single opaque pipe. [4:20] The endpoint has absolutely no [4:22] visibility as to what's happening [4:24] inside. It only sees coarse grain [4:26] information that comes in the form of [4:28] some kind of ECN or CNP something comes [4:30] there. Now when you move from that world [4:33] to a world where start seeing instead of [4:35] giving coarse grain view let's give a [4:37] little bit more fine grain view. [4:39] Essentially can we expose the network uh [4:42] big fat pipe as multiple tiny pipes and [4:45] provide enough information for the [4:47] transport to now make a little bit more [4:49] intelligent decision and that's the [4:51] shift we are talking about where can we [4:53] move the intelligence from the fabric to [4:55] the edge but it has to be done in a in a [4:58] manner that is totally transparent to [4:59] the application and this is our core [5:02] principle of how we built our protocol [5:04] now let's see what the protocol really [5:07] is right So, Metaroki is a fundamentally [5:10] redesigned RDMA transport for AI [5:12] workload that runs on commodity [5:15] Ethernet. So, what we did is instead of [5:17] bandating traditional RDMA, we built [5:19] something new. Our protocol addresses [5:21] three dimensions of scale. One, [5:24] performance at scale. We are targeting [5:26] 90% efficiency that we achieve through [5:28] per packet load balancing and [5:30] multipaththing through the intelligence [5:32] I just described earlier. Number two, [5:35] resiliency. [5:37] We are providing perpath reliability and [5:39] resilient to scale with high [5:41] availability without any fabric level [5:43] intervention. And the third thing which [5:45] is very important is operations at [5:48] scale. Now for this we are really [5:51] shooting for a commodity ethernet [5:52] infrastructure and providing full burps [5:55] level compatibility so that we don't [5:57] have to change applications. This is [5:59] somewhat of a frictionless adoption to [6:00] the application with absolutely no [6:02] special switch required to operate. [6:06] I want to now talk about one click down [6:09] and say one of the core principle of the [6:11] protocol which is what we call as a path [6:14] and this somewhat ties to the tiny pipe [6:16] example I just described a few moments [6:17] ago. Paths are equivalent of that. So [6:20] this is a fundamental basic concept of [6:22] print protocol that we built upon it. [6:24] Now traditionally RDMA pins everything [6:27] to a single flow that results in [6:29] elephant flows and underutilization. We [6:31] start to took a different approach here [6:34] in this example. We want I want to show [6:36] we have an application that posts a [6:37] bunch of messages. The messages get [6:39] split into multiple packets. We spray on [6:41] different parts. Okay. On the receiver [6:43] end, we kind of reassemble it straight [6:45] into the receiver buffer. Now, all of [6:47] this happens independent of the [6:48] semantics that we work off. We not tied [6:50] to a single semantics. We support all [6:52] the possible semantics here. But here's [6:55] more to it, right? Every path is [6:57] identified by an indistinct entropy. [6:59] Fully managed by the transport, [7:01] transparent to the application. Number [7:03] two, every path maintains an independent [7:06] reliability and congestion state. So [7:08] that allows us to scale naturally across [7:10] non-uniform RTS. [7:13] Third, the load balancing that happens [7:15] across the paths is based on the [7:17] measured RT at that point in time and [7:19] available windows. So we can really [7:21] smartly move around the congestion and [7:23] find out where where the hotspots are. [7:26] And lastly on a time out, we do rotate [7:28] entropies and find a better path and [7:30] keep going there. [7:32] Now if you think about it now when we [7:34] operate network at such a scale things [7:37] will go wrong network is there will be [7:39] link flaps there'll be failures [7:41] something or the other will happen at [7:42] this scale so what we do is the paths [7:45] here map to a fabric planes so on [7:48] failure the traffic transparently shifts [7:50] to alternate planes and we don't look [7:52] require any application involvement why [7:54] we do that how we do that is because the [7:56] path each part tracks its own RTD its [7:58] own congestion state it knows enough [8:00] about the state of what's going on. So [8:03] we have a way of moving things around. [8:05] What it really comes down to is one [8:07] failure, one path's failure doesn't drag [8:09] somebody else's down. It only is [8:11] isolated to the path. This gives us a [8:13] great way to resil for resiliency and [8:15] scale also. [8:17] So essentially what it comes down to is [8:19] we have an ability to dynamically [8:21] allocate paths as time as as we see see [8:25] the protocol adapts in real time. [8:28] Now let me tie how we take congestion [8:31] control and reliability everything put [8:33] together. Our congestion control is [8:35] based on senderdriven condition but it's [8:38] on a per path granity. And this is where [8:39] we cannot differ from many other [8:41] implementations here. What this actually [8:43] gives us is an Eon decrease on one path [8:47] only affects a congested path. It [8:49] doesn't drag anybody else's down. So one [8:51] path punishes only itself. It doesn't [8:53] punish anybody else. Now the total [8:56] window that we expose to the QAR or the [8:58] application is a sum of all the parts. [9:00] What effectively gives us is we can add [9:03] paths to increase capacity but does not [9:06] add aggressiveness. That's one thing. [9:08] Now in addition there's one other [9:10] feature that we have is the receiver [9:12] sends rate hints back to the sender [9:15] through the act. What receiver says is [9:17] how many senders are coming to any point [9:19] of time. It computes a fair share and [9:21] distributes the bandwidth across all the [9:23] senders. It is a way for us to manage [9:25] incast. Now in this approach the [9:28] receiver since it's an endpoint it is [9:30] very much application aware or workload [9:32] aware. So it has a mechanism to manage [9:34] incast without switches getting in the [9:36] way. One thing to note all of this runs [9:39] on commodity Ethernet switches. EC and [9:41] ECMP is what what we look for. We don't [9:44] need anything more from the fabric. [9:48] Uh now I want to touch upon how we scale [9:51] and operationalize this and this is the [9:53] one of the key priority for meta and I [9:56] think this is where metari fundamentally [9:57] changed the game for the application. [9:59] Today application scales bandwidth by [10:02] putting more and more q pair. [10:04] Essentially what it does is chunks a [10:06] message takes a message start chunking [10:08] and give it to different qads and they [10:10] put more and more q pair as they go [10:11] along. Now what happens in that world is [10:14] when you start putting more and more QR [10:16] every QR runs in an absolutely [10:17] uncoordinated uncoordinated congestion [10:20] control they end up stopping onto each [10:22] other and that's a chart that's kind of [10:23] showing up here what happens as you put [10:25] more and more QA now this will break at [10:28] scale and we don't start putting more [10:31] and more QA we avoid the Q proliferation [10:34] but instead we go down with a uh a [10:36] concept of adding multiple paths so if [10:39] we need more bandwidth we add more paths [10:41] And but these parts allow us to scale [10:44] bandwidth also allow us to work on a one [10:47] single coordinator congestion control [10:48] and smoons the entire traffic profile [10:50] that we see and this is a very very much [10:53] different uh concept that we built up [10:54] over here. [10:57] Now with this I want to know look at [11:00] what what we are going to so our goal is [11:02] very straightforward. We want to get [11:04] GPUs running on heterogeneous network [11:06] infrastructure. We are not looking for [11:08] fragmentation and we do not want to [11:10] build specialized hardware that needs to [11:13] be customized on a per workload on a per [11:15] vendor basis. So as a result meta is [11:17] opening the protocol to the industry. [11:20] Now the switch ecosystem lies slowly on [11:22] standard features EC and ECMP is all we [11:24] require out of this this no vendor [11:27] locking no proprietary extension but [11:29] openness with our inter interoperability [11:32] is just a spec. Now for this to work we [11:35] need an ecosystem where we can build [11:36] compliance and prove a way that [11:38] different implementations can talk to [11:40] each other bit by bit. Now in that [11:43] regard it is critical that we maintain a [11:45] compliance suite and I want to invite [11:47] Kingsuk from Kite to talk about their [11:50] offering and what the compliance looks [11:52] like. Thank you Arvind. I'm going to [11:55] talk about uh compliance and [11:56] interoperability. [11:58] A protocol specification becomes an [12:00] interoperable system when multiple [12:02] vendors agrees on on the wear. Until [12:05] then, one implementation talking to [12:07] itself is a private implementation and [12:10] confirmance enforces that agreement [12:12] across across vendors. Now, for a brand [12:15] new transport like Metari, specification [12:17] is the first product. Specification is [12:20] written words and the confirmance [12:22] converts this written words into [12:24] testable assertion. And writing text [12:26] early exposes ambiguity while it is [12:29] still cheap to fix. Because there is one [12:31] thing none of us in this room can undo [12:33] and that is hardware is brutally [12:35] expensive to patch. Any ambiguity that [12:38] we leave in the specification while the [12:40] silicon freezes ships with the silicon [12:43] and that's precisely the reason we test [12:45] metari from day one and we developed a [12:48] test plan exactly for that purpose. So [12:50] let me share that test plan with you. [12:54] So what is it? It is a confirmment test [12:56] plan for Metaraki version 1.0. There are [12:59] four thing worth your attention here. It [13:01] firstly it is first of its kind written [13:04] from scratch specifically for Metari. [13:07] Secondly is proven not just specified. [13:10] Every test of this test plan is [13:12] validated against a reference [13:13] implementation. It is running code not [13:16] just paper. The pass fell bar here is [13:19] real. And next is about the [13:21] comprehensive coverage. We cover [13:23] hardware header integrity, reliability, [13:25] congestion, etc. the entire domain. And [13:28] then finally, this test plan explores [13:30] every path not just the happy one. It [13:33] covers the corner areas like knacks, the [13:35] timers, the recovery corners, but the [13:37] real issues hides for long. [13:40] Now before I share a couple of uh case [13:43] studies, there is one thing to [13:45] understand is this. How do we test? [13:48] Testing in this environment is a [13:49] continuum. It is the same test suite, [13:52] the same test cases, the same workflow [13:54] pointed to whatever implementation that [13:56] speaks Metari. In the early days, it [13:59] could be the software reference [14:00] implementation. As the architecture [14:02] matures, it is RTL in emulation and then [14:05] finally the silicon itself gets weighted [14:09] by the same protocol level test case run [14:11] unchanged at every stage. They are [14:14] evaluated against a single pass fail [14:16] pass fail bar. And here is the whole [14:19] merit. This quality bar never moves. You [14:22] find an issue in the software reference [14:24] implementation, the fix is code change. [14:26] You find exactly the same issue after [14:29] tap out. It's going to cost you a [14:31] respin. So in continuum there is no [14:34] compromise during handover from [14:36] architecture to RTL to the silicon all [14:39] evaluated against an uniform postfell [14:41] bar so that bugs do not get sleeping [14:43] during handover and meta wanted this [14:47] confirmance from day one. Deciding to [14:50] get tested so early was a very mature [14:52] decision and that's the reason we are [14:54] here to present you a test plan not a [14:57] postmodm report. So with that let me [14:59] share a couple of couple of case studies [15:02] and the reason this is worth your time [15:04] because our tool ksite AI confirmance [15:06] kai confirmance can drive execution on [15:09] the happy path as well as the unhappy [15:11] path. In the picture at the top we have [15:14] the tool playing the role of the [15:15] receiver and the bottom we have a metari [15:17] nick as the device under test acting as [15:20] the sender. Now let me share a real life [15:23] uh real life use case. If you are doing [15:25] large language model model training then [15:28] you might have heard about mixture of [15:30] expert workloads. The work is [15:32] distributed across many many machines [15:35] and at some point every machine has to [15:37] talk to every other machine all at once. [15:41] In such scenario, some machine may [15:43] temporarily run out of room to receive [15:45] additional data. And that's not a fault [15:47] at all. That's not even an error. The [15:50] machine is just not ready at this point [15:52] at that moment of time. And that's known [15:54] as RNR. Receiver not ready. In a RNR [15:57] situation, the receiver politely let the [15:59] sender know, hey, I am not ready right [16:02] now. Please retry after some time. In [16:04] this picture, the duty at the bottom [16:06] opens up one QP and then utilizes [16:09] multiple tags to to initiate multiple [16:11] flows towards the receiver. Through RNR, [16:14] the receiver says, "I am not ready for [16:16] one particular tag." Met specification [16:20] is very precise that the device must [16:22] pause only the impacted tag and the [16:25] associated message whereas all other [16:27] tags on the same QP should must keep on [16:30] functioning normally. And in our test we [16:33] found that device under test really [16:35] paused the impacted tag but it forgot to [16:38] wake up after the timeout and that [16:40] particular flow sat there forever. [16:43] Now mixture of expert is just an one [16:45] example trigger that can create RNR. [16:47] There are other workload types that can [16:50] deliver create the similar situation as [16:52] well. And with a confirance tool in your [16:54] hand you can recreate such scenario on [16:56] demand whenever you want. [17:00] Moving to case study two. The previous [17:03] one was induced by your workload and [17:05] here your network does this to you. In [17:08] the picture we have the tool and the [17:10] device under test and there are multiple [17:12] paths between them. Remember Arvind [17:14] talked about metari sprays packets over [17:16] multiple path and it tracks every path [17:19] individually. [17:20] One thing Metari tracks for each and [17:22] every path is RTO or the retransmission [17:25] timeout. In simple terms, RTO is used by [17:28] the sender to determine a packet has [17:30] been dropped and it is time to [17:32] retransmit. You mentioned too small an [17:35] RTO, you are going to flood the network [17:37] with too so many duplicates that it's [17:39] going to bring down your goodput. You [17:42] meant RTO to maintain RTO to be extra [17:44] large. Your flows are going to stall [17:47] there for longer and that's going to [17:49] impact your job completion time. Metali [17:52] specification is very explicit that each [17:55] path must maintain their independent RTO [17:58] and this is the single most tempting [18:01] shortcut in the whole design to adapt [18:03] one RTO for the entire QP. [18:06] One RTO for the entire QP is shared [18:08] state. It is it is lot less memory. It [18:11] lo lot lot less code and it is genuinely [18:14] simpler to simpler to implement but [18:17] wrong. So we decided to test it as shown [18:20] in the there in the picture. There are [18:22] multiple paths between the duty and the [18:24] tool. Path zero is intentionally [18:26] maintained as a very high-speed path and [18:28] the fast and healthy path with an [18:30] expectation of very low RTO. Path one is [18:33] deliberately created as congested with [18:35] an expectation of very high RTO and then [18:38] we measure these paths individually. [18:40] In a correct nick implementation, path [18:42] zero will settle down to floor and the [18:44] path one will have very high RTO to [18:47] absorb its own delay and that's a test [18:49] pass. Has there been a shared RTO for [18:52] the entire QP then it is the slowest [18:54] path that is going to drag down all the [18:56] good one. One unhealthy path is going to [18:59] poison the entire good set and the [19:01] moment that happened you have already [19:03] thrown away multipath. You have paid for [19:06] eight individual lens but you are [19:08] driving all of them at the speed of the [19:11] one that is jammed. And the reason this [19:14] is an important confirmance case for you [19:16] because in your lab environment every [19:18] path will be healthy and they will [19:20] barely diverge. [19:22] However, in real life deployment this [19:24] will appear as a mysterious latency [19:26] coupling multiple paths under load and [19:29] issues of this nature even take weeks to [19:32] see. [19:34] Now I will conclude with why this is the [19:36] way forward. This is the way forward for [19:39] all three of you in the room. A brand [19:41] new trans transport technology like [19:43] Metari is not something that you test at [19:46] the end. The protocol must be right from [19:48] day one because any ambiguity that you [19:51] leave in the specification while the [19:52] silicon freezes ships with the die and [19:55] hardware is brutally expensive to patch. [19:58] So the confirance pays in three ways at [20:00] once. If you are designing the protocol [20:02] or the implementing the protocol, you [20:04] discover the ambiguities early while [20:06] your silicon is still editable and the [20:09] spec hardens bit for bit. If you are [20:11] creating a network, then you get a [20:13] multipath that work gracefully with [20:15] proper separations and recovery. All [20:18] proven under fault injection, not just [20:21] on the happy path. And if you are [20:23] operating fabrics, then the issues that [20:25] appears only under real workloads get [20:28] discovered early. At the end, a single [20:32] implementation of a protocol is a [20:34] private implementation. Confirmance from [20:36] day zero turns a draft into an vibrant [20:39] ecosystem. And it is brutally expensive [20:43] to patch a frozen die. With that, I'll [20:46] pass it back to Arvind to talk about the [20:48] future and the next step. [20:50] >> Thank you, Kingshuk, on the introduction [20:51] of uh compliance suite. So, let me wrap [20:55] up from where we stand today and where [20:57] we are headed today. Tomorrow network [20:59] scale is unprecedented. We are building [21:01] million JPU clusters and we are truly [21:03] solving the trillion wire connectivity [21:05] across millions of access. That's the [21:06] problem space where we are in. Metari [21:09] delivers the AI workload demand peak [21:11] performance high availability on [21:13] commodity Ethernet no special switches [21:15] needed no vendor locking. So we are [21:17] opening this to the industry and [21:18] enabling the nick ecosystem to implement [21:20] it. Uh later in the day sep and bali [21:22] will come and talk about their uh how [21:24] they are implementing on AMD platform. [21:26] But one thing to call out is we are not [21:28] done yet. The challenges goes beyond [21:30] scale out. Scale up within the rack and [21:33] a front-end network across regions are [21:34] also something we are actively working [21:36] on. Every layer of the AI network stack [21:39] demands a similar rethinking. So stay [21:41] tuned and thank you all.