Meta, AMD Open-Source MetaRoCE Transport Protocol to Eliminate PFC

Meta与AMD开源MetaRoCE传输协议以消除PFC Meta与AMD正向开放计算项目（OCP）发布MetaRoCE，旨在通过一种由接收端驱动的NIC协议取代无损以太网与优先级流控（PFC），防止高丢包情况下的任务失败。 Meta与AMD推出了MetaRoCE，这是一种旨在通过将传输智能直接移至网卡（NIC）来消除AI集群网络瓶颈的高性能RDMA传输协议。通过将以太网视为本质上的有损网络架构，MetaRoCE利用拥塞感知多路径传输、接收端驱动的流量调度以及细粒度选择性确认（SACK），取代了易出问题的优先级流控（PFC）。本次演讲详细阐述了其从软件参考模型到在AMD P4可编程NIC上完成硬件原型的开发流程，展示了在多平面网络架构、高RTT跨集群扩展网络以及严重丢包环境下的弹性线速扩展能力。Meta在结尾宣布在OCP全球峰会上全面开源MetaRoCE规范、参考代码及合规性工具。 Meta 联合 AMD 推出专为大规模 AI 数据中心设计的 MetaRoCE 传输协议，将网络智能完全下沉至边缘主机与网卡（NIC），使以太网无需依赖复杂的网络交换矩阵即可具备路径感知与弹性能力。 (0:03) MetaRoCE 基于 AMD P4 可编程网卡平台原型实现，通过 P4-DMA 流水线处理动态拥塞控制、多路径选择、SACK 重传及多平面路由，摆脱了漫长 ASIC 研发周期对协议验证的制约。 (0:43) 协议引入了接收端驱动的拥塞提示以防御 Incast 崩溃，并支持细粒度 SACK 重传；在 1% 丢包环境下仍能确保工作负载收敛，而传统的 RoCEv2 Go-Back-N 机制则会出现崩溃。 (11:00) 在 8×800 Gbps 多平面网络测试中，MetaRoCE 彻底消除了易引发死锁与队头阻塞的 PFC 流控，支持亚秒级带内故障检测，并在 10 秒内恢复至全线速运行。 (16:24) Meta 宣布将在 OCP 全球峰会上全面开源 MetaRoCE 规范、用户空间软件参考实现及一致性测试套件，携手行业伙伴共同推动技术落地。 (18:44)

Meta and AMD presented MetaRoCE, a custom RDMA transport protocol engineered to eliminate AI cluster networking bottlenecks by moving transport intelligence from the fabric core onto programmable edge NICs. According to presenters Sandeep Nagaraj and Balakrishnan Raman, the protocol replaces brittle hop-by-hop flow control with fine-grained edge rate pacing, path spraying, and selective retransmissions across lossy Ethernet. Pushing Intelligence to the Network Edge (0:03) Sandeep Nagaraj explained that standard AI fabrics suffer from severe throughput bottlenecks when relying on complex, centralized network-level routing. To solve this, Meta designed MetaRoCE to push path awareness, resilience, and congestion logic directly to network interface cards (NICs) and end hosts. To establish an authoritative implementation without waiting for custom silicon, Meta developed a socket-based software reference model in user space on standard Linux CPUs and commodity NICs. This implementation functioned as a canonical golden model for third-party hardware validation, operating inside a continuous feedback loop alongside automated compliance suites and empirical benchmark tools. Programmable Prototyping on AMD Architecture (0:43) To accelerate real-world testing without undergoing multi-year ASIC manufacturing cycles, Meta partnered with AMD to prototype MetaRoCE on AMD's Pensando P4-programmable NIC platform. Balakrishnan Raman detailed how the hardware maps protocol tasks into discrete pipelines: programmable ingress parsers and egress deparsers handle evolving header formats, while the P4-Net pipeline enforces low-level network checks and inline security, such as PSP encryption. The NIC's P4-DMA pipeline serves as the central transport engine, executing dynamic congestion control, selective acknowledgments, multiplane routing, and flexible host memory interaction across programmable PCIe descriptors. Congestion Multipathing and Incast Mitigation (11:00) Rather than relying on unmanaged packet spraying, MetaRoCE maintains explicit per-path congestion windows on every Queue Pair (QP) informed by incoming Explicit Congestion Notifications (ECN). In asymmetrical test topologies where link bandwidth degraded from 400 Gbps to 100 Gbps, the protocol dynamically throttled transmission over degraded paths while fully loading available healthy routes. To support high-latency "scale-across" traffic between distant data center buildings—where round-trip times are 50× to 100× higher than intra-facility links—MetaRoCE dynamically scales from 12 paths up to 30–50 paths per QP to saturate aggregate line rate without increasing hardware window limits. For many-to-one fan-in incast scenarios, the receiving NIC monitors posted buffer allocations and sends explicit per-QP rate hints back to senders, throttling them before switch buffers drop packets. PFC Elimination and Loss-Tolerant Convergence (16:24) Validated inside Meta's labs on an 8 × 800 Gbps multiplane fabric, MetaRoCE sustained full line-rate performance across unidirectional and bidirectional flows. During simulated single-plane fabric failures, the protocol demonstrated subsecond in-band fault detection, rerouted traffic across healthy planes within 5 seconds, and resumed full line rate within 10 seconds of plane recovery without relying on out-of-band control plane intervention. Critically, MetaRoCE completely eliminates Ethernet Priority Flow Control (PFC) pause frames, which traditionally cause deadlocks and head-of-line blocking in RoCEv2. By treating Ethernet as an inherently lossy medium and using fine-grained Selective Acknowledgments (SACK), MetaRoCE matched zero-loss NCCL collective performance under 0.1% artificial packet drop rates and successfully converged training iterations under severe 1% packet loss, whereas RoCEv2 Go-Back-N failed. Meta announced that it will open-source the MetaRoCE architectural specification, software reference stack, and compliance test suites through the Open Compute Project (OCP) at the October OCP Global Summit. While lab results demonstrate resilience across 800 Gbps fabrics and high-loss environments, production success will depend on whether third-party hardware vendors can consistently replicate these edge-driven capabilities across diverse multi-tenant environments.

Meta与AMD在At-Scale网络大会上介绍了为大规模AI数据中心定制的RDMA传输协议MetaRoCE，其核心主张是将拥塞控制与故障恢复等智能逻辑从网络交换核心下沉至网卡等边缘设备，并彻底弃用传统RoCEv2依赖的优先流控制（PFC）。 边缘智能架构与迭代研发闭环 (0:03) Sandeep Nagaraj指出，MetaRoCE的设计理念是将路径感知、高弹性与高性能逻辑直接推移至网卡（NIC）和终端主机边缘，减少对复杂网络交换织网核心智能的依赖。为了快速实现从理论构想到生产级部署的跨越，Meta构建了一套迭代研发闭环：从规范设计出发，历经软件参考实现、硬件原型开发与实测验证，最终将测试反馈持续反哺并修正规范。 在这一流程中，Meta首先在用户空间编写了基于标准Linux操作系统、通用CPU和商用网卡的套接字软件参考实现，完全不依赖专有内核模块。该参考实现作为MetaRoCE的权威基准模型（golden model），不仅确立了协议在各种网络场景下的标准行为，还配套提供了合规性测试套件与性能基准工具，成为指导第三方硬件厂商研发与自测的标准依据。 基于可编程网卡的硬件原型验证 (0:43) 从纯软件模型直接跃进到专用ASIC硬件通常需要数月乃至数年的设计周期。为此，Meta与AMD合作，基于AMD的Pensando架构P4可编程AI网卡平台搭建原型。AMD工程师Balakrishnan Raman解释称，该平台的灵活性与硬件调试能力支持团队在真实的AI计算与集合通信负载下评估快速演进中的协议。 硬件架构层面，可编程解析器（Parser）与逆解析器（Deparser）支持数据包头的动态提取与重新封装，以适应MetaRoCE规范的迭代更新。网卡内置的P4流水线分为P4-Net和P4-DMA两部分：前者负责DMA引擎与上行端口之间的底层报文逻辑及PSP内联加密；后者作为MetaRoCE核心协议引擎，负责执行多路径调度、动态拥塞控制、选择性确认（SACK）与多平面路由。配合可自定义读写主机内存描述符的PCIe接口与集成硬件队列，网卡能实现全链路QoS调度。 拥塞感知多路径与丢包容忍机制 (11:00) Balakrishnan Raman通过技术案例展示了MetaRoCE的具体运行机制。在多路径传输方面，MetaRoCE摒弃了无差别的报文喷洒，而是为每个队列对（QP）在各路径上独立维护拥塞窗口（cwnd）。在一条链路降级至100 Gbps、另一条保持400 Gbps的非对称测试拓扑中，网卡根据收到的逐路径显式拥塞通知（ECN）动态限制受阻路径发包，充分利用健康路径以缩短流完成时间（FCT）。对于集群内部（RTT约为5至15微秒）与跨楼宇（RTT高出50至100倍）传输共存的场景，MetaRoCE保持单路径窗口上限不变，而是动态将每个QP的路径数从12条扩展至30至50条，将总窗口提升50至100倍以跑满高延迟长途带宽。 针对Incast流量突发导致的交换机缓冲区溢出，MetaRoCE引入了接收端驱动的流量调度，由接收网卡监控已分配缓冲区的活跃QP并计算明确的速率指示，使各发送端在源头执行QP级限速。此外，针对NCCL的all-to-all和all-reduce集合通信评测表明，传统RoCEv2因采用回退N步（Go-Back-N）重传，在丢包时重传整个窗口会导致吞吐量骤降乃至任务停滞；而MetaRoCE依靠精细的SACK机制，在0.1%丢包率下性能几乎与零丢包基准持平，即使在1%的严重丢包条件下仍能保障计算任务顺利收敛。 多平面织网验证与彻底消除PFC (16:24) Sandeep Nagaraj展示了Meta实验室在$8 \times 800\text{ Gbps}$多平面隔离织网架构下的验证数据。测试表明，在小包至大包、单向至双向的全流量测试中，MetaRoCE均在各平面实现了全线速传输。在模拟单平面与远端节点完全失效的容灾测试中，协议内置的边缘逻辑可在1秒内完成故障检测，5秒内将流量平滑重路由至健康平面，并在故障平面恢复后10秒内重新恢复满线速运行，全程完全依赖协议带内检测，无需带外信令或控制平面介入。 Sandeep Nagaraj强调，MetaRoCE相比传统RoCEv2的关键运维优势在于彻底消除了以太网优先流控制（PFC）。RoCEv2依赖逐跳PFC暂停帧确保无损传输，但在多租户AI集群中极易引发死锁与队头阻塞（HoL blocking）。MetaRoCE将以太网视作固有有损网络，利用接收端限速、路径动态喷洒和SACK重传替代PFC暂停机制，不仅消除了队头阻塞和缓冲区积压，还大幅加速了超大规模分布式AI集群中的NCCL集合通信效率。 Meta宣布计划于10月的开放计算项目（OCP）全球峰会上正式开源MetaRoCE协议规范、软件参考模型及合规测试套件，并进行实时现场演示。演讲者呼吁产业界共同协作开发，但该协议在更多异构硬件生态与大规模生产集群中的实际适配度仍有待开源落地后检验。

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布，记录了在圣克拉拉会议中心（SCCC）举办的技术分享。Meta 工程师 Sandeep Nagaraj 与 AMD 架构师 Balakrishnan Raman 共同登台，介绍了针对超大规模 AI 数据中心自研的定制化传输协议 MetaRoCE 从规范制定、AMD 可编程网卡硬件落地到实测验证并计划在 OCP 全面开源的全过程。 ## TL;DR MetaRoCE 将传输协议智能全面下沉至边缘网卡，把以太网视作天然有损网络，通过软件参考实现与 AMD P4 可编程网卡构建了高效的闭环迭代验证体系。网卡硬件依托 P4 DMA 流水线实现了拥塞感知多路径喷洒、自适应 SACK 重传、逐路径独立拥塞窗口以及接收端驱动的 Incast 速率控制，兼顾了集群内低延迟 Scale-Out 与广域跨园区 Scale-Across。实测表明，协议在 1% 丢包率下仍能保证集合通信收敛，在 8x400G 多平面网络中实现亚秒级故障感知与自愈，并彻底摒弃了易导致死锁的 PFC 机制。最后，Meta 宣布将在 OCP 全球峰会上全面开源 MetaRoCE 规范、用户态参考实现及合规测试套件。 ## Atmosphere & style 整场演讲呈现出典型顶级系统工程大会的扎实、硬核与快节奏，两位讲者配合默契，分别从 Meta 的协议顶层设计与 AMD 的硅片流水线实现两个互补视角层层拆解。技术叙事严密流畅，从架构动机、硬件解构直切真实的实测波形与丢包压力数据，没有虚头巴脑的宏大叙事，全是工业界破局极端网络瓶颈的实战干货。结尾公布向 OCP 全面开源的计划时，语气从严谨的工程剖析转为开放振奋的生态号召，令人深切感受到超大规模数据中心网络走向开放标准的时代脉动。 ## Bulletin - [00:03](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=3s) Meta 与 AMD 携手推出 MetaRoCE 定制 RDMA 协议，将全部调度与传输智能下沉至可编程网卡边缘，并成功在 AMD P4 AI NIC 上实现生产级落地。 - [15:00](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=900s) MetaRoCE 原生引入 SACK 重传以替代传统 RoCEv2 的 Go-Back-N 机制，在 0.1% 丢包率下性能近乎无损，面对 1% 极端恶劣丢包仍能保证 AI 集合通信作业平稳收敛。 - [12:11](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=731s) 协议通过固定单路径窗口并动态引导 30-50 条路径的方式，弥补跨建筑高延迟对带宽时延积的需求，成功统一了低时延 Scale-Out 与远距离 Scale-Across 架构。 - [18:36](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=1116s) Meta 宣布将在 10 月的 OCP 全球峰会上全面开源 MetaRoCE 协议规范、基于用户态 Socket 的软件参考实现以及自动化合规测试套件。 ## Key takeaways - [00:03](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=3s) MetaRoCE 核心设计理念是将网络智能全面推向 NIC 边缘，将以太网底层视作有损网络，摆脱了对复杂交换机 Fabric 特性的重度依赖。 - [01:24](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=84s) 团队采用“规范定义—软件参考实现—硬件落地—合规测试”的闭环迭代开发模式，快速打磨自适应拥塞控制与传输算法。 - [03:16](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=196s) 纯用户态 Socket 软件参考实现充当协议的“黄金标准模型”，不仅与私有驱动解耦，还用于指导和验证第三方硬件实现的合规性。 - [04:13](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=253s) 面对传统 ASIC 芯片流片周期长与 AI 工作负载快速演进的矛盾，Meta 选择在 AMD P4 可编程 AI 网卡上敏捷落地 MetaRoCE 数据平面。 - [05:05](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=305s) AMD P4 网卡硬件内部通过可编程 Parser/Deparser、P4 网络流水线、报文缓冲区及 P4 DMA 流水线协同承载 MetaRoCE 传输逻辑。 - [11:13](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=673s) 协议按每条路径独立维护拥塞窗口（CWND）并监控 ECN，有效杜绝热点路径拥塞，即便在 100G 与 400G 异构非对称链路下也能压榨全路径带宽。 - [07:43](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=463s) MetaRoCE 结合启发式逻辑百分之百区分“报文级动态负载均衡引起的乱序”与“物理丢包”，避免不必要的降速与伪重传。 - [12:11](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=731s) 针对集群内 5-15 微秒 RTT 与跨集群相差 50-100 倍的高延迟场景，软件层动态扩展至 30-50 条路径以撑大在途数据量，轻松跑满远程线速。 - [13:53](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=833s) 引入接收端驱动的拥塞通知机制，通过实时跟踪挂载缓冲区的活跃 QP 负荷并下发速率提示（Rate Hint），从源头遏制 Incast 冲击。 - [15:00](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=900s) 在 AMD RCCL 集合通信测试中，SACK 机制彻底终结了 Go-Back-N 的雪崩效应，即便在 1% 丢包率下也能杜绝作业崩溃报错。 - [16:24](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=984s) 在 8x400G 多平面 FPF 实验网中，MetaRoCE 支持亚秒级带内故障检测，5 秒内优雅降级且对其他平面零干扰，故障消除后 10 秒内恢复线速。 - [17:48](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=1068s) MetaRoCE 彻底摒弃 PFC 优先级流控，从根本上消除了级联拥塞扩散、死锁与队头阻塞（HoL）风险，大幅压低了交换机缓冲区占用。 - [18:36](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=1116s) Meta 决定在 OCP 峰会全面开源协议规范、软件参考栈与合规套件，邀请全行业芯片厂商共同构建开放的 AI 传输生态。 ## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=3s) 开场介绍：MetaRoCE 的演进旅程与设计理念 来自 Meta 数据中心网络架构团队的 Sandeep Nagaraj 携手 AMD 的 Balakrishnan Raman（网络圈内常亲切称其为“Balki”）共同登台，开启了关于 MetaRoCE 的深度分享。MetaRoCE 是 Meta 针对超大规模 AI 数据中心自主研发的定制化 RDMA 传输协议。与传统重度依赖网络 Fabric 底层交换机智能的方案不同，MetaRoCE 的核心架构理念是将全部智能下沉到网络边缘（网络接口卡/NIC），使边缘网卡具备更强的路径感知能力、拥塞自适应能力与弹性，从而在普通商用以太网之上实现顶级的性能、容灾韧性与超大规模运维能力。在此之前，团队成员 Arvind 与 Kingshuk 已在前面的演讲中拆解了 MetaRoCE 的协议规范、算法与理论设计；而本场分享的主题则聚焦于“从规范落地到硅片硬件”的真实工程历程、实测验证数据，以及最终走向 OCP 全面开源的宏伟计划。 ### [01:24](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=84s) 从规范到硬件：紧密迭代的“闭环开发循环” 在确立了初步的协议规范之后，Meta 团队并未直接将其固化，而是建立了一套在规范定义、原型实现与实际测试之间快速循环迭代的紧密反馈环。整个工程演进的第一步是构建一套软件参考实现（Software Reference Implementation）。这套软件栈充当了整个协议的“权威真理之源（Canonical Source of Truth）”，能够对“在特定网络极端场景下 MetaRoCE 究竟应当展现何种行为”给出确切的标准答案。在此基础之上，团队迅速将协议推向真实硬件实现，从而能够在真实的以太网 Fabric、超高网络带宽、GPU 集合通信算子（Collective Operations）以及生产级 AI 工作负载下开展全规模实测。 单纯编写协议实现是远远不够的，团队还同步研发了全套合规测试套件（Compliance Test Suites）与性能度量工具。这些工具源源不断地提供关键的经验数据，帮助工程师客观评估究竟是硬件实现存在偏差，还是协议规范本身需要进一步修正。通过“定义规范—实现落地—测试验证—修正规范”的周而复始，Meta 攻克了诸多极端挑战。例如，团队成功让 MetaRoCE 在拥塞控制和可靠性传输算法中动态调整控制参数与熵值（Entropy），使其既能在单集群内高效 Scale-out，又具备跨长距离网络传输的能力，从而解锁了长途广域 RDMA（Long-Distance RDMA）。这种软件规范与硬件实现并进的高频迭代能力，成为了整个项目破局的关键。 ```mermaid flowchart LR accTitle: MetaRoCE 协议开发与验证闭环迭代流程 accDescr: 展示从规范定义、软件参考实现、硬件实现到合规测试的紧密闭环反馈机制 subgraph Design["协议规范与软件参考"] A["MetaRoCE 协议规范"] --> B["软件参考实现<br/>(权威真理模型)"] end subgraph Hardware["硬件落地与验证"] B --> C["AMD P4 AI NIC<br/>硬件原型验证"] C --> D["合规测试套件 &<br/>实测性能分析"] end D -- "经验反馈 / 参数调优" --> A ``` ### [03:16](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=196s) 软件参考实现：纯用户态的“黄金模型” Sandeep 详细拆解了 MetaRoCE 软件参考实现的系统架构。该实现完整覆盖了 MetaRoCE 的发送（Transmit）与接收（Receive）机制，原生集成了拥塞控制、端到端可靠性重传、用于拓扑扩展的多路径元数据标签等全部核心特性。在软件接口上，它无缝封装并兼容标准 RDMA 动词（Verbs）与所有标准语义调用。更重要的是，它被设计为一个基于 Socket 的纯用户空间（User-Space）实现： - **零专有依赖**：无需绑定任何私有硬件驱动栈，也不依赖特定的内核模块，能够直接在任何通用商用 CPU 和普通网卡上顺畅运行。 - **产业级黄金模型**：该软件栈与配套的合规测试套件相结合，成为了验证任何 MetaRoCE 硬件网卡实现的“黄金标准模型（Golden Model）”。 - **生态赋能**：该参考实现未来将开源，旨在直接指导并赋能整个网卡开发者社区，帮助业界芯片厂商快速验证自家硬件对 MetaRoCE 的兼容性与合规性。 ### [04:13](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=253s) 跨越落地鸿沟：为何选择 AMD P4 可编程网卡 从纯软件模型迈向真实硬件落地的过程中，工程团队遭遇了巨大的现实挑战。如果遵循传统的硬件开发路径，从头流片研发一款定制 ASIC 芯片网卡来实现 MetaRoCE，整个周期动辄需要数月甚至数年之久，根本无法追赶 AI 工作负载快速演进的步伐。团队亟需一种兼具极高原型开发速度、极致灵活性且易于现场调试的平台，以便能够清晰区分到底是网卡固件实现需要打磨，还是协议规范本身需要调整。更重要的是，这绝非一次停留于实验室象牙塔内的概念验证（PoC），它必须具备直接部署于数据中心生产环境的就绪度。 基于上述严苛指标，Meta 最终选择与 AMD 展开深度战略合作，将 MetaRoCE 全面实现在 AMD 的 P4 可编程 AI 网卡（P4 Programmable AI NIC）之上。AMD 网卡所具备的高性能 P4 可编程数据平面，赋予了团队随时演进协议所需的敏捷度与生产级线速吞吐能力。 ### [05:05](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=305s) AMD P4 可编程 AI 网卡硬件内部架构解构 AMD 架构师 Balakrishnan（Balki）随后深入剖析了网卡内部支撑 MetaRoCE 协议实现的关键硬件功能模块： - **可编程解析器与反解析器（Programmable Parser & Deparser）**： 在协议快速演进期，报文头部格式与字段会频繁变更、扩充或修改。可编程 Deparser 允许在发送端灵活封装各种新型报头与定制字段；而在接收端，可编程 Parser 则能快速适配全新的报文格式，精准提取关键元数据并投递给后续流水线处理。 - **P4 可编程多核网络流水线（P4 Net Pipeline）**： 专职处理主机 DMA 流水线与网络上行物理端口（Uplink Port）之间的报文逻辑，承担上行端口收包时的安全合法性校验、端到端行内加密机制（如 PSP 协议硬件加解密）以及各类定制流控逻辑。 - **硬件报文缓冲区（Packet Buffer）**： 在流水线处理报文期间暂存数据载荷（Payload）。在收包方向，载荷暂存后经 DMA 写入主机内存；在发包方向，自内存提取并切片后的报文在缓冲区排队发送。该模块拥有深度的硬件队列支持，能够全面承载各种端到端服务质量（QoS）优先级映射。 - **P4 可编程 DMA 流水线（P4 DMA Pipeline）**： 此模块是整个网卡硬件的“大脑”，也是 MetaRoCE 传输协议逻辑的真正实现中枢。 ```mermaid flowchart TD accTitle: AMD P4 可编程 AI 网卡内部数据路径架构 accDescr: 描述主机内存、P4 DMA 流水线、报文缓冲区、P4 Net 流水线到上行物理端口的处理链路 subgraph Host["主机侧 (Host)"] H["主机内存<br/>(Host Memory)"] end subgraph Core["P4 AI 网卡硬件引擎"] DMA["P4 可编程 DMA 流水线<br/>(MetaRoCE 核心逻辑: 多路径 / SACK / 拥塞控制)"] BUF["硬件报文缓冲区<br/>(Packet Buffer / QoS 队列)"] NET["P4 网络流水线<br/>(Parser / Deparser / 安全检查 / PSP 加密)"] end subgraph Fabric["网络侧 (Network)"] UP["物理上行端口<br/>(Uplink Port)"] end H <--> DMA DMA <--> BUF BUF <--> NET NET <--> UP ``` ### [07:43](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=463s) 拥塞感知喷洒、SACK 乱序区分与多平面负载优化 Balki 详细拆解了运行在 P4 DMA 流水线中的四大 MetaRoCE 核心技术机制： 1. **拥塞感知多路径传输（Congestion-Aware Multipathing）**： MetaRoCE 并非在各个路径层级上进行“盲目的数据包随机喷洒（Blind Packet Spraying）”。协议栈在软件中为每条路径精细维护着独立的拥塞窗口（CWND）。网卡通过实时监控来自各个路径的 ECN（显式拥塞通知）标记，动态弹性调整拥塞窗口，精确限制每条网络路径上的最大在途报文数（Outstanding Packets），彻底杜绝热点路径拥塞劣化。 2. **完全可编程的拥塞控制框架**： 拥塞控制模块被抽象为一组完全可编程、可实时调谐参数的通用框架。网卡内部不仅实现了 MetaRoCE 内置的基线算法，还深度集成了基于接收端的拥塞通知（Receiver-based Congestion Notification）机制，允许网络架构师根据特定流量特征绑定自定义拥塞控制逻辑。 3. **区分“网络乱序”与“真实丢包”的自适应 SACK 重传**： 在典型数据中心网络中，多采用基于流熵（Flow Entropy）的 ECMP 哈希负载均衡；但当网络引入基于报文级的动态负载均衡（Dynamic Load Balancing, DLB）时，不可避免地会导致严重的报文乱序到达。MetaRoCE 在发送端与接收端均构建了精准的启发式逻辑，能够百分之百区分“DLB 引起的数据包乱序”与“链路真实的丢包”。网卡对乱序具有极高的容忍度，唯有在确认出现真实物理丢包时才会触发精确的选择性重传（Selective Retransmission）。 4. **多平面拓扑（Multiplane）与运维度量体系**： 在超大规模 AI 集群中，为保证可用性，GPU 通常跨越多个独立的网络平面（Planes）互联。MetaRoCE 能够保证跨所有平面的整体流量达到绝对均匀的负载均衡以最大化网络吞吐，同时在单个平面内部充分榨干所有可达路径以大幅压低流完成时间（Flow Completion Time, FCT）。此外，网卡内部通过纯软件方式内嵌了上下文感知的全套运维度量指标（Operational Metrics），无需重新烧录固件即可动态增删性能遥测计数器。 ### [10:57](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=657s) 实测案例研讨：多逻辑链路拓扑下的拥塞感知路径喷洒 在系统化阐述理论与架构之后，Balki 引导全场切入首个实战案例分析——测试 MetaRoCE 在高密度拥塞网络下的“多路径报文喷洒与拥塞感知自适应能力”。在此项实验评估中，测试团队将多个 GPU 计算集群通过多条逻辑物理链路（Logical Links）交叉互联至网络交换机群。在这种存在非对称链路状态与潜在局部瓶颈的拓扑中，测试重点验证 MetaRoCE 是否能精确避免拥塞路径并实现全带宽饱和。 ### [11:13](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=673s) 每路径独立拥塞窗口与动态多路径调度 在多路径网络中，各条物理链路的拥塞状态往往极不均衡。Balakrishnan 举例指出，在某一时刻，一条路径可能正经历网络拥塞，可用带宽缩减至 100 Gbps，而另一条路径却拥有高达 400 Gbps 的充裕带宽。MetaRoCE 针对这一挑战，引入了基于每条路径（Per-path）独立维护的拥塞窗口机制。在 MetaRoCE 的设计中，一个队列对（Queue Pair, QP）会跨越并映射到所有可用的网络路径，而无论各路径当前的可用带宽充裕与否。 通过在路径级别持续监控显式拥塞通知（ECN）信号，网卡能够独立调整该路径的拥塞窗口尺寸，精准限制在途报文（Outstanding Packets）数量。这种做法有效防止了流量在拥塞路径上发生丢包，同时又不会拖累畅通路径的传输效率。不论网络局部是否存在拥塞，协议都会竭尽全力压榨所有可用路径的吞吐潜力，从而在整体上最大程度地缩短流完成时间（Flow Completion Time, FCT）。 - **带宽异构适配**：在一条路径可用带宽为 100 Gbps、另一条为 400 Gbps 的非对称拥塞场景下，MetaRoCE 依然能并行分发流量。 - **基于 ECN 的窗口整形**：按路径独立采集 ECN，精准约束在途数据量，从源头避免缓冲区溢出丢包。 - **全局路径利用率**：无需将流量全部绕开拥塞路径，而是按路径承载能力自适应分配，实现 FCT 最小化。 --- ### [12:11](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=731s) 应对异构延迟：Scale-Out 与 Scale-Across 并行连接架构 随后探讨的案例深入到了现代 AI 算力中心最棘手的组网挑战之一：同时支撑集群内扩展（Scale-Out）与跨集群扩展（Scale-Across）。所谓 Scale-Out，是指同一栋数据中心建筑内 GPU 之间的互联，其往返时延（RTT）极低，通常仅为 5 至 15 微秒；而 Scale-Across 则是跨建筑、跨园区的远距离 GPU 互联，其物理距离导致 RTT 剧增，通常达到 Scale-Out 的 50 到 100 倍之多。 对于低延迟的 Scale-Out 连接，单个 QP 维持 72 个报文的拥塞窗口便足以打满带宽，这通常由单个 QP 分配在 12 条路径上共同提供。为了保证硬件与逻辑实现的简洁性，MetaRoCE 规定每条路径上的最大拥塞窗口尺寸是固定不变的。那么，面对 RTT 骤增数十至上百倍的 Scale-Across 连接，如何维持足够的在途报文以跑满带宽？Meta 团队在软件层面实现了路径动态引导（Dynamic Path Bootstrapping）机制： ```mermaid flowchart TD accTitle: Scale-Out 与 Scale-Across 动态路径分配机制 accDescr: 展示 MetaRoCE 如何通过固定单路径窗口大小，并动态增加跨数据中心路径数量来弥补高 RTT 带来的带宽时延积需求。 subgraph ScaleOut ["数据中心内 Scale-Out 连接 (RTT: 5~15 μs)"] A1["单个 QP (Queue Pair)"] --> A2["分配 12 条网络路径"] A2 --> A3["总拥塞窗口: ~72 个报文"] A3 --> A4["达到集群内线速"] end subgraph ScaleAcross ["跨数据中心 Scale-Across 连接 (RTT: 50~100x)"] B1["单个 QP (Queue Pair)"] --> B2["软件动态引导: 30~50 条网络路径"] B2 --> B3["总拥塞窗口成倍展开 (50~100x)"] B3 --> B4["维持更多在途报文，达到跨区线速"] end ``` 软件根据实测 RTT 的急剧增加，自动为该 QP 引导并激活更多的通信路径（例如扩展到 30 至 50 条路径）。通过横向堆叠更多的固定窗口路径，整个 QP 的并发拥塞窗口被放大到了原本的数十倍，从而在极长距离、高延迟的 Scale-Across 场景下依然能够轻松跑满网络线速（Line Rate）。 - **RTT 跨度巨大**：建筑内（Scale-Out）RTT 为 5–15 μs，而跨建筑（Scale-Across）RTT 增长 50–100 倍。 - **固定单路径窗口**：每条路径拥塞窗口上限固定，极大简化了硬件 NIC 的设计与实现复杂度。 - **动态路径引导**：面对高延迟连接，软件层动态分配 30–50 条额外路径，撑大在途数据量，确保远程传输打满线速。 --- ### [13:53](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=833s) 接收端驱动的 Incast 拥塞控制与速率提示 除网络链路中的拥塞外，多对一通信中的接收端拥塞（Incast 场景）也是分布式 AI 训练的经典痛点。当成百上千个发送端同时向同一个接收端推送数据时，接收端网卡的输入队列和主机内存缓冲很容易被瞬间打爆。针对这一问题，MetaRoCE 摒弃了传统的被动拥塞响应，采用了“接收端驱动拥塞通知”（Receiver-based Congestion Notification）机制。 在实现上，软件层构建了精细的监控逻辑：接收端会持续追踪当前处于活跃接收状态的 QP 数量，重点检测那些已经挂载了接收缓冲区（Receive Buffer Posted）的连接——这意味着海量数据包即将或正在涌入这些通道。基于各活跃 QP 的实时负荷，接收端主动计算并向发送端下发针对特定 QP 的速率提示（Rate Hint）。发送端收到提示后，在逐 QP 粒度上执行精确的速率限制（Rate Limiting），防止接收端被洪流淹没。这套机制还允许用户根据实际部署场景自定义评估逻辑，以灵活适配不同的上层 AI 工作负载。 - **活跃 QP 精准感知**：通过跟踪挂载了接收缓冲区的 QP，准确判断哪些连接正在承载激增流量。 - **逐 QP 速率提示**：接收端主动向各发送端反向提供限速建议，从源头上遏制 Incast 冲击。 - **高度可定制化**：软件栈提供开放接口，支持运营商根据特定集群形态和算力模式定制监测算法。 --- ### [15:00](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=900s) 选择性重传（SACK）与 RCCL 集合通信抗丢包实测 传统 RoCEv2 最为人诟病的缺陷之一便是基于 Go-Back-N 的回退重传机制：一旦链路上出现丢包，后续所有已经正确到达的报文都必须废弃并全数重传，造成灾难性的吞吐量断崖式暴跌和网络雪崩。针对这一痛点，MetaRoCE 原生引入了选择性确认（SACK）与选择性重传机制。Balakrishnan 展示了使用 AMD ROCm 通信集合库（RCCL）在丢包与无丢包环境下执行 All-to-All 和 All-Reduce 算子的对比评测结果。 测试数据显示，在千分之一（1/1000）的随机丢包率下，MetaRoCE 的吞吐表现柱状图几乎与完全无丢包（No Drop）的基准线持平，偶尔发生的网络丢包对整体性能的扰动微乎其微。即使将网络恶化到极端恶劣的百分之一（1/100）丢包率，尽管重传次数增多导致性能出现一定程度的下滑，但任务依然能够稳定推进并顺利收敛，集群计算进程完全没有中断。相比之下，传统的 RoCEv2 在面对 1% 丢包率时，Go-Back-N 机制会彻底陷入重传风暴，导致任务无法收敛直至整个计算任务直接报错崩溃。 - **0.1% 丢包近乎无损**：在 1/1000 丢包率下，集合通信性能与零丢包环境相比几乎没有明显衰退。 - **1% 极端丢包仍可收敛**：在 1/100 丢包率的严苛条件下，MetaRoCE 依旧保证任务收敛，彻底杜绝了任务异常失败。 - **彻底告别 Go-Back-N**：消除了大量无效报文的重复传输，显著节约了宝贵的网络与算力资源。 --- ### [16:24](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=984s) 8x400G Multiplane FPF 网络实测与高可用韧性 随后，Meta 工程师 Sandeep Nagaraj 登台，详细阐述了 MetaRoCE 的路径化与接收端驱动架构在 Meta 实验室环境下的系统性验证。为了验证协议在大规模多平面架构下的能力，团队构建了一套由相互隔离的独立平面（Disjoint Planes）组成的 8x400G 多平面 FPF（Fabric Plane Fabric）实验网络。多平面拓扑在网卡层面引入了额外的资源仲裁与拓扑感知层，这对 MetaRoCE 的多路径报文喷发和负载均衡逻辑提出了极高要求。 实测结果表明，在各种消息尺寸以及单向/双向传输流量测试中，MetaRoCE 均能够无缝跨所有可用路径均衡分发报文，达到物理线速。更引人注目的是其高可用容灾表现：工程团队在单平面内人工注入了远端节点故障（Remote Endpoint Failure）并模拟其随后的恢复过程。 ```mermaid flowchart LR accTitle: MetaRoCE 单平面故障自愈流程 accDescr: 展现 MetaRoCE 在 8x400G 多平面网络中进行带内故障检测、亚秒级响应、优雅降级和恢复线速的闭环过程。 A["注入单平面远端节点故障"] --> B["亚秒级 (<1 s)<br/>带内协议完成故障感知"] B --> C["5 秒以内<br/>平滑优雅降级<br/>(其余平面完全不受影响)"] C --> D["故障节点恢复"] D --> E["10 秒以内<br/>全平面重新打满物理线速"] ``` 在整个容灾过程中，故障在亚秒级（Subsecond）内便被协议感知，集群在 5 秒之内完成了流量的优雅降级；当故障排除后，系统在 10 秒之内重新回到了全线速运行状态。尤为关键的是，整个检测与恢复全部依靠协议带内（Inband）自洽完成，完全不需要任何带外（Out-of-band）网络信令干预，且其他正常运行的网络平面全程没有受到任何抖动或性能冲击。 - **全场景打满线速**：无论是单向还是双向、大包还是小包，多平面流量全部实现线速传输。 - **亚秒级带内感知**：故障检测用时低于 1 秒，优雅降级控制在 5 秒内，系统恢复至线速用时低于 10 秒。 - **零跨平面干扰**：单平面波动被完全限制在隔离域内，其余数据平面的生产流量毫无感知。 --- ### [17:48](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=1068s) 告别 PFC：根除队头阻塞与加速集合通信 Sandeep 进一步总结了 MetaRoCE 相比于传统 RoCEv2 的架构代差优势。得益于动态 Incast 治理与报文级多路径喷发，MetaRoCE 实现了更短的集合通信完成时间。更根本的变革在于它对以太网拥塞控制逻辑的重构：RoCEv2 深度依赖基于优先级的流量控制（Priority-based Flow Control, PFC）来构建无损以太网，而 PFC 极易引发死锁、拥塞扩散以及恶名昭彰的队头阻塞（Head-of-Line Blocking, HoL）。 MetaRoCE 用接收端驱动的精准流量调度机制彻底取代了 PFC。这一转变带来了双重收益：一方面将交换机内部的缓冲区占用率压缩到了极低水平；另一方面，彻底消除了困扰 RoCEv2 已久的队头阻塞现象。特别是在超大规模集群和多租户高并发工作负载场景下，MetaRoCE 展现出了远胜 RoCEv2 的稳定性与确定性，真正将以太网打造成了一个既容忍物理丢包、又具备极致性能的高性能 AI 底座。 - **摒弃 PFC 机制**：摆脱了对无损网络的严苛依赖，从根本上消除了级联拥塞扩散和死锁隐患。 - **消除队头阻塞**：在大规模集群高并发流量下，避免了慢流阻塞快流的顽疾。 - **轻量交换机缓冲**：极度压低网络设备内部的 Buffer 占用，使网络运行更加轻盈透明。 --- ### [18:36](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=1116s) 拥抱开源：MetaRoCE 规范、参考实现与 OCP 发布计划 在演讲的尾声，Sandeep 代表团队公布了激动人心的开源路线图。为了赋能整个网卡（NIC）生态圈和芯片厂商在各自的产品上实现并扩展 MetaRoCE，Meta 决定将该技术的全套核心资产向开源社区全面开放。 该计划将在即将于 10 月召开的开放计算项目全球峰会（OCP Global Summit）上正式落地。届时，Meta 将正式开源 MetaRoCE 规范全文、完整的参考软件实现，以及严格的合规性测试套件（Compliance Test Suites）。Sandeep 向全行业发出诚挚邀请，欢迎各界同行莅临 OCP 峰会现场，届时不仅会有 MetaRoCE 在真实硬件上运行的实时动态演示（Live Demo）和专题技术演讲，开发者还能与负责 MetaRoCE 研发的 Meta 核心工程团队进行面对面的深度技术交流与探讨。 - **开源交付物**：包括 MetaRoCE 官方规范、NIC 参考软件实现及完整的合规性自动化测试套件。 - **发布时间节点**：定于 10 月的 OCP 全球峰会正式公布。 - **现场互动**：峰会现场将提供端到端 Live Demo 演示，并安排 Meta 核心协议架构团队现场答疑。 ## Notable quotes > [00:03](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=3s) MetaRoCE 的核心架构理念是将全部智能下沉到网络边缘（网络接口卡/NIC），使边缘网卡具备更强的路径感知能力、拥塞自适应能力与弹性，从而在普通商用以太网之上实现顶级的性能、容灾韧性与超大规模运维能力。 — Sandeep Nagaraj > [01:24](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=84s) 这套软件栈充当了整个协议的“权威真理之源（Canonical Source of Truth）”，能够对“在特定网络极端场景下 MetaRoCE 究竟应当展现何种行为”给出确切的标准答案。 — Sandeep Nagaraj > [04:13](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=253s) 如果遵循传统的硬件开发路径，从头流片研发一款定制 ASIC 芯片网卡来实现 MetaRoCE，整个周期动辄需要数月甚至数年之久，根本无法追赶 AI 工作负载快速演进的步伐。 — Sandeep Nagaraj > [04:13](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=253s) AMD 网卡所具备的高性能 P4 可编程数据平面，赋予了团队随时演进协议所需的敏捷度与生产级线速吞吐能力。 — Sandeep Nagaraj > [07:43](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=463s) MetaRoCE 并非在各个路径层级上进行“盲目的数据包随机喷洒（Blind Packet Spraying）”。协议栈在软件中为每条路径精细维护着独立的拥塞窗口（CWND）。 — Balakrishnan Raman > [07:43](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=463s) MetaRoCE 在发送端与接收端均构建了精准的启发式逻辑，能够百分之百区分“DLB 引起的数据包乱序”与“链路真实的丢包”。网卡对乱序具有极高的容忍度，唯有在确认出现真实物理丢包时才会触发精确的选择性重传（Selective Retransmission）。 — Balakrishnan Raman > [11:13](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=673s) 通过在路径级别持续监控显式拥塞通知（ECN）信号，网卡能够独立调整该路径的拥塞窗口尺寸，精准限制在途报文（Outstanding Packets）数量。这种做法有效防止了流量在拥塞路径上发生丢包，同时又不会拖累畅通路径的传输效率。 — Balakrishnan Raman > [12:11](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=731s) 软件根据实测 RTT 的急剧增加，自动为该 QP 引导并激活更多的通信路径（例如扩展到 30 至 50 条路径）。通过横向堆叠更多的固定窗口路径，整个 QP 的并发拥塞窗口被放大到了原本的数十倍，从而在极长距离、高延迟的 Scale-Across 场景下依然能够轻松跑满网络线速（Line Rate）。 — Balakrishnan Raman > [13:53](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=833s) 基于各活跃 QP 的实时负荷，接收端主动计算并向发送端下发针对特定 QP 的速率提示（Rate Hint）。发送端收到提示后，在逐 QP 粒度上执行精确的速率限制（Rate Limiting），防止接收端被洪流淹没。 — Balakrishnan Raman > [15:00](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=900s) 即使将网络恶化到极端恶劣的百分之一（1/100）丢包率，尽管重传次数增多导致性能出现一定程度的下滑，但任务依然能够稳定推进并顺利收敛，集群计算进程完全没有中断。相比之下，传统的 RoCEv2 在面对 1% 丢包率时，Go-Back-N 机制会彻底陷入重传风暴，导致任务无法收敛直至整个计算任务直接报错崩溃。 — Balakrishnan Raman > [16:24](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=984s) 在整个容灾过程中，故障在亚秒级（Subsecond）内便被协议感知，集群在 5 秒之内完成了流量的优雅降级；当故障排除后，系统在 10 秒之内重新回到了全线速运行状态。尤为关键的是，整个检测与恢复全部依靠协议带内（Inband）自洽完成，完全不需要任何带外（Out-of-band）网络信令干预。 — Sandeep Nagaraj > [17:48](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=1068s) MetaRoCE 用接收端驱动的精准流量调度机制彻底取代了 PFC。这一转变带来了双重收益：一方面将交换机内部的缓冲区占用率压缩到了极低水平；另一方面，彻底消除了困扰 RoCEv2 已久的队头阻塞现象。 — Sandeep Nagaraj > [18:36](https://www.youtube.com/watch?v=PmfxO4VEHQc&t=1116s) 届时，Meta 将正式开源 MetaRoCE 规范全文、完整的参考软件实现，以及严格的合规性测试套件（Compliance Test Suites）。 — Sandeep Nagaraj ## Mentioned Meta, AMD, Sandeep Nagaraj, Balakrishnan Raman (Balki), MetaRoCE, RoCEv2, P4 可编程 AI 网卡, AMD ROCm 通信集合库 (RCCL), OCP (Open Compute Project), PFC (Priority-based Flow Control), SACK (Selective Acknowledgment), ECN (Explicit Congestion Notification), DLB (Dynamic Load Balancing), FPF (Fabric Plane Fabric), PSP 协议 ## Video description Balakrishnan Raman, AMD and Sandeep Nagaraj, Meta present "MetaRoCE: From Spec to NIC to Open Source" live from the Santa Clara Convention Center. As AI and distributed workloads push datacenter fabrics to their limits, Meta's answer is MetaRoCE — a multipath, out-of-order, receiver-driven protocol that treats Ethernet as inherently lossy and pushes all intelligence into the NIC. This talk traces MetaRoCE's journey from specification to silicon, showcasing its implementation on AMD Programmable NICs via reference software implementation and demonstrating how the architecture elegantly scales out within a cluster and scales across fabrics to meet diverse deployment needs. We present real-world performance results spanning multiplane FPF topologies, tail-latency optimization, and long-distance RDMA — proving MetaRoce approach solves AI data center requirements. Finally, we present a first look at Meta's upcoming open-source release of the MetaRoCE specification, reference implementation, and compliance suites at OCP — inviting the industry to build, extend, and innovate on MetaRoCE. Learn more about @Scale here: atscaleconference.com [Source](https://www.youtube.com/watch?v=PmfxO4VEHQc)

[0:03] Sandeep Nagaraj: MetaRoCE 的核心架构理念是将全部智能下沉到网络边缘（网络接口卡/NIC），使边缘网卡具备更强的路径感知能力、拥塞自适应能力与弹性，从而在普通商用以太网之上实现顶级的性能、容灾韧性与超大规模运维能力。 [1:24] Sandeep Nagaraj: 这套软件栈充当了整个协议的“权威真理之源（Canonical Source of Truth）”，能够对“在特定网络极端场景下 MetaRoCE 究竟应当展现何种行为”给出确切的标准答案。 [4:13] Sandeep Nagaraj: 如果遵循传统的硬件开发路径，从头流片研发一款定制 ASIC 芯片网卡来实现 MetaRoCE，整个周期动辄需要数月甚至数年之久，根本无法追赶 AI 工作负载快速演进的步伐。 [4:13] Sandeep Nagaraj: AMD 网卡所具备的高性能 P4 可编程数据平面，赋予了团队随时演进协议所需的敏捷度与生产级线速吞吐能力。 [7:43] Balakrishnan Raman: MetaRoCE 并非在各个路径层级上进行“盲目的数据包随机喷洒（Blind Packet Spraying）”。协议栈在软件中为每条路径精细维护着独立的拥塞窗口（CWND）。 [7:43] Balakrishnan Raman: MetaRoCE 在发送端与接收端均构建了精准的启发式逻辑，能够百分之百区分“DLB 引起的数据包乱序”与“链路真实的丢包”。网卡对乱序具有极高的容忍度，唯有在确认出现真实物理丢包时才会触发精确的选择性重传（Selective Retransmission）。 [11:13] Balakrishnan Raman: 通过在路径级别持续监控显式拥塞通知（ECN）信号，网卡能够独立调整该路径的拥塞窗口尺寸，精准限制在途报文（Outstanding Packets）数量。这种做法有效防止了流量在拥塞路径上发生丢包，同时又不会拖累畅通路径的传输效率。 [12:11] Balakrishnan Raman: 软件根据实测 RTT 的急剧增加，自动为该 QP 引导并激活更多的通信路径（例如扩展到 30 至 50 条路径）。通过横向堆叠更多的固定窗口路径，整个 QP 的并发拥塞窗口被放大到了原本的数十倍，从而在极长距离、高延迟的 Scale-Across 场景下依然能够轻松跑满网络线速（Line Rate）。 [13:53] Balakrishnan Raman: 基于各活跃 QP 的实时负荷，接收端主动计算并向发送端下发针对特定 QP 的速率提示（Rate Hint）。发送端收到提示后，在逐 QP 粒度上执行精确的速率限制（Rate Limiting），防止接收端被洪流淹没。 [15:00] Balakrishnan Raman: 即使将网络恶化到极端恶劣的百分之一（1/100）丢包率，尽管重传次数增多导致性能出现一定程度的下滑，但任务依然能够稳定推进并顺利收敛，集群计算进程完全没有中断。相比之下，传统的 RoCEv2 在面对 1% 丢包率时，Go-Back-N 机制会彻底陷入重传风暴，导致任务无法收敛直至整个计算任务直接报错崩溃。 [16:24] Sandeep Nagaraj: 在整个容灾过程中，故障在亚秒级（Subsecond）内便被协议感知，集群在 5 秒之内完成了流量的优雅降级；当故障排除后，系统在 10 秒之内重新回到了全线速运行状态。尤为关键的是，整个检测与恢复全部依靠协议带内（Inband）自洽完成，完全不需要任何带外（Out-of-band）网络信令干预。 [17:48] Sandeep Nagaraj: MetaRoCE 用接收端驱动的精准流量调度机制彻底取代了 PFC。这一转变带来了双重收益：一方面将交换机内部的缓冲区占用率压缩到了极低水平；另一方面，彻底消除了困扰 RoCEv2 已久的队头阻塞现象。 [18:36] Sandeep Nagaraj: 届时，Meta 将正式开源 MetaRoCE 规范全文、完整的参考软件实现，以及严格的合规性测试套件（Compliance Test Suites）。

[0:03] Sandeep Nagaraj: making end hosts and Network Interface Cards (NICs) path-aware, resilient, and highly performant without depending entirely on complex, fabric-level network intelligence. [0:03] Sandeep Nagaraj: Dynamic parameter adjustments allow MetaRoCE to handle both scale-out (intra-datacenter) and scale-across (long-distance RDMA) topologies simultaneously. [0:43] Sandeep Nagaraj: To avoid delaying real-world production feedback, Meta partnered with AMD to prototype MetaRoCE on AMD's P4-programmable AI NIC platform (Pensando architecture). [0:43] Balakrishnan Raman: Programmability ensures seamless updates to header formats as the MetaRoCE spec evolves. [11:00] Balakrishnan Raman: By continuously monitoring incoming Explicit Congestion Notifications (ECN) per path, the NIC dynamically adjusts per-path window sizes to restrict outstanding packets on congested links, preventing drops while saturating all available healthy paths to minimize Flow Completion Time (FCT). [11:00] Balakrishnan Raman: This expands the aggregate window by 50–100×, allowing long-distance connections to achieve full line rate without stalling. [11:00] Balakrishnan Raman: MetaRoCE implements a receiver-driven traffic scheduling framework. The receiving NIC monitors active receiving QPs that have posted buffers and calculates an explicit rate hint, sending it back to each remote sender so they can rate-limit at the per-QP level before switch buffers overflow. [11:00] Balakrishnan Raman: In contrast, RoCEv2’s Go-Back-N retransmission mechanism retransmits large windows of valid packets upon a single drop, causing catastrophic throughput loss and job non-convergence. [16:24] Sandeep Nagaraj: Across all evaluation benchmarks—covering small-to-large message sizes in both unidirectional and bidirectional flows—MetaRoCE achieved full line rate across all planes. [16:24] Sandeep Nagaraj: MetaRoCE treats Ethernet as an inherently lossy fabric, replacing PFC pause frames with receiver-driven packet pacing, path spraying, and robust SACK retransmissions. [18:44] Sandeep Nagaraj: Meta is officially committing to open-sourcing the MetaRoCE specification by October at the Open Compute Project (OCP) Global Summit.

[0:03] Hi, welcome to Atscale networking. I'm [0:07] Sep. I'm part of the data center [0:08] networking organization at Meta. And on [0:11] stage with me is Bal Krishnan Raman from [0:13] AMD. He is dearly known as Balki in the [0:16] networking circle. So thank you for [0:17] joining Balki. [0:20] All right. So we're going to be talking [0:22] a little bit more about Metaroki, right? [0:24] Uh which is Meta's custom RDMA [0:26] transport. We'll share with you our [0:29] implementation journey of Metaroki. Um [0:32] the results that we gathered showing [0:34] capabilities of Metaroki etc. and [0:36] towards the end tie it all together to [0:38] our uh plans to open source Moroi. [0:43] Okay. So Mari optimizes RDMA transport [0:46] for AI data centers uh at scale. Uh it [0:49] does so by making the network edge more [0:52] intelligent uh more path aware etc. uh [0:55] to deliver performance uh resiliency and [0:57] operational scale uh without wholly [1:00] relying on the fabric level [1:01] intelligence. You heard from Arvin and [1:04] Kingshuk a little bit earlier. They took [1:07] you deeper into the Metari protocol, its [1:09] algorithms, designs and how it all [1:11] works. Um Balk and I will take you [1:13] through the implementation journey of [1:14] Metari, right? And the experiment [1:16] results that shaped it all. [1:24] All right. So once we had the initial [1:26] specification, our development process [1:28] was pretty much a tight loop between the [1:30] specification and gathering results and [1:32] understanding how we refine the [1:33] specification. Right. Uh so we started [1:36] with the software reference [1:37] implementation [1:39] uh which became like a canonical source [1:40] of truth like uh a definitive answer to [1:44] how should Metari behave and what should [1:46] it do in a given network situation [1:48] right? Uh from there we moved to uh real [1:51] hardware implementation which allowed us [1:53] to gather results at scale over Ethernet [1:57] fabric higher bandwidth running compute [2:00] collectives and real AI workloads etc. [2:04] Cool. So building metal itself wasn't [2:07] enough right of course we needed [2:08] something to validate it with. So we [2:10] developed compliance test suites and [2:12] performance tools u everything that [2:15] would give us empirical data to [2:16] understand whether we need to refine the [2:18] implementation or we need to refine the [2:20] specification etc. So all of this would [2:23] uh be the key elements of this [2:26] development process right. So right so [2:28] once we have this key elements we kept [2:31] iterating right we repeated until we got [2:34] things right going from spec to [2:35] implementation to test and refine until [2:38] Metari would meet the goals and solve [2:40] the challenges that it was supposed to [2:42] solve. As an example, we'll talk a [2:45] little bit later here today about how [2:47] Metari can dynamically adjust its [2:49] parameters and entropy values within its [2:51] congestion and reliability algorithms to [2:54] be able to simultaneously scale out and [2:56] scale across long distances enabling [2:58] long-distance RDMA and for all of this [3:01] to get like for our ability to iterate [3:04] and rapidly uh evolve our implementation [3:07] and the specification simultaneously was [3:10] key to making all of this possible. [3:16] Cool. So the software reference [3:18] implementation that we we wrote covers [3:21] metrois both transmit and receive [3:23] mechanisms right. So it covers the [3:25] congestion control reliability multia [3:28] tags for scaling etc and all of the [3:30] features and this is packaged to work [3:33] with RDMA and supports all of the [3:35] functions and uh semantics and words [3:37] that it invokes. It's a socket-based [3:40] user space implementation that can run [3:42] on any commodity CPU and generic nick. [3:45] Um, no proprietary software stacks or no [3:48] kernel module dependencies etc. Right? [3:51] So this paired with the compliance test [3:53] we uh I just spoke about became like a [3:55] golden model to validate any of the [3:57] metari hardware implementation. Right? [4:00] So and we also expect this model to [4:02] continue to guide the nick community in [4:04] validating your own implementation of [4:06] metari on your hardware. [4:13] So going from software reference [4:15] implementation to hardware right [4:17] developing a custom ASC a neck that [4:20] implements Metaroki could take months if [4:22] not years before we can put it to real [4:23] world tests. So we needed something [4:26] different. We needed speed and [4:28] flexibility in prototyping ability to [4:30] debug and understand the differences [4:32] between implementation needing work or [4:34] the specification needing work etc. So [4:37] and also we we kept operational [4:39] readiness in mind right this wasn't just [4:41] for lab curiosity so we needed this to [4:43] work uh in real life in production so [4:45] that's where we partnered with AMD um we [4:49] implemented Metari on their P4 [4:51] programmable AI neck which the [4:53] programmability gives us all the [4:54] features and the flexibility that we [4:56] needed so Balki will take us through the [4:58] implementation and the results [5:05] >> thanks Adep So as San mentioned hardware [5:08] implementation of an evolving protocol [5:11] needs to um match those attributes that [5:14] are specified in the previous slide. Now [5:17] let's see how AMD's programmable ANA [5:19] satisfies those requirements. [5:22] So in this uh picture we see some of the [5:25] key blocks of the ANA hardware that are [5:28] typically involved in the development of [5:30] a transport protocol. [5:34] Firstly, there is a programmable depser [5:36] and parser. [5:38] So the dep parser uh helps in [5:40] encapsulating the packets with headers. [5:43] So with an evolving protocol you have to [5:46] encapsulate new headers. There will be [5:49] additions or modifications to the fields [5:50] in the header. So a programmable depser [5:54] is suited you know um to encapsulate the [5:57] packets with any changes to the fields [5:59] in the headers. [6:01] Likewise on the receive side there is a [6:03] programmable parser where it needs to [6:06] parse the packet uh specifically the [6:09] headers extract the metadata feeds it to [6:11] the pipeline so that the pipeline can [6:13] process the packet. Again a programmable [6:16] parser can take care of new header [6:18] formats uh any modifications or [6:21] additions to the fields in the header. [6:26] Next is the P4 programmable multi-core [6:29] pipeline. in the P4et pipeline which [6:32] deals with packet logic. Specifically, [6:35] it deals with the packets between the [6:37] P4DMA pipeline you see at the top and [6:39] the uplink port. [6:41] So, it deals with the logic such as uh [6:44] doing security checks of packets that [6:45] are received from the uplink port um [6:48] with respect to you know setting up the [6:50] packet for inline encryption like PSP [6:53] and any other custom logic. [6:57] Next is the packet buffer which is the [7:00] hardware block where the payload of the [7:02] packet is kept until the packet is [7:04] processed by the pipeline and the [7:07] payload is DMA to the host memory in the [7:09] receive direction or the DM8 packet from [7:12] the host memory which is packetized and [7:14] kept in the packet buffer is transmitted [7:16] out of the upink port in the transmit [7:18] direction. [7:20] So the packet buffer has cues uh it [7:23] basically supports all sorts of quality [7:25] of service classes and hence uh it [7:27] provides you the end toend quality of [7:29] service support. [7:33] This is the P4 programmable multi-core [7:36] um pipelines uh which is the P4 DMA [7:39] pipeline. This is where the metari [7:41] protocol is implemented. [7:43] So all the metari uh features like [7:46] multipaththing, [7:47] selective acknowledgement, selective [7:49] retransmission, congestion control [7:51] algorithms [7:52] and the multiplane are all implemented [7:55] in this programmable pipeline. [7:58] So the the multiping aspect of metari [8:01] protocol it is congestion aware. It's [8:03] not like a blind spraying of packets at [8:06] every path level. The software maintains [8:08] a congestion window. So where the where [8:11] where wherein it can monitor the ECN [8:13] received from the path and it can adjust [8:15] the congestion window and thereby adjust [8:17] the number of packets outstanding per [8:18] path. [8:21] The congestion control itself is [8:22] implemented as a framework where it has [8:25] programmable parameters that can be [8:27] tuned. So we have implemented the [8:30] metaris built-in congestion control [8:32] algorithms the baseline algorithm and [8:34] then the receiver based congestion [8:36] notification which can tune these [8:38] parameters and thereby control the [8:40] framework for the protocol. [8:44] So you can come up with custom logic [8:46] custom congestion control logic which in [8:48] in turn can tie in with the framework [8:50] and control these parameters. [8:53] Next is the selective acknowledgement [8:55] and retransmission. Again it is [8:57] completely tunable. Um so typically you [9:00] want to have ECMP based hashing in the [9:03] network uh wherein the packets are uh [9:06] load balanced based on flow entropies. [9:09] But in case if you have uh say DB or [9:12] dynamic load balancing based packet u [9:16] you know load balancing in the network [9:18] say flow or packet based DLB. Even with [9:21] the packet DLB where you will see out of [9:24] order reception, we have the logic in [9:26] the receiver and the sender where it can [9:28] um distinguish between out of order [9:31] reception of the packet versus actual [9:33] packet drops and it will retransmit only [9:36] when there is packet drop the actual [9:37] drop. [9:39] Next multiplane uh which is basically um [9:42] where there are multiple planes of [9:44] network fabric where the GPUs are [9:46] connected uh for availability reasons. [9:49] So with the multiplane uh basically the [9:51] implementation ensures that traffic is [9:54] evenly load balanced across all the [9:56] planes so that you can maximize the [9:58] throughput and also within the plane it [10:01] makes use of all possible parts. So all [10:03] possible network parts are used so that [10:05] you can minimize the flow completion [10:07] time. [10:09] The operational metrics are implemented [10:11] completely in the software. you can [10:12] modify, you can add and are context [10:15] aware so that it can help in um quick [10:18] turnaround of the implementation. [10:20] So we'll do we'll go over the con case [10:22] studies of all these um features. [10:26] We'll see how we iterated through the [10:28] development uh to meet the AA network [10:30] performance goals. [10:35] So there is also a you know programmable [10:37] PCA interface uh through which you can [10:40] you know uh read or write any data [10:42] structure like typically the descriptor [10:44] that you read it can be any form it [10:46] doesn't have to be a fixed format and [10:48] also you can keep any other data [10:50] structures that is needed for the [10:52] protocol support itself. [10:57] So let's look at the first case study. [10:59] Let's look at the congestion aware [11:00] multiping or the packet spraying aspect [11:02] of the metari. So what we did is we [11:05] connected the clusters uh with uh so in [11:08] this case there are multiple logical [11:10] links connected to the switches. So [11:12] there is one link where it is [11:13] experiencing congestion as a result of [11:15] which it has 100 Gbps of available [11:17] bandwidth whereas the other one has [11:19] available bandwidth of 400 Gbps. [11:22] So the per the per path we have the [11:26] congestion window. So here um for the Q [11:30] pair it has multiple paths and the paths [11:33] are mapped to all available network [11:35] paths [11:37] whether it is uh having the full [11:39] available bandwidth or not. So because [11:41] it has the congestion window at the per [11:42] path level it monitors the ECN and then [11:46] based on that it sizes the congestion [11:48] window where it can limit the number of [11:49] outstanding packets and thereby it can [11:51] avoid drops in those spots that are [11:54] experiencing congestion. [11:56] So whether irrespective of whether there [11:58] is congestion or not it tries to make [12:01] use of all available network paths and [12:04] thereby it you know minimizes the flow [12:06] completion time. [12:11] The next case study is about the [12:13] simultaneous scale out and scale across [12:15] connections. Scale out is the one where [12:17] you interconnect GPUs within the [12:19] building. The scale across is the one [12:21] where you interconnect GPUs across the [12:24] buildings. So naturally the RT of the [12:27] scale across connection will be quite [12:28] high. So in this case in this example [12:31] let's see like 50 to 100 times that of [12:33] the scale out connection. So for a scale [12:36] out connection let's say if the RTD is [12:39] like 5 to 15 microsconds. So the Q pair [12:42] has a congestion window of 72 packets. [12:46] So we have a so there are like 12 parts [12:49] per QP. uh so which basically you know [12:52] contributes to that uh 75 um you know [12:56] packets of congestion window to keep [12:58] that many outstanding packets for that [13:00] RT with the metari uh one of the key [13:04] things is that uh the maximum size of [13:07] the congestion window per path is fixed [13:10] which makes the implementation very [13:11] simple. So now in order to take care of [13:14] this high RTT for the scale across [13:17] connection. [13:19] So basically in the software we [13:21] dynamically bootstrap the paths per per [13:23] Q pair. So basically based on the RT [13:26] that you see here the RT is like 50 to [13:28] 100 times it dynamically bootstraps more [13:31] paths for the QP. So it bootstraps [13:33] around 30 to 50 paths here which opens [13:36] up the congestion window. So to 50 to [13:39] 100 times that of the scale out [13:40] connection that you are seeing as a [13:42] result it keeps more packets outstand [13:44] outstanding and you can still reach the [13:47] line rate for a scale across connection. [13:53] So we looked at the uh network [13:55] congestion which are handled based on [13:57] the network congestion signals such as [13:58] the ECN RT. There is also congestion at [14:02] the receiver when multiple senders try [14:04] to send uh to the receiver at the same [14:06] time. the incast scenario. So, metari [14:09] has this receiver-based congestion [14:11] notification where it monitors the [14:13] number of keepers that are active and at [14:15] the per QP level it hints the rate um to [14:19] the sender so that the sender can rate [14:21] limit at the per QP level to avoid [14:24] overwhelming the receiver. [14:26] So in the software we have built the [14:28] logic where the receiver [14:31] monitors the number of Q pairs that are [14:32] actively receiving traffic Q pairs for [14:35] which receive buffers are posted which [14:37] which which which tells you that more [14:40] packets are received for those QRS. So [14:42] based on that it can accurately you know [14:44] notify the rate hint per QP. You can [14:47] also come up with custom logic uh based [14:50] on the deployment needs uh to monitor [14:52] the number of QPS that are active so [14:54] that you can accurately hint the rate to [14:56] the sender. [14:59] The next case study is about the [15:01] selective acknowledgement and selective [15:03] retransmission. So we did it we did the [15:05] performance analysis uh with a a rickle. [15:08] So we did the all to all and all reduce [15:11] collectives um in a drop and no drop [15:14] scenarios. [15:15] So as you see here uh with a 1 in 1,000 [15:19] percentage drop so the bar is still [15:21] closer to the no drop bar. Basically [15:24] what it says is [15:26] there is not much performance [15:28] degradation with occasional drops. This [15:30] is unlike go backend retransmission [15:32] mechanism with a rocky v2 where whenever [15:35] there is drop it can result in unwanted [15:37] retransmissions uh unwanted packets will [15:40] get retransmitted [15:42] uh and thereby reduces the throughput. [15:45] As you see when the percentage of drops [15:47] increases like one in 100 we see there [15:50] is some performance degradation because [15:52] there are more number of retransmissions [15:54] but still it allows the workload to [15:57] converge. There is no failure in [15:58] convergence. [16:00] This is unlike go back end because with [16:02] go back in as you increase the [16:04] percentage of drops go back end doesn't [16:06] converge and ends the workload. [16:09] Now next let's see the case study of [16:12] multiplane where we did the performance [16:13] analysis and also the availability [16:16] aspect of the multiplane in the meta [16:18] lab. Sundep will cover the details. [16:21] Thank you. [16:24] >> Thank you BI. Um so we discussed how [16:27] Maroki protocol makes the network edge [16:29] intelligent through its path-based [16:31] receiver-driven approach. To [16:34] systematically validate and showcase [16:35] this we built an 8x00 gig multiplane FPF [16:39] network in our lab with disjoint planes. [16:42] uh this kind of validation is important [16:44] because multiplane essentially adds an [16:47] additional layer of resource arbitration [16:49] and u awareness and uh uh intelligence [16:53] at the nick level that has implications [16:54] to metarok's pathbased and load [16:57] balancing mechanisms. So if you see here [17:01] we have the uh traffic per test results [17:04] across various message sizes and both [17:06] unidirectional and birectional uh [17:09] achieves line rate by utilizing all the [17:11] paths available. And to add to this, we [17:14] simulated remote endpoint failure and [17:17] recovery within the fabric for one [17:19] plane. And the event itself was detected [17:22] in subsecond and graceful degradation [17:24] was achieved in under 5 seconds. And [17:28] upon recovery, we see line rate uh being [17:30] achieved in under 10 seconds. And all of [17:32] which without any impact to traffic or [17:36] uh of on any other planes, right? So all [17:39] of this is happening inband within the [17:41] protocol without requiring any out [17:43] ofband signaling u and fabric signaling. [17:48] So bali showed results of how metari [17:50] manages incast and load balancing [17:53] situations dynamically right. Uh as a [17:55] result of this, Metari achieves faster [17:57] collective comput uh completion times [18:00] compared to Rocky V2 and it does so by [18:03] spraying packets across all available [18:04] paths and maximizing bandwidth [18:06] utilization within the network and also [18:08] keeps the switch buffer utilization to a [18:10] minimal by replacing the PFC uh Arai [18:15] V2's PFC based congestion control with [18:17] Metaroki's receiver-driven traffic [18:19] scheduling. It's able to achieve all [18:20] this. uh by eliminating PFC, Metari is [18:24] also avoiding the head-offline blocking [18:25] that tends to impact proy v2 [18:28] significantly when there is workload [18:31] concurrency and cluster scale increases. [18:36] All right, so what's next as we move [18:38] forward? Uh we want to enable the Nick [18:40] community to develop Metari on your [18:42] products, right? So for this we're going [18:44] to open source the Metari specification [18:46] by October at OCP. We'll also open [18:49] source the reference implementation, the [18:51] compliance test suites etc that we [18:53] discussed and we welcome you to join us [18:57] at OCP global conference. Uh we will [18:59] have a demo of metaroki working live and [19:02] also uh presentations or more [19:04] presentations on metaroki and you get a [19:06] chance to interact with meta engineering [19:08] team working on metari as well. So thank [19:11] you