Meta, AMD Tailor NIC Hardware to Fix AI Network Drops

Meta与AMD定制NIC硬件，化解AI网络丢包难题 Meta将传输智能下推至网络边缘设备，以消除网络架构拥塞，并在极端丢包情况下实现接近理论上限的RDMA性能。 演讲者梳理了MetaRoCE从软件参考实现演进到利用AMD P4可编程NIC平台实现芯片部署的历程。他们阐述了关键架构机制，包括智能多路径、接收端驱动拥塞通知（RCN）以及选择性确认（SACK）。在多平面FPF拓扑下的实验结果表明，该方案可实现线速吞吐量，并在0.1%的丢包率下表现出近乎为零的影响。最后，Meta宣布计划通过OCP开源相关规范、参考代码及合规测试套件。 Meta 推出面向 AI 数据中心的 MetaRoCE 传输协议，通过将传输智能下沉至网卡（NIC）边缘，并基于套接字的软件参考实现进行语义规范与验证。 (0:05) 为避免耗时数年的 ASIC 开发周期，Meta 联合 AMD 利用其 P4 可编程网卡架构，快速在硬件上实现了解析器、P4 DMA 流水线与传输逻辑迭代。 (3:03) MetaRoCE 摒弃无差别的报文喷发，结合 ECN 反馈实施逐路径拥塞控制，并针对高延迟跨机房连接动态扩展单 QP 并行路径以打满线速。 (8:06) 协议引入接收端驱动拥塞通知（RCN）以缓解 incast 拥塞，并借助选择性重传（SACK）机制在 0.1% 丢包率下仍保持近乎无损的 AI 集合通信性能。 (12:10) Meta 宣布将在 OCP 全球峰会上全面开源 MetaRoCE 规范、软件参考实现以及一致性与性能测试套件，推动行业软硬件生态协同构建。 (15:39)

Meta and AMD engineered MetaRoCE, a custom RDMA transport protocol that moves network intelligence to the adapter edge to improve AI data center networking at scale. Implemented via a canonical software reference and AMD's P4-programmable NIC platform, the architecture optimizes multipathing, congestion management, and loss recovery for intra- and inter-building AI workloads. Edge-Driven Architecture and Rapid Hardware Prototyping (3:03) MetaRoCE pushes transport intelligence directly to the network edge, avoiding total reliance on fabric-level switches to deliver performance, resiliency, and operational control. To establish the protocol's semantics, Meta initially developed a socket-based user-space software reference implementation operating on commodity CPUs and standard NICs without proprietary software or kernel dependencies. This canonical implementation was paired with a compliance test suite to establish a verification baseline for hardware development. To bypass conventional multi-year ASIC manufacturing cycles, Meta collaborated with AMD to deploy MetaRoCE on AMD's P4-programmable network adapters. The platform uses a programmable parser/deparser to encapsulate custom headers dynamically, alongside a P4 packet pipeline handling security checks, interface lookups, and inline encryption like PSP. Transport features—including multipathing, congestion control, multiplane handling, and selective acknowledgments—run within P4-driven multi-core DMA memory engines, while a programmable PCIe interface fetches host data structures to supplement on-chip NIC state. Adaptive Multipathing and Dynamic Congestion Management (8:06) Rather than relying on uncoordinated packet spraying, MetaRoCE implements congestion-aware load balancing using per-path congestion windows governed by Explicit Congestion Notification (ECN) signals. When links exhibit asymmetric capacity or degradation—such as a 100 Gbps path alongside an undamaged 400 Gbps path—the sender throttles in-flight bytes on the constrained route while saturating the pristine link, preventing slow paths from degrading overall flow completion time. The protocol also adapts to significant round-trip time (RTT) disparities between scale-out intra-building connections (5 to 15 microseconds) and scale-across inter-building links with RTTs up to 50 times higher. Because individual per-path congestion windows are capped at 256 in current deployments, high-delay scale-across flows cannot hit line rate over limited paths. MetaRoCE counters this bandwidth-delay product limitation by dynamically instantiating 30 to 50 parallel paths per Queue Pair (QP). To manage receiver-side incast, Receiver-Driven Congestion Notification (RCN) monitors active QPs and buffer pressure, returning rate hints that trigger senders to reduce congestion windows before buffer exhaustion occurs. Selective Retransmission and Multiplane Network Performance (12:10) Standard RoCEv2 relies on Go-Back-N recovery, which re-sends broad blocks of data after a single drop and often stalls under heavy packet loss. MetaRoCE replaces this with a tunable Selective Acknowledgment (SACK) engine that uses heuristics to separate out-of-order packet delivery caused by dynamic routing from genuine network drops, retransmitting only missing packets. In benchmarks evaluating All-to-All and All-Reduce collectives, workloads subject to a 0.1% packet loss rate matched zero-loss baseline performance, and jobs continued to make progress and converge under 1% loss conditions. In laboratory tests over an 8x100G multiplane Fabric-Port-First (FPF) topology with disjoint planes, MetaRoCE sustained full hardware line rate without transport overhead or NIC arbitration bottlenecks. In-band detection mechanisms identified remote endpoint failures in under one second to degrade traffic safely, and line-rate performance resumed within 10 seconds once endpoints recovered without impacting adjacent planes. By substituting receiver-driven scheduling for Priority Flow Control (PFC) pause frames, the protocol avoids switch bufferbloat and head-of-line blocking. Open Source Release and Standardization Roadmap (15:39) Meta announced plans to open-source the MetaRoCE ecosystem to enable industry adoption and multi-vendor NIC validation. The company is contributing the complete protocol specification to the Open Compute Project (OCP). Alongside the specification, Meta will release its canonical user-space software reference implementation and its internal hardware compliance and performance verification test suites. By offloading transport intelligence to programmable NICs and eliminating PFC dependencies, MetaRoCE demonstrates near-roofline collective performance under loss and incast. Meta and AMD plan to open-source the specification and validation tooling through OCP to broaden industry adoption across third-party silicon.

Meta与AMD联合公布了专为AI数据中心定制的RDMA传输协议MetaRoCE，该方案将传输智能下沉至网络边缘网卡，以摆脱对网络交换机织网智能的重度依赖。双方基于用户态软件参考模型与AMD P4可编程网卡实现了从规范制定、硬件原型加速到实测验证的全周期协同，并计划通过开放计算项目（OCP）全面开源该技术与测试验证套件。 软硬件协同定义与P4可编程网卡架构 (3:03) 为了在AI数据中心解决传统RoCEv2在大规模流量下的局限性，Meta工程团队确立了以网络边缘网卡为主导的协议设计思路。Meta首先构建了一套基于套接字（socket-based）的用户态软件参考实现，运行于通用CPU与普通网卡硬件之上。该参考实现完全独立于内核模块与专有软件栈，并结合一致性测试套件与性能评测工具，充当了MetaRoCE语义验证的绝对“真实来源”（source of truth）。此举确保了团队能够在实际硅片流片前，基于真实AI计算集合通信负载对协议的收发机制与RDMA标准verbs接口进行系统化验证。 鉴于传统网卡ASIC长达数月乃至数年的硬件研发周期无法匹配快速迭代需求，Meta选择与AMD合作，借助其P4可编程网卡平台承载MetaRoCE规范。AMD工程师Balakrishnan Raman指出，该网卡架构包含数个关键组件：可编程解析器与反解析器（Parser/Deparser）支持动态封装与解析自定义报头；P4数据包流水线负责安全检查、PSP线速加解密及接口查找；数据包缓冲区结合服务质量（QoS）分类与动态调度算法暂存载荷；多核P4 DMA内存流水线则直接运行包括多路径管理、拥塞控制、选择性确认（SACK）在内的核心传输逻辑；此外，可编程PCIe接口允许硬件直接从主机内存抓取任意数据结构以扩充片上状态。 感知拥塞的多路径调度与跨数据中心扩展 (8:06) 针对传统数据中心网络粗暴喷洒报文（packet spraying）所导致的无序和拥塞问题，MetaRoCE采用了精细的拥塞感知负载均衡机制。该协议通过接收端返回的显式拥塞通知（ECN）信号，为每一条物理路径独立维护拥塞窗口。例如，当网络中同时存在降级的100 Gbps链路与健康的400 Gbps链路时，发送端能够动态压缩拥塞路径的在途数据量，同时最大化利用高带宽路径，从而避免低速或拥堵链路拖慢整体流完成时间（FCT）。 在处理机房内横向扩展（scale-out）与跨机房分布式互联（scale-across）共存的复杂拓扑时，Meta团队面临往返时延（RTT）差异剧烈的挑战。机房内RTT通常介于5至15微秒之间，而跨机房RTT最高可达前者的50倍。受限于单路径拥塞窗口上限（当前实现为256），跨机房的高时延带宽积（BDP）无法通过少数路径打满线速。为此，MetaRoCE为跨机房队列对（QP）动态建立多达30至50条并行子路径以填满传输管道。同时，针对多对一通信中的接收端突发涌入（incast）拥塞，MetaRoCE设计了接收端驱动拥塞通知（RCN）机制：接收端实时监控活跃QP数量与接收缓冲区水位，一旦面临缓冲耗尽风险，便直接向发送端反馈路径级速率调节指令，从根本上替代依赖PFC停帧的传统模式。 SACK高容错重传与多平面拓扑线速验证 (12:10) 标准RoCEv2协议采用回退N帧（Go-Back-N）重传机制，一旦发生丢包就会重传后续所有报文，在动态负载均衡环境下极易恶化网络利用率。MetaRoCE引入了具备自适应启发式引擎的选择性确认（SACK）架构，能够精准区分因ECMP或数据包级多路径造成的乱序到达与真正的物理丢包，仅触发缺失分段的重传。测试数据显示，在All-to-All和All-Reduce等集合通信负载下，面对千分之一（0.1%）的丢包率，MetaRoCE的通信性能与零丢包基线几乎一致；即便在丢包率高达百分之一（1%）的极端劣化网络中，MetaRoCE作业依然能够持续收敛，而同期对照的传统RoCEv2网络则频繁发生停顿甚至彻底失败。 Meta的Sandeep Nagaraj展示了在8x100G多平面织网端口优先（Fabric-Port-First, FPF）独立拓扑下的实验室实测表现。实测表明，MetaRoCE在各独立网络平面上均达成了硬件线速吞吐，未遭遇网卡内部仲裁瓶颈或传输协议开销抑制。在模拟远端节点物理宕机的容错实验中，带内检测机制在1秒以内便完成故障识别并实现流量优雅降级；当故障节点恢复后，MetaRoCE在10秒内重新恢复至全线速运行，且邻近网络平面的数据流完全未受影响。结合接收端流量调度，系统消除了多租户混部场景下的队头阻塞与交换机缓冲膨胀，集合通信完成时间逼近硬件理论上限。 OCP开源计划与产业生态系统构建 (15:39) 为了降低产业界跟进AI网络边缘传输优化的门槛，Meta与AMD宣布将全面开源MetaRoCE的技术资产。根据双方规划，Meta将把完整的MetaRoCE协议规范移交至开放计算项目（OCP）进行标准化发布。此外，Meta内部用于验证逻辑正确性的规范级用户态软件参考实现，以及全套硬件一致性与性能评测测试集也将同步向社区公开，从而使第三方网卡制造商与网络设备供应商能够独立构建、验证符合该规范的硬件产品。双方亦邀请业界工程团队在当年度的OCP全球峰会上参与现场硬件演示与技术交流。 MetaRoCE展示了将传输控制重心从交换网络转移至可编程网卡边缘的架构优势，在解决跨机房长RTT通信、高丢包容忍度与多平面线速调度方面取得了实证成果。然而，该协议在大规模异构硬件环境中的长期通用性，以及第三方芯片供应商在非P4架构网卡上实现该复杂状态机时的开销与难度，仍有待在开放计算项目（OCP）生态落地后接受更广泛生产环境的检验。

## Background 该视频由频道 @Scale 于 2026 年 9 月 3 日发布在 YouTube 平台上。视频简介指出，随着 AI 和分布式负载将数据中心网络推向极限，来自 Meta 的 Sandeep Nagaraj 与来自 AMD 的 Balakrishnan Raman 共同介绍了多路径、乱序、接收端驱动的 MetaRoCE 传输协议，展示其从协议规范、基于 AMD 可编程网卡的原型落地到 OCP 开源生态的完整历程。 ## TL;DR Meta 携手 AMD 推出了专为大规模 AI 工作负载打造的新型 RDMA 协议 MetaRoCE，其核心理念是将网络智能全面下沉至端侧网卡，将以太网视为天然有损网络。该协议通过用户态软件参考实现与 AMD P4 可编程网卡硬件构建研发闭环，采用基于接收端拥塞通知（RCN）的单路径速率提示与高容错 SACK 重传机制，彻底摒弃了易导致死锁的 PFC。实测表明，MetaRoCE 在 8×100G 多平面网络与跨园区长距离 RDMA 场景下均能跑满线速，具备亚秒级故障自愈能力，并在高丢包率与重度 Incast 场景下保持接近理论上限的任务完成时间。Meta 宣布将在 OCP 全面开源该规范、参考软件及合规测试套件以推动行业生态共建。 ## Atmosphere & style 整场演讲呈现出严谨扎实、务实干练的顶尖系统工程技术范儿，两位演讲者分工明确、配合默契，叙事紧凑而胸有成竹。从对 RoCEv2 传统痛点的精准剖析，到 AMD P4 硬件实现架构的层层抽丝剥茧，再到多平面与高丢包实测数据的直观对比，技术说服力层层递进。观众能清晰感受到超大规模数据中心一线架构师直面网络物理极限时的工程智慧与底气，结尾宣布全套开源的号召更是将全场推向开放协作的行业愿景高潮。 ## Bulletin - [00:05](https://www.youtube.com/watch?v=cnAd8val8P8&t=5s) Meta 联合 AMD 发布专为大规模 AI 工作负载设计的定制 RDMA 传输协议 MetaRoCE，采用多路径、乱序处理与接收端驱动架构，全面将网络智能下沉至网卡边缘。 - [01:13](https://www.youtube.com/watch?v=cnAd8val8P8&t=73s) 协议验证打破漫长的 ASIC 研发周期，通过用户空间纯软件参考实现与 AMD P4 可编程网卡硬件原型实现快速闭环，并成功在 8×100 Gbps 多平面 FPF 拓扑中跑满线速。 - [10:46](https://www.youtube.com/watch?v=cnAd8val8P8&t=646s) MetaRoCE 彻底摒弃了易导致死锁的逐跳 PFC 流控，引入接收端拥塞通知（RCN）与选择性重传（SACK），在高达 1% 的极端丢包率下仍能保障集合通信收敛。 - [09:01](https://www.youtube.com/watch?v=cnAd8val8P8&t=541s) 针对跨园区长距离传输面临的 50 倍 RTT 暴增，协议通过为单个 QP 动态扩展 30 至 50 条并发路径，利用多路径窗口叠加突破单路径 256 cwnd 的硬件限制。 - [15:39](https://www.youtube.com/watch?v=cnAd8val8P8&t=939s) Meta 宣布将在 OCP 全球峰会上全面开源 MetaRoCE 协议规范、用户态软件参考实现及合规测试套件，推动下一代 AI 高性能传输协议标准化。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=cnAd8val8P8&t=5s) MetaRoCE 的核心设计理念是将智能推向网卡端侧（NIC），将以太网视为天然有损网络，降低对底层交换矩阵复杂机制的依赖。 - [01:13](https://www.youtube.com/watch?v=cnAd8val8P8&t=73s) 基于通用套接字的用户空间软件参考实现充当协议语义的权威标准，无需内核模块或专有软件栈即可兼容标准 RDMA Verbs 接口。 - [02:20](https://www.youtube.com/watch?v=cnAd8val8P8&t=140s) 采用 AMD P4 可编程网卡硬件作为原型验证载体，大幅压缩了传统网卡 ASIC 芯片数月至数年的漫长研发周期。 - [03:46](https://www.youtube.com/watch?v=cnAd8val8P8&t=226s) AMD P4 网卡架构将可编程解析器、P4ET 数据包流水线与 P4 DMA 内存流水线结合，以纯软件形式支撑了 MetaRoCE 的核心传输逻辑。 - [08:06](https://www.youtube.com/watch?v=cnAd8val8P8&t=486s) 多路径数据分发坚持拥塞感知原则，基于单路径拥塞窗口（per-path cwnd）动态分配流量，彻底杜绝盲目喷洒导致的连锁丢包。 - [09:01](https://www.youtube.com/watch?v=cnAd8val8P8&t=541s) 跨建筑 Scale-Across 连接的时延可达同集群 Scale-Out 的 50 倍，MetaRoCE 通过动态绑定 30 至 50 条并发路径优雅解决 BDP 瓶颈并跑满线速。 - [10:46](https://www.youtube.com/watch?v=cnAd8val8P8&t=646s) 针对 AI 集合通信中严重的多打一 Incast 拥塞，MetaRoCE 引入接收端拥塞通知（RCN），由接收端根据活跃 QP 与接收缓冲余量主动回传速率提示进行节流。 - [11:28](https://www.youtube.com/watch?v=cnAd8val8P8&t=688s) 选择性重传（SACK）取代了传统 RoCEv2 的回退 N 步（Go-Back-N）机制，在千分之一丢包率下几乎无性能损失，百分之一丢包率下依然维持收敛。 - [13:26](https://www.youtube.com/watch?v=cnAd8val8P8&t=806s) 在 8×100 Gbps 多平面 FPF 拓扑实测中，系统通过带内协议机制实现远端节点故障的亚秒级感知降级，并在节点恢复后不到 10 秒内恢复满线速。 - [14:40](https://www.youtube.com/watch?v=cnAd8val8P8&t=880s) MetaRoCE 彻底淘汰了 PFC（基于优先级的流量控制），有效规避了队头阻塞（Head-of-Line Blocking）与缓冲区死锁问题，集合通信完成时间逼近理论上限。 - [15:39](https://www.youtube.com/watch?v=cnAd8val8P8&t=939s) Meta 将通过 OCP 正式开放 MetaRoCE 规范、参考软件实现以及配套合规测试套件，降低全行业网卡厂商与系统集成商的适配门槛。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=cnAd8val8P8&t=5s) MetaRoCE 概述：从规范到产品与开源愿景 来自 Meta 基础设施团队的 Sandeep Nagaraj 与来自 AMD 的 Balakrishnan Raman（Bali）共同登台，分享 Meta 针对大规模 AI 数据中心自主研发的定制 RDMA 传输协议——MetaRoCE。该协议的核心设计哲学在于“将网络边缘做深做透”，通过赋予网卡端侧极高的智能，在不完全依赖网络交换矩阵（fabric）底层复杂机制的前提下，全面实现高吞吐、强韧性与卓越的规模化运维能力。在另一场配套技术分享中，Meta 的 Arvind Srinivasan 深入阐释了 MetaRoCE 的协议架构、运作机制及其如何化解 AI 算力集群对底层网络的苛刻要求；而本场分享则着重回顾 MetaRoCE 如何从一张白纸的设计规范演进为真正的硬件产品，并公布其落地验证的关键实验数据。值得注意的是，Meta 正式宣布将这一从规范到参考实现的全套方案向开源社区与 OCP 开放。 - MetaRoCE 是 Meta 针对大规模 AI 工作负载自主研发的多路径、乱序、接收端驱动的 RDMA 传输协议。 - 设计核心理念是将智能下沉至网络边缘（NIC），而非单纯依赖交换网络层。 --- ### [01:13](https://www.youtube.com/watch?v=cnAd8val8P8&t=73s) 规范与实验闭环：参考软件实现与可编程硬件原型 MetaRoCE 的研发历程由“规范定义与实验探索之间的严密闭环”所驱动。为了建立规范的基准，团队最初构建了一套软件参考实现（Software Reference Implementation），作为 MetaRoCE 协议语义的权威“真实源头”（canonical source of truth）。这套参考实现基于通用套接字（socket-based）运行在用户空间，能够直接跑在通用 CPU 与标准普通网卡上，完全脱离了任何内核模块或专有软件栈的依赖，并完整向上兼容 RDMA 动词（Verbs）接口。与这套软件实现配套的还有一套合规测试套件（Compliance Test Suite）与性能基准工具，为持续迭代协议与规范提供了必不可少的经验数据。 ```mermaid flowchart TD accTitle: MetaRoCE 研发与验证闭环流程 accDescr: 展示从规范定义、软件参考实现到可编程硬件验证及最终落地的闭环开发流 subgraph SpecCycle["规范与基准验证"] A["MetaRoCE 协议规范"] --> B["用户态软件参考实现<br/>(通用 CPU / NIC)"] B --> C["合规测试套件与性能工具"] C --> A end subgraph HardwareCycle["硬件落地与集群验证"] A --> D["AMD P4 可编程网卡实现"] D --> E["大规模真实 AI 集群评测<br/>(高带宽 / 通信原语 / 真实负载)"] E -->|实测反馈数据| A end E --> F["满足 AI 数据中心集群需求<br/>(开源至 OCP)"] ``` 然而，一款全新网卡的 ASIC 硬件研发往往需要耗费数月甚至数年之久，团队面对瞬息万变的 AI 算力演化节奏，根本“没有这种挥霍时间的奢侈”。为了实现快速原型验证并即时获得硬件级实测反馈，Meta 携手 AMD，选用其 P4 可编程网卡硬件作为载体。这使得 MetaRoCE 能够在极短时间内在真实的高带宽、大规模算力通信原语（Compute Collectives）及实际工作负载下完成真机实测，从而打通了从理论定义到数据中心实际落地的完整闭环。 - 软件参考实现无需定制内核驱动或专用私有软件栈，为网卡供应商和社区生态提供了金标准。 - 研发闭环结合了规范、测试工具、软件栈及 AMD P4 硬件原型，支持横向扩展（Scale-out）以及跨数据中心建筑的长距离 RDMA（Scale-across）。 --- ### [03:46](https://www.youtube.com/watch?v=cnAd8val8P8&t=226s) AMD P4 可编程网卡架构剖析 AMD 的 Balakrishnan Raman（Bali）接过话茬，详细拆解了 AMD P4 可编程网络适配器如何完美承载像 MetaRoCE 这样快速演进的网络传输协议。该网卡内部配备了一整套高度可编程的处理模块，从包头处理到内存引擎均具备极强的动态配置能力： ```mermaid flowchart LR accTitle: AMD P4 可编程网卡核心处理架构 accDescr: 展示数据包在可编程解析器、P4ET流水线、数据包缓存以及P4 DMA流水线之间的流转路径 Uplink["上行链路 Uplink / Host"] --> Parser["可编程解析 / 解封装器<br/>(Custom Parser/Deparser)"] Parser --> P4ET["P4ET 数据包处理流水线<br/>(安全检查 / PSP 加解密 / 路由)"] P4ET <--> PktBuf["数据包缓冲区 Packet Buffer<br/>(端到端 QoS / 调度算法)"] P4ET --> P4DMA["P4 DMA 内存流水线<br/>(MetaRoCE 传输协议全软件实现)"] P4DMA <--> HostMem["主机内存 Host Memory<br/>(PCIe 接口扩展存储)"] ``` 网卡的核心处理单元及其在 MetaRoCE 中的分工包括： 1. **可编程解析器与解封装器（Custom Parser/Deparser）**：在发送端，解封装器可以灵活封装任意自定义协议头或对头部字段进行动态重写；在接收端，解析器能够解析任何形式的定制报头，为协议字段演进提供了极大宽容度。 2. **P4 数据包处理流水线（P4ET Pipeline）**：由多核 P4 可编程处理器驱动，负责拦截处理来自上行链路或 P4 DMA 引擎的数据包，能够内嵌执行安全检查、PSP（Packet Security Protocol）行内加解密，以及复杂的接口路由查找逻辑。 3. **数据包缓冲区（Packet Buffer）**：用于在 P4ET 及 P4 DMA 流水线对报文进行深层传输逻辑处理期间暂存数据净荷（Payload），具备完备的服务质量（QoS）类别支持与流量调度算法，保障端到端的 QoS。 4. **多核内存引擎（P4 DMA Pipeline）**：MetaRoCE 的核心传输流水线（包括发送端和接收端）完全以软件形式在 P4 DMA 流水线中编写运行。多路径（Multipathing）、拥塞控制、选择性重传（SACK）与多平面调度完全在可编程软件中实现。 5. **可编程 PCIe 接口**：支持网卡灵活向主机内存拉取各种数据结构，使主机内存能作为网卡板载资源的有力补充。 6. **全面的可观测性与运维指标**：提供硬件级统计与追踪，允许管理员在全集群范围内实时监控和动态调优协议运转。 - 选择性重传（SACK）逻辑具有高度可调性：无论底层网络是基于 ECMP、动态负载均衡（DLB）还是包级负载均衡，SACK 启发式算法均能精准辨别报文是“暂时乱序”还是“真实丢包”，杜绝不必要的无谓重传。 - 拥塞控制支持大量的可编程覆盖参数（programmable overrides），便于针对不同的通信原语（如 All-to-All 对比 All-Reduce）以及组网规模进行定制化策略调优。 --- ### [08:06](https://www.youtube.com/watch?v=cnAd8val8P8&t=486s) 拥塞感知负载均衡与多路径智能分流 针对多路径场景，Bali 强调 MetaRoCE 绝不采用那种对网络状况一无所知的“盲目喷洒”（mindless spraying）。相反，MetaRoCE 在每个细分路径（per-path）维度上都独立维护并跟踪拥塞窗口（Congestion Window, cwnd），根据路径的实时拥塞状况智能分配数据包。 举例来说，当网络中存在不同负载状态的路径时（例如某条路径因拥塞导致等效带宽降至 100 Gbps，而另一条通畅路径可跑满 400 Gbps），MetaRoCE 会实时监控从网络反向返回的 ECN（显式拥塞通知）标记，动态缩放每条路径的拥塞窗口。网卡端根据调整后的窗口分别将两端的未完成报文数量（outstanding packets）控制在 100 Gbps 和 400 Gbps 的传输容量相匹配的水平。这种机制既保障了高吞吐路径跑满，又不会彻底遗弃利用率较低的次优路径，在充分利用网络中所有可用物理路径的同时，极大缩短了整体流完成时间（Flow Completion Time, FCT）。 - 路径分流基于每路径拥塞窗口（Per-path cwnd），杜绝全网盲发导致的连锁丢包。 - 动态利用包括降级或非均衡链路在内的所有路径，最大化聚合网络容量。 --- ### [09:01](https://www.youtube.com/watch?v=cnAd8val8P8&t=541s) 兼顾 Scale-Out 与 Scale-Across：长距离 RDMA 的路径扩展机制 在现代化 AI 基础设施中，网络通信往往需要同时支持两种迥异的拓扑模式：同建筑集群内部的横向扩展（Scale-Out）与跨建筑乃至跨数据中心的扩展（Scale-Across）。两者面临的最大差异在于往返时延（RTT）：同数据中心内的 Scale-Out 连接 RTT 通常仅在 5 到 15 微秒左右；而跨越建筑的 Scale-Across 连接，其 RTT 可能会暴增 50 倍以上。 根据时延带宽积（BDP）原理，更大的 RTT 意味着必须在网络中维持数量显著更多的未完成数据包（outstanding packets），才能维持住线速吞吐。然而，MetaRoCE 在协议层面为单条路径设定的最大拥塞窗口仅为 256。若一条 Scale-Out 链路所需的拥塞窗口是 75，面对跨建筑 50 倍的 RTT 需求，单条路径的窗口容量根本无法支撑所需的大量飞线数据包。MetaRoCE 给出的软件解法非常优雅：根据所感知到的连接 RTT 动态扩展每个队列对（Queue Pair, QP）所实例化的路径数量。例如，为 Scale-Across 连接动态绑定 30 到 50 条并发路径，利用多条独立路径的窗口叠加效应，迅速拉高并发在途报文总量，从而即使在高 RTT 的跨建筑长距离传输下，也能轻松跑满网卡线速。 - Scale-Out 典型 RTT：5 ~ 15 微秒；Scale-Across 的 RTT 可达其 50 倍之高。 - 硬件机制受限：单路径最大拥塞窗口（Max cwnd）为 256。 - 软件适配策略：对长距离大 RTT 连接，动态为单个 QP 分配 30 ~ 50 条路径，通过路径数量聚合突破单路径窗口瓶颈，实现跨园区长距离 RDMA 线速通信。 --- ### [10:46](https://www.youtube.com/watch?v=cnAd8val8P8&t=646s) 应对 Incast 拥塞：接收端拥塞通知（RCN） 除了由网络交换链路带宽瓶颈引起的网络拥塞（通过 RTT 和 ECN 应对），AI 算力通信中更普遍且致命的瓶颈来自于接收端拥塞——即多个发送端在通信原语汇聚时瞬间向同一个接收端涌入海量数据，引发剧烈的多打一拥塞（Incast）。 为了在最前端化解这一难题，MetaRoCE 引入了基于接收端的拥塞通知机制（Receiver Congestion Notification, RCN）。由于 MetaRoCE 本质上是一种接收端驱动的协议架构，接收端网卡能够实时且高精细度地监控自身的状态指标，包括当前正在接收报文的活跃 QP（Active QPs）总数，以及在主机内存中已经挂载就绪的接收缓冲区（Posted Receive Buffers）剩余余量。一旦发现接收端资源濒临透支，接收端网卡就会主动生成 RCN 提示信息回传至各个发送端，指导其即刻节流调整注入速率，从源头上遏制 Incast 导致的缓冲区崩溃。 - 传统 ECN 侧重反馈交换机网络中间节点的拥塞，而 RCN 专注防范端侧接收队列的溢出。 - 接收端通过动态关联活跃 QP 数与可用接收缓冲资源，在多打一并发激增时主动向发送端“提示发送速率”，实现高压 Incast 下的平稳收敛。 ### [11:28](https://www.youtube.com/watch?v=cnAd8val8P8&t=688s) 基于单路径速率提示的拥塞控制与 SACK 重传容错机制 在 MetaRoCE 的设计架构中，接收端驱动的拥塞控制通过在每个 QP（Queue Pair）的单路径级别（per QP path level）向发送端返回“速率提示”（rate hint）。当发送端收到针对特定路径的速率提示后，即可在细粒度的路径级别动态调整拥塞窗口（congestion window），进而精确控制单条路径上的在途数据包数量（outstanding packets）。这种由接收端全面掌控反馈的软件实现方案，赋予了系统极高的灵活性——工程团队能够在接收端编写定制化逻辑，实时监控活跃的 QP 数量与流量分布，并针对具体的数据中心部署场景精准生成速率提示，从根源上化解令传统网络头疼的 Incast 拥塞风险。 ```mermaid flowchart LR accTitle: MetaRoCE 接收端驱动速率调节与选择性重传 accDescr: 示意发送端与接收端之间通过路径级速率提示和 SACK 机制协同工作，规避 Incast 并实现高效重传。 subgraph RX["接收端 (Receiver)"] A["监控活跃 QP 队列"] --> B["计算路径级 Rate Hint"] E["检测丢包并生成 SACK"] end subgraph TX["发送端 (Sender)"] C["调整单路径 Congestion Window"] F["仅重传 SACK 指定的丢失数据包"] end B -- "Rate Hint (Per Path)" --> C C -- "按配额发送数据" --> RX E -- "SACK 反馈" --> F ``` 与此同时，选择性确认（SACK, Selective Acknowledgment）机制彻底重塑了 MetaRoCE 的丢包恢复性能。传统的 RoCEv2 协议在丢包时采用回退 N 步（Go-Back-N）机制，迫使发送端重新传输大量本已被接收端正确接收的无辜报文，导致宝贵的网络带宽被无效重传严重消耗。MetaRoCE 仅对真正丢失的报文发起重传，在针对 All-to-All 和 All-Reduce 等典型 AI 分布式集合通信算子的丢包压测中展现出极强的鲁棒性： - 在高达千分之一（1/1000）的丢包率下，性能开销微乎其微，测试柱状图的高度几乎与零丢包基准持平。 - 即使面对百分之一（1/100）的极端丢包率，系统虽然会出现一定的性能降级，但整体工作负载仍然稳健收敛（converges）；反观 RoCEv2，一旦丢包率攀升至此等量级，整个通信过程将彻底崩溃并无法收敛。 --- ### [13:26](https://www.youtube.com/watch?v=cnAd8val8P8&t=806s) 多平面 FPF 架构验证与秒级亚秒级故障自愈能力 来自 Meta 的 Sandeep 接过演讲，系统展示了 Meta 实验室针对 MetaRoCE 的端侧智能与路径感知机制所完成的严谨验证。Meta 构建了一个 8×100 Gbps 的多平面（multiplane）FPF（Fabric Plane Fabric）网络拓扑，各平面之间保持物理隔离。多平面架构的引入意味着网卡（NIC）层面必须承担额外的资源仲裁职责，这对 MetaRoCE 的路径管理和流量负载均衡构成了直接的工程挑战。iPerf 基准测试表明，MetaRoCE 在该多平面网络上轻松跑满了线速（line rate），且未暴露任何协议层、网络平面或硬件实现上的性能瓶颈。 ```mermaid flowchart TD accTitle: 8x 100G 多平面容灾与亚秒级收敛过程 accDescr: 展示远端节点发生故障及恢复过程中，各平面的流量表现与自愈时间节点。 subgraph Detect["故障发生与亚秒级检测"] F1["远端节点发生故障"] --> F2["带内协议机制 (Inband)"] F2 --> F3["亚秒级 (<1s) 完成故障隔离与优雅降级"] end subgraph Recover["节点恢复与线速复原"] R1["远端节点恢复就绪"] --> R2["系统平滑回切路径"] R2 --> R3["不到 10 秒内恢复 100% 满线速"] end Detect --> Recover ``` 在网络高可用与弹性测试中，团队人工模拟了远端节点故障及随后的恢复全过程。测试数据显示，MetaRoCE 展现出了卓越的自愈能力： - **故障检测与优雅降级**：系统完全通过带内（in-band）协议机制在“亚秒级”（sub-second）时间内感知到远端节点故障，迅速调整流量分发，实现了业务层面的优雅降级。 - **恢复与无损回切**：当远端节点恢复后，MetaRoCE 在不到 10 秒内即重新打满全量线速（full line rate），并且整个故障与自愈过程完全被平滑隔离，未对其他并行网络平面上的业务流量造成任何连带冲击（no collateral impact）。 --- ### [14:40](https://www.youtube.com/watch?v=cnAd8val8P8&t=880s) 消除 PFC 阻断 head-of-line 阻塞，实现近理论上限的完成时间 在集体通信（collective completion time）的表现上，MetaRoCE 相比传统 RoCEv2 呈现出压倒性优势。RoCEv2 往往因为协议层面的瓶颈而受限于极少数网络路径，而 MetaRoCE 则能够主动将海量数据包均匀喷洒（spreading packets）到全网所有可用的网络路径上，将整体网络带宽利用率压榨到极致。更核心的变革在于，MetaRoCE 彻底摒弃了 RoCEv2 长期依赖的基于优先级的流量控制（PFC, Priority Flow Control），全面转向接收端驱动的流量调度策略。 | 评估维度 | 传统 RoCEv2 | MetaRoCE 创新架构 | |:--- |:--- |:--- | | **流控基础机制** | 依赖逐跳 PFC（基于优先级的流量控制） | 完全抛弃 PFC，采用端侧接收端调度 | | **拥塞行为表现** | 易引发 Incast 拥塞，交换机浅缓存被迅速占满 | 精确预防 Incast，交换机缓存占用大幅降低 | | **混合负载干扰** | 频繁遭遇队头阻塞（Head-of-Line Blocking） | 从源头杜绝队头阻塞，业务隔离性强 | | **大规模集群伸缩性** | 随节点规模与并发度上升，性能发生严重雪崩 | 即使在重度 Incast 下依然维持近“顶线”（near roofline）水平 | - PFC 的彻底消除让网络设备摆脱了死锁与级联拥塞的梦魇，混合工作负载下的性能不再受到队头阻塞的拖累。 - 无论是在高度并发的 AI 训练场景还是突发吞吐下，MetaRoCE 的任务完成时间均紧贴理论上的“顶线”（roofline completion time），而 RoCEv2 则随着集群规模的扩大和任务并发量的剧增而发生显著的性能退化。 --- ### [15:39](https://www.youtube.com/watch?v=cnAd8val8P8&t=939s) 推动社区生态：OCP 全面开源规范、参考实现与合规套件 在演讲的尾声，Meta 表达了将 MetaRoCE 推向整个数据中心生态的决心，致力于帮助广大网卡厂商与系统集成商在自有硬件上实现这一下一代高性能传输协议： - **发布开放规范**：Meta 正式宣布将在 OCP（开放计算项目，Open Compute Project）公开发布完整的 MetaRoCE 规范文档。 - **开源核心组件**：为了帮助行业合作伙伴降低研发门槛、加速适配落地，Meta 将同步开源端到端的软件参考实现（software reference implementation）以及内部严苛的 Meta 合规测试套件（Meta compliance test suites）。 - **大会现场互动邀约**：Sandeep 热情邀请业界同仁参加即将于 10 月举行的 OCP 全球峰会（OCP Global Conference），届时 Meta 工程团队将在现场带来 MetaRoCE 的实机演示，并与广大网络与芯片工程师展开面对面的深度技术交流。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=cnAd8val8P8&t=5s) 将网络边缘做深做透 — Sandeep Nagaraj > [01:13](https://www.youtube.com/watch?v=cnAd8val8P8&t=73s) 真实源头 — Sandeep Nagaraj > [01:13](https://www.youtube.com/watch?v=cnAd8val8P8&t=73s) 没有这种挥霍时间的奢侈 — Sandeep Nagaraj > [08:06](https://www.youtube.com/watch?v=cnAd8val8P8&t=486s) 盲目喷洒 — Balakrishnan Raman > [11:28](https://www.youtube.com/watch?v=cnAd8val8P8&t=688s) 速率提示 — Balakrishnan Raman > [13:26](https://www.youtube.com/watch?v=cnAd8val8P8&t=806s) 亚秒级 — Sandeep Nagaraj > [14:40](https://www.youtube.com/watch?v=cnAd8val8P8&t=880s) 顶线 — Sandeep Nagaraj > [01:13](https://www.youtube.com/watch?v=cnAd8val8P8&t=73s) MetaRoCE 协议规范 — Sandeep Nagaraj ## Mentioned Meta, AMD, MetaRoCE, Sandeep Nagaraj, Balakrishnan Raman (Bali), Arvind Srinivasan, OCP (Open Compute Project), P4, RoCEv2, FPF (Fabric Plane Fabric), PSP (Packet Security Protocol), ECMP, DLB, SACK, ECN, RCN, PFC, iPerf ## Video description As AI and distributed workloads push datacenter fabrics to their limits, Meta's answer is MetaRoCE — a multipath, out-of-order, receiver-driven protocol that treats Ethernet as inherently lossy and pushes all intelligence into the NIC. This talk traces MetaRoCE's journey from specification to silicon, showcasing its implementation on AMD Programmable NICs via reference software implementation and demonstrating how the architecture elegantly scales out within a cluster and scales across fabrics to meet diverse deployment needs. We present real-world performance results spanning multiplane FPF topologies, tail-latency optimization, and long-distance RDMA — proving MetaRoce approach solves AI data center requirements. Finally, we present a first look at Meta's upcoming open-source release of the MetaRoCE specification, reference implementation, and compliance suites at OCP — inviting the industry to build, extend, and innovate on MetaRoCE. Learn more about the @Scale conference here: https://atscaleconference.com/ [Source](https://www.youtube.com/watch?v=cnAd8val8P8)

[0:05] Sandeep Nagaraj: 将网络边缘做深做透 [1:13] Sandeep Nagaraj: 真实源头 [1:13] Sandeep Nagaraj: 没有这种挥霍时间的奢侈 [8:06] Balakrishnan Raman: 盲目喷洒 [11:28] Balakrishnan Raman: 速率提示 [13:26] Sandeep Nagaraj: 亚秒级 [14:40] Sandeep Nagaraj: 顶线 [1:13] Sandeep Nagaraj: MetaRoCE 协议规范

[0:05] Sandeep Nagaraj: MetaRoCE: From Spec to NIC to Open Source [0:05] Software Reference Implementation: A socket-based user-space implementation running on commodity CPUs and generic NIC hardware, fully independent of kernel modules or proprietary software stacks. [3:03] Balakrishnan Raman: Programmable Parser/Deparser: On the transmit side, the deparser encapsulates packets with arbitrary custom headers and modifies header fields on the fly. [3:03] Balakrishnan Raman: Programmable PCIe Interface: Enables direct fetching of arbitrary data structures from host memory to augment on-chip NIC state. [8:06] Instead, it implements congestion-aware load balancing by maintaining per-path congestion windows based on incoming Explicit Congestion Notification (ECN) signals. [8:06] To fill the pipe, MetaRoCE dynamically instantiates significantly more parallel paths per Queue Pair (QP)—scaling up to 30 to 50 paths for inter-building connections—to maintain line-rate throughput across high-latency links. [8:06] When buffer pressure builds, the receiver emits rate hints directly back to the senders at a per-QP path level, instructing them to shrink their congestion windows before buffer exhaustion occurs. [12:10] In benchmark testing covering All-to-All and All-Reduce collective operations, a 1-in-1,000 packet loss rate (0.1%) showed performance metrics virtually identical to zero-loss baselines. [12:10] Even under severe 1-in-100 loss conditions (1%), MetaRoCE workloads continued to make forward progress and converged successfully, whereas RoCEv2 fabrics frequently stalled or failed altogether. [12:10] Sandeep Nagaraj: By replacing Priority Flow Control (PFC) pause frames with receiver-driven traffic scheduling, MetaRoCE prevents switch bufferbloat and eliminates head-of-line blocking across mixed traffic classes. [15:39] Meta is open-sourcing its canonical software reference implementation alongside its internal hardware compliance and performance verification test suites.

[0:05] Hello, I'm Sep Nagaraja and I'm part of [0:07] Meta's infrastructure organization. I'm [0:10] here today with Balrashnan Raman from [0:12] AMD. Today we are going to talk about [0:15] the implementation journey of Metaroki [0:17] which is Meta's custom RDMA transport [0:19] protocol and how we took it from an idea [0:22] and a specification to an actual [0:24] product. [0:26] This product will soon be available to [0:28] the open source community. Metari [0:31] optimizes RDMA transport for AI data [0:33] centers at scale by making the network [0:35] edge intelligent eventually delivering [0:38] performance resiliency and operational [0:41] capability at scale without wholly [0:43] relying on fabric level intelligence. In [0:46] a companion talk, Arvin Trinasan covers [0:49] the protocol details in a little more [0:51] depth, speaking about the protocol [0:53] design and its mechanisms and how the [0:56] protocol achieves its goals for AI data [0:59] centers at scale. [1:01] Here, Bali and I will focus on the [1:03] implementation journey that delivered a [1:05] working Metari product and present [1:08] experimental results to showcase that it [1:10] meets our requirement. Our development [1:13] process for Medori was driven by a tight [1:16] loop between specification and [1:18] experimentation. We began with a [1:20] software reference implementation that [1:22] acts as a canonical source of truth for [1:24] Metari protocol semantics and this [1:27] reference implementation will also be [1:29] available to the open source community. [1:31] This reference guided a real hardware [1:33] implementation enabling us to gather [1:35] experimental results in real world at [1:38] scale at higher bandwidth along with [1:41] compute collectives and real workloads. [1:46] Alongside these components, we also [1:48] developed a compliance test suite and [1:50] performance tool that provided empirical [1:53] data needed to refine the specification [1:56] and the metro protocol iteratively. [2:00] and the loop is closed when we meet the [2:02] actual demand for AI data centers at [2:04] scale. [2:06] As an example, in this talk, we present [2:08] how Metari can scale out and scale [2:10] across to facilitate long-distance RDMA. [2:14] Achieving this required iterative [2:16] improvements to the protocol and the [2:17] implementation itself. [2:21] Metaroki software reference [2:22] implementation covers the protocols [2:25] transmit and receive mechanisms and is [2:27] packaged to work with RDMAN supporting [2:30] all the words that it invokes. This is a [2:34] socket-based user space implementation [2:36] that can run on commodity CPU and [2:39] generic nick hardware without having any [2:41] dependency on kernel modules or [2:44] proprietary software stack. The [2:46] implementation combined with Meta's [2:48] compliance test suite serves as a [2:51] definitive source of truth enabling Meta [2:54] and the Nick community to develop [2:56] hardware implementation and validate [2:59] against this source of truth. [3:03] As we move from software reference [3:05] implementation to hardware [3:07] implementation of Metari, we needed [3:09] flexibility to and speed of prototyping. [3:13] Typical hardware implementations of a [3:14] new nick can take months if not years [3:17] before we can actually put it to [3:19] experimental results in real world. We [3:22] did not have that luxury. We needed [3:24] flexibility and speed in prototyping the [3:26] implementation and iteratively gathering [3:29] results and improving the specification. [3:32] This is where we partner with AMD and [3:35] use their P4 programmable neck that [3:37] gives us all the features needed in the [3:40] hardware implementation and achieve our [3:42] goal within a timeline. [3:46] Bali from AMD will take us through the [3:49] next phase of the presentation. Thanks [3:51] separate [3:53] hardware implementation of an evolving [3:55] protocol requires all those attributes. [3:58] Now let's see how AMD's programmable ANA [4:01] satisfy those hardware implementation [4:03] requirements. [4:05] Custom parser dep parser. So these are [4:08] programmable parser dep parser. The dep [4:10] parser helps in encapsulating the packet [4:13] with any type of custom headers. [4:17] Also during the implementation it helps [4:20] in encapsulating headers with the [4:22] modifications to the fields in the [4:24] headers. Similarly, programmable parser [4:28] like what the parser did in the transmit [4:31] on the receive side. It can parse any [4:33] type of packet headers, [4:35] any type of custom headers or it can [4:38] also allow any type of modifications to [4:40] the headers. [4:43] The programmable multi-core processors, [4:46] the P4et pipeline. So these are P4 [4:50] programmable pipelines. They handle [4:52] packets coming from the uplink or coming [4:55] from the P4DMA pipelines. This is where [4:58] you implement all sorts of logic custom [5:01] logic if you say like security checks [5:04] for the packets. [5:06] You can set up the packet for inline [5:08] encryption or decryption uh for say PSP [5:12] encryption decryption any kind of [5:15] interface lookup things like that. [5:18] The packet buffer this is where the [5:20] payload of the packet is buffered. This [5:23] is buffered while P4ET pipeline the P4 [5:26] DMA pipeline are processing the packet. [5:29] Whatever transport logic is being [5:31] implemented they process the packet [5:34] while the packet reside in the packet [5:35] buffer. The packet buffer supports [5:38] complete quality of service classes and [5:40] scheduling algorithms. So it supports [5:43] the end toend quality of service [5:47] programmable multi-core memory engines. [5:49] These are the P4 DMA pipeline which runs [5:52] the P4 programs. The metari transport [5:55] pipeline is implemented in these [5:57] pipelines. Both the transmit and the [5:58] receive pipelines. So things like all [6:01] the features like multipaththing, [6:03] congestion control, selective [6:05] acknowledgement, multiplane, everything [6:07] is implemented in the software. So [6:09] multipaththing it's basically spraying [6:12] of packets across the paths with the [6:14] metari. Obviously it's intelligent [6:17] spraying wherein every path has a [6:19] congestion window where it monitors the [6:21] congestion at the path level and based [6:24] on the congestion window it adjusts the [6:27] number of packets that are outstanding [6:28] at the per path level. [6:31] Next, the congestion control is [6:34] completely implemented in the software [6:36] and it has tons of programmable [6:38] overrides and what it helps is these [6:41] overrides. It helps in catering to the [6:44] congestion scenarios like in scale out [6:47] versus scale across [6:49] different types of collectives like all [6:51] to all or all reduce. Then the selective [6:54] acknowledgement it is a cudable [6:57] implementation wherein the selective [6:59] acknowledgement [7:01] can be you know can cater to any type of [7:04] network configuration like either it can [7:08] be a ECMP based or it can be a dynamic [7:10] load balancing based flow flow based [7:14] load balancing or a packet level load [7:16] balancing. So you can build urists [7:19] for the sack logic where it can [7:21] determine whether it's out of order [7:23] packet or it's a dropped case so that it [7:27] can do retransmission only if it is out [7:29] of order. So we'll go over all these [7:32] cases as case studies and see the [7:35] results whatever that we measured with [7:37] the hardware implementation in the [7:39] cluster. [7:40] Also there is a complete you know [7:42] operational metric support. So it can [7:45] the the transport protocol [7:48] implementation can be completely [7:49] monitored and tuned. [7:53] Next is the programmable PCA interface. [7:55] So this allows to fetch any type of data [7:58] structures from the host memory and the [8:00] host memory can act as a supplement for [8:02] storing any kind of data structures for [8:04] the protocol. [8:06] Next the congestion aware load balancing [8:09] in the network. So basically Metari [8:12] doesn't do any kind of mindless [8:13] spraying. It sprays packet across parts [8:16] based on the congestion window at [8:18] maintained at the per path level. So as [8:21] you see here at the per path level there [8:23] is a congestion and so one leak is [8:25] operating at the 100 gig link and the [8:27] other gig other link is operating at the [8:28] 400 gig link. So the metari monitors the [8:32] ECM coming from these things and then it [8:35] adjusts the congestion window at the per [8:37] path level. So the number of outstanding [8:39] packets will be adjusted to 100 Gbits in [8:42] the 100 gig link and to 400 GBs per [8:46] second in the second link. So based on [8:49] this it can still use all possible paths [8:52] including the underutilized path but all [8:55] possible paths are utilized and thereby [8:57] it helps in flow completion time. [9:01] Next we look at the support for [9:04] simultaneous scale out and scale across [9:06] connections. The scale out is the one [9:08] where endpoints are within the same [9:10] building and scale across connections [9:12] are the ones where the endpoints are [9:15] across multiple buildings. So the RTD of [9:18] these connections are going to be quite [9:19] different with the scale across having a [9:21] larger RT except for instance we assume [9:24] that five to 15 microconds of RT for a [9:27] scale out connection and 50 times of [9:29] that for a scale across connection. So [9:31] once the RT is larger you need to keep [9:34] more number of more packets outstanding. [9:37] So how do you do that? You have a larger [9:40] congestion window. So you need to adjust [9:42] the congestion window according to the [9:44] RT. [9:46] So for instance here for the scale [9:48] across if the congestion window is like [9:49] 75 50 times of that is needed for the [9:52] scale across connection. So you have to [9:55] have a larger congestion window. In case [9:57] of metari the max size of a congestion [10:00] window per path is 256. So in order to [10:03] increase the number of outstanding [10:05] packets so you need to increase the [10:07] congestion window and the way we do that [10:10] is you can instantiate the number of [10:12] paths per QP to be more. So as you see [10:15] here the number of paths that are [10:17] instantiated for a scale across [10:19] connection is like 30 to 50 paths for [10:22] the scale across connection and thereby [10:24] the number of outstanding packets is [10:26] increased and so we can still achieve [10:28] the line rate even with a higher RT [10:32] with a with a larger number of PS. So [10:35] this the software implementation allows [10:38] dynamic instantiation of paths according [10:41] to the RT that is that is seen for the [10:43] connection. [10:46] So we looked at the congestion scenarios [10:48] in the network which are like handled [10:51] based on the RT and the ECN. There is [10:54] also congestion on the receiver when [10:56] multiple centers overwhelm the receiver. [10:58] This is the INC scenario. So here in [11:02] case of metari congestion control there [11:05] is a RCN receiver based congestion [11:08] notification where the receiver can [11:10] continuously monitor the number of [11:12] active coopers in the receiver. The [11:14] number of active keepers are [11:17] continuously monitored based on number [11:19] of QPS that are receiving packets, [11:21] number of receive buffers that are [11:23] posted for the receivers and based on [11:25] this the receiver can hint the rate at [11:28] the per QP path level to the senders. So [11:32] when the senders looks at the rate hint [11:35] at the per path level they can adjust [11:38] their congestion window and thereby they [11:40] can adjust the number of outstanding [11:42] packets at the per path level and [11:44] thereby they can avoid overwhelming the [11:47] receiver. [11:48] So the software implementation of this [11:51] congestion control allows you to build [11:53] any type of custom logic in the receiver [11:56] to monitor the number of active Q pairs [11:59] and thereby you know depending on the [12:02] deployment scenario you can accurately [12:05] advertise the rate hint to the senders [12:07] to avoid incast. [12:10] Next, let's look at the SAC [12:12] implementation where we do a selective [12:15] acknowledgement of the drops and thereby [12:18] the sender can retransmit only those [12:21] dropped packets and unlike the Rocky V2 [12:25] where it's a go back in and it will [12:29] result in retransmitting unnecessary [12:31] packets which are already received by [12:32] the receiver. So as you see here even in [12:35] one in 1,000 drops these are the graphs [12:38] for the all to all and the all reduced [12:40] collectors with drops. So even in one in [12:43] 1,000 drops percentage as you see here [12:46] that one in 1,000 drops bar is as close [12:48] to the no drops bar. So the performance [12:51] degradation is not that much because of [12:53] the selective acknowledgement and [12:54] retransmission of only those dropped [12:56] packets and the performance degradation [12:59] is not that much and even in one in 100 [13:01] there is performance degradation but [13:03] still the workload converges. Never is [13:06] the case where the workload not [13:08] converges unlike a rocky v2 where when [13:11] you increase the percentage of drops it [13:14] fails to converge. Next let's look at [13:17] the multiplane results. So I'll hand it [13:20] over to Sundep who will cover the [13:22] multiplane results that we measured in [13:24] the meta lab. Thank you. [13:26] >> Thank you Bali for that implementation [13:28] journey and experimental results. [13:31] We've discussed how Medoroki makes the [13:33] ed edge intelligent through its [13:36] path-based receiver-driven approach. To [13:40] showcase this through systematic [13:41] validation, we built an 8x 100 gig [13:45] multiplane FPF fabric with disjoint [13:47] planes. [13:49] This validation is key because [13:51] multiplane introduces an additional [13:53] layer of resource arbitration at the [13:56] neck level that also has implications [13:58] for metarocarchies path management and [14:00] load balancing. IPF test results show [14:04] the line rate performance here on a [14:07] multiplane fabric with no protocol [14:10] fabric or implementation bottlenecks. We [14:13] also simulated remote endpoint failure [14:16] and recovery on this fabric. And as you [14:19] can see, the failure was detected [14:21] through inband protocol mechanisms in [14:24] subsecond to achieve graceful [14:27] degradation. And upon recovery of the [14:31] remote endpoint, full line rate was [14:33] achieved in under 10 seconds with no [14:35] collateral impact to traffic on other [14:38] planes. [14:40] Metaroki also achieves faster collective [14:42] completion time than compared to Rocky [14:45] V2 by spreading packets across all [14:47] available network paths, maximizing [14:50] bandwidth utilized instead of the [14:53] protocol becoming a bottleneck by [14:55] focusing on a fewer paths. This replaces [14:58] Rocky V2's PFC based congestion control [15:02] with receiver-driven traffic scheduling. [15:05] This prevents incast congestion at [15:06] endpoints and keeps the switched buffer [15:09] utilization dramatically lower. By [15:12] eliminating PFC entirely, Metari avoids [15:15] the head-to- offline blocking that [15:17] impacts Rocky V2 in mixed workloads. [15:21] The net result here is a near roof line [15:25] completion times even under heavy [15:27] incast. [15:29] While rock EV2 comparatively degrades [15:32] significantly as cluster scale and [15:35] workload concurrency increases. [15:39] As a next step we want to enable the [15:41] nick community to develop metari on [15:44] their products. For this we are going to [15:46] publish the metari specification at OCP. [15:49] We will also open source the software [15:51] reference implementation and meta's [15:53] compliance test suites to enable faster [15:56] and smoother implementation. [15:58] Come join us at the OCP global [16:00] conference in October for a live demo [16:04] presentation and interaction with Meta's [16:07] engineering team on Metaroki. [16:10] Thank you.