Meta, AMD Replace Standard RoCE With Custom Transport Protocol

Meta与AMD用定制传输协议替代标准RoCE Meta RoCE通过将多路径智能移至边缘终端并支持数据包乱序交付，无需更换交换机硬件即可解决数据中心热点问题。 专家组探讨了Meta为何开发Meta RoCE——这是一种专为解决标准RoCE v2在大规模部署下性能失效问题而设计的定制乱序RDMA传输协议。专家们深入分析了多项核心架构概念，包括多路径性能验证、分层拥塞控制、长距离下扩展至多达72条逻辑路径，以及将传输智能下推到边缘终端。此外，AMD与Meta强调了硬件可编程性与协同设计的重要性，并确认该协议可在标准商用交换机基础设施上运行，且将在OCP峰会上开源。 Meta 因标准 RoCE v2 缺乏报文级多路径报头且无法处理乱序到达问题，从零开始自研了 Meta RoCE 传输协议。 (0:03) 针对远距离数据中心架构（RTT 高达 1.5 毫秒），Meta RoCE 通过在可用网络链路上将报文打散喷洒至多达 72 条逻辑路径来保障高可靠传输。 (4:12) AMD 利用硬件可编程性与 Meta 协同设计并演进协议，在硬件端平衡实现了对 4,000 个队列对（QP）及高达 8,000 个路径上下文的支持。 (5:45) Meta RoCE 将所有传输智能移至边缘端点，采用子路径级分层拥塞控制，并完全兼容现有的通用商用交换机（仅需标准 ECN 与 ECMP 支持）。 (9:18) Meta 及其合作伙伴计划在 OCP 峰会上开源 Meta RoCE 规范及参考实现，以推动整个硬件生态系统的协同采用。 (16:45)

Meta, in collaboration with AMD and Keysight Technologies, developed Meta RoCE as a custom out-of-order RDMA transport protocol designed to eliminate network hot spots in massive AI clusters without abandoning commodity Ethernet infrastructure. Limitations of RoCE v2 in Scaled Clusters (0:03) Arvind Srinivasan of Meta explains that scaling data center networks caused hot spots because standard RoCE v2, as defined by the InfiniBand Trade Association (IBTA), cannot handle packet-level multipathing and fails when packets arrive out of order. Because legacy RoCE v2 packet headers lacked the architectural headroom needed to support out-of-order delivery, Meta abandoned incremental patches and designed an entirely new transport protocol from scratch. Kingshuk Mandal from Keysight Technologies notes that machine-sensitive east-west AI training workloads make fabric congestion, packet arrival precision, and tail latency hyper-critical compared to traditional traffic. Srinivasan clarifies that Meta RoCE resolves these constraints at the network endpoints while preserving existing network investments, requiring only standard commodity switches capable of basic Explicit Congestion Notification (ECN) marking and Equal-Cost Multi-Path (ECMP) forwarding. Edge Intelligence and Hierarchical Congestion Control (9:18) Meta RoCE's transport architecture pushes all delivery intelligence to endpoint devices while keeping intervening network switches simple. Because endpoints possess direct insight into workload intent, they monitor basic path feedback like round-trip time (RTT) and ECN markers from standard switches to balance traffic across fine-grained sub-path abstractions termed "tiny network pipes." Mandal identifies hierarchical congestion control as a key protocol innovation: rather than throttling bandwidth exclusively at the Queue Pair (QP) level, Meta RoCE manages congestion at the individual sub-path level, driving higher fabric utilization. Sandeep Nagaraja of Meta and Balakrishnan "Balky" Raman of AMD acknowledge that this design required navigating engineering trade-offs, such as balancing hardware state limits against path granularity and deciding which control parameters to hardcode in silicon versus leaving open for software tuning. Multipath Validation and Long-Distance Fabric Scaling (1:05) Mandal outlines four operational requirements essential to validating true multipath performance: providing sufficient entropy differentiation so split flows do not collapse onto the same physical link, maintaining symmetric return multipathing for acknowledgments to keep RTT calculations accurate, tracking dedicated per-path bookkeeping rather than sharing Retransmission Timeouts (RTO), and dynamically responding to link failures and traffic spikes. Keysight's test suite validated these multipath dynamics across up to 8 physical paths. To evaluate "scale-out" and "scale-across" deployments where data centers expand across longer physical distances, Meta tested the protocol over fiber spools reaching round-trip times of up to 1.5 milliseconds. By dynamically adjusting congestion windows and spraying traffic across up to 72 logical paths over the physical test fabric, Meta demonstrated that the protocol reliably sustains end-to-end RDMA across extended geographical spans. Programmable Hardware Co-Design and Open Collaboration (5:45) AMD co-designed the hardware implementation alongside Meta while the protocol specification was still evolving. Raman highlights that hardware programmability was indispensable, allowing AMD to update logic to match specification adjustments without waiting for silicon respins. AMD initially validated core congestion control and multipathing mechanisms on a baseline subset of RDMA operations before expanding feature coverage, eventually scaling hardware resources to maintain 4,000 QPs with up to 8,000 path contexts. To accelerate ecosystem-wide adoption, Nagaraja announced that Meta will open-source the Meta RoCE specification and reference implementation at the Open Compute Project (OCP) Global Summit. The project team engineered the reference implementation to guide community developers and partners, scheduling deep-dive architectural presentations and interactive hardware demonstrations for the summit. Meta RoCE demonstrates that packet-level multipathing and out-of-order RDMA can function across commodity Ethernet switches by relocating transport intelligence to programmable endpoints, though wider adoption hinges on how easily the broader hardware ecosystem can support granular sub-path state tracking following its open-source OCP release.

Meta 联合 Keysight Technologies 与 AMD 开发了自研传输协议 Meta RoCE，旨在突破标准 RoCE v2 无法支持数据包级多路径与乱序接收的架构瓶颈，以满足大规模 AI 训练集群的高吞吐与低长尾延迟需求。 标准 RoCE 的架构瓶颈与自研动因 (0:03) Arvind Srinivasan（Meta）指出，在 Meta 扩展数据中心集群规模时，底层网络迅速出现局部拥塞热点。InfiniBand 贸易组织（IBTA）发布的标准 RoCE v2 规范存在根本性缺陷：其协议头格式无法支持数据包级别的乱序接收与多路径传输，一旦数据包乱序到达便会导致传输系统失效。在确认无法通过修改现有报头向后兼容该特性后，Meta 决定完全从零构建支持乱序投递的自研 RDMA 传输协议。 Kingshuk Mandal（Keysight）补充强调，与传统互联网浏览流量不同，AI 训练高度依赖对延迟极端敏感的“东西向”机器间流量。长尾延迟、报文到达精度以及织构拥塞对 AI 算力效率具有决定性影响，这促使互联架构必须打破传统单路径转发的束缚。 多路径运行承诺与长距离扩展验证 (1:05) Kingshuk Mandal 阐述了多路径性能验证必须兑现的四项技术指标：一是充分的熵分化（Entropy Differentiation），将大象流拆解为小鼠流时必须具备足够熵值，避免流量重新汇聚到单一物理路径；二是接收端对称多路径回传，确认报文（ACK）必须同样经由多路径返回，以防单路径拥塞扭曲往返时间（RTT）测算；三是针对每个子路径建立独立的账务跟踪（Bookkeeping），共享重传超时（RTO）会导致状态跟踪失效；四是具备动态剔除故障路径与应对负载突发的自愈能力。目前 Keysight 测试套件已完成最高 8 条物理路径的验证。 在长距离集群扩展方面，Sandeep Nagaraja（Meta）与 Balakrishnan Raman（AMD）分享了实验数据：Meta RoCE 在长达 1.5 毫秒 RTT 的光纤卷实验环境中实现了可靠传输。在物理测试设置评估 8 条直连链路的基础上，Meta 在可用网络织构中构建了多达 72 条逻辑路径进行报文喷洒，以适配跨地域地理分布的数据中心端到端 RDMA 扩容需求。 边缘端点智能与分层拥塞控制机制 (9:18) Arvind Srinivasan 阐明了 Meta RoCE 的核心设计哲学：将所有传输智能推向网络边缘终端，保持交换机网络管道极致精简。由于只有端点理解具体工作负载意图，Meta 避免了部署昂贵复杂的可编程交换机网络，协议仅要求通用网络设备提供标准的显式拥塞通知（ECN）标记和等价多路径路由（ECMP）转发。端点负责从基础网络设备收集 ECN 和 RTT 反馈，在抽象出的“微小路径”间执行细粒度负载均衡。 Kingshuk Mandal 强调，Meta RoCE 的一大突破在于引入了分层拥塞控制架构。该机制改变了以往仅在队列对（QP）层级实施流控的做法，将拥塞控制细化落实到每个独立的子路径上。Sandeep 与 Raman 进一步说明，设计中需权衡硬件资源开销与状态跟踪，例如通过适当折减支持的 QP 数量以换取单流更高的子路径上下文容量，同时明确划定硬件固化参数与软件调优参数的边界。 硬件可编程协同设计与开源生态推进 (5:45) 在协议规范持续演进的过程中，AMD 借助硬件可编程性实现了快速迭代。Balakrishnan Raman 强调，硬件可编程能力使团队能够在规范不断调整的情况下灵活适配，避免了耗时的芯片重新流片（respin）周期。AMD 优先在部分核心 RDMA 操作上验证了拥塞控制和多路径逻辑，随后扩展至多平面传输等高级特性，并成功在硬件层面支撑了 4000 个队列对（QP）以及多达 8000 个子路径上下文的状态维护。 Sandeep Nagaraja 宣布，Meta 将在开放计算项目（OCP）全球峰会上正式开源 Meta RoCE 规范与参考实现。该参考实现的开发初衷不仅是满足 Meta 内部与 AMD 的工程落地，更旨在为行业合作伙伴与开发者提供统一的技术基座，加速产业界对下一代以太网 RDMA 技术的协同开发。 Meta RoCE 通过在商用以太网交换机之上强化端点智能与可编程硬件协同，为突破传统 RoCE 的扩展瓶颈提供了工程路径。不过，该协议在更大规模生产集群中的实际收益以及多厂商生态互操作性，仍有待即将发布的 OCP 开源规范与后续行业应用检验。

## Background 该内容由 @Scale 频道于 2026 年 9 月 3 日发布在 YouTube 平台上。视频记录了在 @Scale 大会期间举办的一场现场联合问答（Q&A），由来自 Meta、是德科技（Keysight Technologies）和 AMD 的核心技术专家共同参与，围绕 Meta 自研的新一代网络协议 Meta RoCE 的技术内幕与工程落地展开深度探讨。 ## TL;DR 面对超大规模 AI 集群中传统 RoCEv2 无法支持乱序交付与多路径传输的硬伤，Meta 联合 AMD 与 Keysight 从零打造了原生支持多路径的新一代 Meta RoCE 协议。专家深入剖析了多路径验证必须信守的“四大承诺”、长距离园区互联（Scale-across）下的动态拥塞控制，以及将全部智能推向边缘端点、保持底层网络极简的架构哲学。团队还披露了在网卡片上内存严苛受限下削减 QP 总量以换取细微路径分层拥塞控制的软硬件博弈细节，并强调该方案可完全复用现有仅支持标准 ECMP 和 ECN 的商用交换机；最后，Meta 宣布将在 OCP 峰会上全面开源 Meta RoCE 协议规范与参考实现。 ## Atmosphere & style 整场研讨散发着一线系统架构师与芯片专家之间极度坦诚、默契且富于幽默感的硬核工程师氛围。在探讨协议规范快速演进与硬件资源压榨时，Meta 与 AMD 工程师关于“在芯片里多抠出一点资源”的软磨硬泡互动令人会心一笑，生动还原了技术狂飙期软硬件协同破局的现实张力。交流没有空洞的高调宣讲，而是充满了对片上内存、单包微秒级抖动和真实集群热点等工程痛点的深切关切，传递出扎根于超大规模生产实践的务实与沉稳。 ## Bulletin - [00:03](https://www.youtube.com/watch?v=-KjG16c5fWU&t=3s) Meta 因原生 RoCEv2 无法支持数据包乱序接收和多路径分发，正式联合 AMD 与 Keysight 开发并验证了全新原生多路径协议 Meta RoCE。 - [09:18](https://www.youtube.com/watch?v=-KjG16c5fWU&t=558s) Meta RoCE 贯彻“将智能完全推向边缘计算端点”的架构哲学，在中间交换结构仅要求标准 ECMP 和 ECN 支持，实现零成本复用现有网络基础设施。 - [05:45](https://www.youtube.com/watch?v=-KjG16c5fWU&t=345s) 在 AMD 网卡硬件资源受限的博弈下，团队选择削减 QP 总量以换取更多细微路径资源，最终落地了支持 4K 活跃 QP 与驱动 8K 条路径调度的硬件方案。 - [16:42](https://www.youtube.com/watch?v=-KjG16c5fWU&t=1002s) Meta 宣布将完全开源 Meta RoCE 协议规范与黄金参考实现，并在接下来的 OCP 全球峰会上公开技术细节与开发指南。 ## Key takeaways - [00:03](https://www.youtube.com/watch?v=-KjG16c5fWU&t=3s) 原生 RoCEv2 的传统报头与规范一旦遭遇数据包乱序便会导致连接瘫痪，促使 Meta 放弃局部修补，全面自研原生支持乱序交付的 Meta RoCE。 - [01:05](https://www.youtube.com/watch?v=-KjG16c5fWU&t=65s) Keysight 提出多路径传输真正落地必须信守的四大承诺：引入充分的熵差异、ACK 对称多路径回传、独立的单路径 RTO 记账，以及快速故障切换与动态路径扩缩。 - [03:33](https://www.youtube.com/watch?v=-KjG16c5fWU&t=213s) Keysight 目前已在测试床中完成了多达 8 条物理路径的深度验证，且 Meta RoCE 架构原生具备面向更大规模路径与端口速率的扩展潜力。 - [04:12](https://www.youtube.com/watch?v=-KjG16c5fWU&t=252s) 为满足 Scale-across 跨园区级远距离通信需求，Meta RoCE 在高达 1.5 毫秒长 RTT 光纤环境下，通过动态喷洒至多达 72 条等价路径维持高速 RDMA 吞吐。 - [05:45](https://www.youtube.com/watch?v=-KjG16c5fWU&t=345s) 面对持续演进的动态协议规范，AMD 依托芯片的可编程性，采用先打磨拥塞控制与多路径核心子集、再逐步叠加多平面特性的敏捷硬件落地策略。 - [05:45](https://www.youtube.com/watch?v=-KjG16c5fWU&t=345s) 面对网卡有限的片上内存资源，Meta 与 AMD 经过权衡折中，最终落地方案支持 4K 活跃 QP，并精细驱动多达 8K 条路径的状态追踪。 - [09:18](https://www.youtube.com/watch?v=-KjG16c5fWU&t=558s) 系统设计确立了极简网络哲学：仅有边缘端点真正理解 AI 分布式训练负载的意图，中间交换 Fabric 应保持轻量精简，仅负责回传状态遥测。 - [10:09](https://www.youtube.com/watch?v=-KjG16c5fWU&t=609s) Meta RoCE 提出了“细微路径”（tiny paths）机制，将拥塞控制由传统的 QP 层级下沉至细微路径粒度，实现更灵活的分层拥塞控制与负载均衡。 - [11:23](https://www.youtube.com/watch?v=-KjG16c5fWU&t=683s) 硬件资源配置上果断放弃追求过多的 QP 规模，转而增加细微路径容量，以大幅增强应对各种复杂网络拥塞与异常场景的调度鲁棒性。 - [13:31](https://www.youtube.com/watch?v=-KjG16c5fWU&t=811s) 针对 AI 训练中极端依赖东西向流量、对微秒级延迟抖动高度敏感的特性，Keysight 重塑了测试方法学并推出专用测试套件 KAI。 - [15:22](https://www.youtube.com/watch?v=-KjG16c5fWU&t=922s) Meta RoCE 承诺完全向后兼容现有数据中心设备，仅需普通商用交换机提供标准 ECMP 转发与 ECN 打标功能，不依赖任何专有私有特性。 - [16:42](https://www.youtube.com/watch?v=-KjG16c5fWU&t=1002s) Meta 将在即将召开的 OCP 峰会上公开深度协议解析与实机 Demo，并完全开源协议规范及配套的参考实现，推动跨厂商硬件生态建设。 ## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=-KjG16c5fWU&t=3s) 告别原生 RoCEv2：Meta 自研 Meta RoCE 的根由 问答环节由来自 Meta 的主持人 James Zeng 拉开序幕。他首先向 Meta 的 Arvind Srinivasan 抛出了一个核心疑问：究竟是什么促使 Meta 决定彻底抛弃现有的 RoCEv2，转而从零打造一套全新的 Meta RoCE？为什么大家不能继续沿用传统的 RoCEv2 呢？ Arvind 解释道，随着 Meta 的 AI 集群规模不断呈指数级扩张，网络内部不可避免地开始频繁出现热点（hot spots）。为了支撑这种超大规模网络，团队深入审视了 RoCE 的底层机制。依照 InfiniBand 贸易协会（IBTA）发布的传统 RoCE 标准，协议本身根本无法支持多路径传输（multipathing）——在传统规范下，一旦出现数据包乱序（out of order），整个传输连接基本上就直接瘫痪崩溃。 面对这一致命瓶颈，团队最初试图在传统 RoCE 的框架内修修补补，为它硬加上多路径功能。然而工程师们很快发现，原生的报头格式（headers）存在天然硬伤，根本无法容纳乱序交付所需的信息。与其在死胡同里缝缝补补，Meta 最终决定另辟蹊径，全力构建一套原生支持数据包乱序接收和多路径分发的新协议，这也正式开启了后续在硬件与系统实现层面的一系列严苛挑战。 --- ### [01:05](https://www.youtube.com/watch?v=-KjG16c5fWU&t=65s) 多路径传输的“四大承诺”与验证严谨性 James 紧接着转向来自 Keysight Technologies 的 Kingshuk Mandal。在前面的演讲中，多路径（Multipath）被列为核心特性，且测试不仅要覆盖顺畅的“理想路径”（happy path），更要严格检验异常情况下的“非理想路径”（unhappy path）。James 请 Kingshuk 深入剖析 Keysight 是如何对多路径性能展开系统性验证的。 Kingshuk 指出，“多路径”听上去只是一个单一的功能特性，但在底层工程实现上却极其厚重。要想真正兑现多路径的收益，系统必须做出并严格信守“四大承诺”： ```mermaid flowchart TD accTitle: Multipath Validation Four Promises accDescr: 展现多路径传输成功运行必须履行的四个核心承诺阶段 subgraph Phase1["发送端与前向路径"] A["大象流拆分为小鼠流"] --> B["承诺一：引入充分的熵差异 (Entropy)"] B --> C["数据包沿多条物理/逻辑路径送达目的地"] end subgraph Phase2["反向确认与内部记账"] C --> D["承诺二：ACK 确认包同样走多路径反向回传"] D --> E["承诺三：独立的每路径多路径记账 (Bookkeeping & RTO)"] end subgraph Phase3["弹性容错与伸缩"] E --> F["承诺四：故障快速切换 (Failover) 与动态扩缩活动路径"] end ``` - **第一项承诺：充分的熵差异化（Entropy Differentiation）。** 当系统将一条大象流（elephant flow）拆解为若干细小的老鼠流（mice flows）时，必须引入足够且有效的熵变化。否则，即使发送端自认为分流了，数据包在网络交换中仍会哈希汇聚到同一条物理线路上，最终变成伪多路径。 - **第二项承诺：确认包（ACK）的多路径对称回传。** 如果前向业务流通过多路径送达目的地，但接收端返回的 ACK 确认包却全部挤在单条线路上返回，整体的往返时延（RTT）测量就会彻底失真，导致拥塞控制陷入混乱。 - **第三项承诺：细粒度记账机制（Bookkeeping）。** 内部状态必须针对每条路径独立维护。如果像传统做法那样所有路径共享同一个超时重传计时器（RTO），就谈不上真正的多路径管理。 - **第四项承诺：冗余切换与弹性扩缩（Redundancy & Scaling）。** 当其中某条路径故障中断时，系统能否无缝故障转移至备用路径；或者当业务负载激增时，系统能否平滑地将更多活跃路径动态注入到可用路径池中。 --- ### [03:33](https://www.youtube.com/watch?v=-KjG16c5fWU&t=213s) 物理探测路径与 Fabric 拓扑的规模延展 针对测试规模，James 提出了一个非常务实的追问：在演讲演示中通常只展示了两条路径的对比示意，但在实际的物理测试床中，究竟需要压测多少条物理路径才能确认协议真正稳定？是两到四条，还是更宏大的规模？ Kingshuk 笑着回应，测试的维度远不止三四条路径。在目前的验证阶段，Keysight 已经针对多达 8 条物理路径进行了深入测试。但 Meta RoCE 本身所具备的能力远超于此，它在协议架构层面能够原生支持更庞大的路径数量。随着未来集群不断向更大规模演进，测试方案还将进一步提升端口速率、吞吐负载以及路径维度。 --- ### [04:12](https://www.youtube.com/watch?v=-KjG16c5fWU&t=252s) Scale-Across 破局：长距离数据中心互联与动态调节 James 顺势切入 Meta RoCE 的另一大杀手锏——不仅要支撑集群内部的“横向扩展”（Scale-out），还要能够驾驭跨数据中心的“跨域扩展”（Scale-across）。这意味着网络需要应对更长物理距离的传输考验。他请 Meta 的 Sandeep Nagaraja 详述 Meta RoCE 是如何征服长距离通信这一行业难题的。 Sandeep 解释说，正如 AMD 的 Balakrishnan Raman（Balky）在演讲中所展示的，团队已经在实验室环境下利用长距离光纤卷轴（fiber spools）成功测试了高达 1.5 毫秒的长往返时延（RTT）。能够跨越如此遥远的距离保持高速 RDMA 通信，关键在于协议能够**动态调整拥塞窗口（Congestion Windows）和活跃路径数量**。 - **物理与逻辑路径叠加**：虽然目前端到端压测的独立物理路径数为 8 条，但在底层交换 Fabric 结构中，流量通过动态喷洒（packet spraying）的方式被分发到了多达 72 条等价路径上，从而在长延迟链路上平摊压力并维持极高吞吐。 - **面向未来超大规模数据中心**：随着 Meta 的 AI 训练园区和数据中心建筑群急剧扩建，物理距离拉长已是不可逆的趋势。端到端 RDMA 必须具备穿透园区级长距离的能力，Meta RoCE 从立项之初就精准锚定了这一战略需求。 --- ### [05:45](https://www.youtube.com/watch?v=-KjG16c5fWU&t=345s) 动态演进下的软硬协同：AMD 与 Meta 的攻防拉锯 由于 Meta RoCE 是一套仍在快速迭代演进中的全新协议，James 向来自 AMD、主导硬件实现的 Balky 提问：面对一个规范不断变动、却又必须按期交出可用硬件实现的协议，到底是一种怎样的体验？ Balky 坦言，与一个不断进化的标准赛跑确实极具挑战。所幸 AMD 硬件的可编程性（programmability）发挥了决定性作用，为协议的动态调整提供了弹性空间。面对 Meta RoCE 庞大的特性集与丰富操作，AMD 采取了步步为营的策略：先剥离出核心重点——优先攻克拥塞控制机制与多路径传输，在一小部分关键操作子集上打磨成熟，随后再逐步向外叠加多平面（multiplane）等进阶特性。 Kingshuk 此时幽默地插话补充，这种参与制定未来标准的过程充满激情，但不可否认也是一个“移动的靶子”（moving target）。起初每次收到协议修改邮件时，心里难免会嘀咕“天哪，怎么又来一次变更”；但当你明白每一次推倒重来都是为了把系统打磨得更强大时，整个团队就真正开始享受这一创新蜕变的过程了。 Sandeep 则道出了背后的现实压力：当前 AI 领域的演进速度太惊人了，上层业务和用户的诉求几乎每隔几个月就彻底翻新一次。协议本身要进化，业务场景在剧烈膨胀，而最底层的硬件物理资源终究是有限的。Sandeep 打趣道，每当业务需要新功能时，自己就会跑去找 Balky 软磨硬泡：“Balky，能不能在你的硬件芯片里再多抠出一点点资源来？”而 Balky 每次都会无奈地翻出芯片规格书，扶一扶眼镜说：“行吧……我再回去仔细盘算盘算。” 现场随后深入探讨了硬件物理资源与协议设计的直接权衡： - **队列对（QP）与路径规模**：Meta 要求支持极高密度的 QPR 扩展能力，而硬件芯片内部用于维护连接状态的内存相当有限。 - **博弈折中后的落地规格**：经过双方在软硬件边界上的反复拉锯与优化，最终方案实现了支持 4K 规模的活跃 QP，并且在保持精细状态跟踪的同时，能以该规格稳定驱动多达 8K 条路径的调度。 --- ### [09:18](https://www.youtube.com/watch?v=-KjG16c5fWU&t=558s) 极简网络哲学：将智能彻底推向边缘 话题重新回到系统设计的顶层哲学。James 向 Arvind 询问关于 Meta 在演讲中提到的一个颠覆性理念：“将所有智能全部推向边缘端点（Push all the intelligence to the edge），保持网络传输管道（Tiny network pipes）尽可能绝对精简”。这种架构背后到底依赖怎样的端到端传输机制来落地？ Arvind 直截了当地指出，这归根结底取决于一个根本问题：**整套系统中究竟谁才真正懂得工作负载的意图（workload intent）？** 答案显然是位于网络边缘的计算节点（端点 endpoints）。中间的交换网络与传输织网（Fabric）永远不可能真正理解 AI 分布式训练的计算拓扑或阶段意图，只有端点最清楚。因此，架构的最佳策略不是在网络交换机里堆砌复杂的专用处理逻辑，而是让中间的网络 Fabric 保持极致简洁轻量，仅负责快速、精准地向边缘端点反馈网络的真实状态信息。只要端点拿到了足够的 Fabric 状态遥测，决策的控制权就完全可以交给端侧，由端侧自主决定如何调度、喷洒数据包和调节拥塞。 ### [10:09](https://www.youtube.com/watch?v=-KjG16c5fWU&t=609s) 细微路径概念与分层拥塞控制架构 Arvind Srinivasan 回顾了网络架构演进的底层逻辑。团队最初意识到，试图构建一个完全可编程的网络交换结构（programmable fabric）代价过高，会对基础设施造成不可承受的负担。因此，核心诉求演化为：如何从现有的网络结构中提取出必要的信息，使终端节点（endpoints）能够自主做出智能的负载均衡决策。基于这一目标，Meta 提出了“细微路径”（tiny paths 或 tiny pipes）的概念。终端仅需获取足以支撑负载均衡的关键指标（例如 RTT 和 ECN 标记），便能搭建起一套可规模化复制的基础构建模块。 这种设计使得网络能够在多个维度上协同扩展：既能支持横向扩展跨度上的 RTT 容忍度，也能纵向提升整体网络带宽，同时自然继承了系统的弹性容灾能力（resilience）。随后，Sandeep Nagaraja 补充指出了这一架构最核心的技术突破之一：拥塞控制是在“细微路径”级别上独立实施的，而非停留在传统的队列对（Queue Pair, QP）层级。这种创新的“分层拥塞控制”（hierarchical congestion control）赋予了端点极高的调度灵活性，并大幅跃升了整体传输效率。 ```mermaid flowchart TD accTitle: Tiny Path Hierarchical Congestion Control accDescr: 细微路径分层拥塞控制架构展示 subgraph QP_Layer["QP 抽象层"] QP["Queue Pair (QP)"] end subgraph Tiny_Path_Layer["细微路径层 (Tiny Paths)"] direction LR P1["Path 1<br/>拥塞控制 & 状态跟踪"] P2["Path 2<br/>拥塞控制 & 状态跟踪"] P3["Path 3<br/>拥塞控制 & 状态跟踪"] end subgraph Fabric_Layer["底层网络交换结构 (Standard Fabric)"] direction LR ECN["ECN 标记反馈"] RTT["RTT 往返时延测量"] end QP --> P1 QP --> P2 QP --> P3 P1 -. 监控.- RTT P2 -. 监控.- ECN P3 -. 监控.- RTT ``` - 核心诉求：避免重构底层网络可编程结构，转而依靠终端感知网络状态。 - 分层控制：将拥塞控制下沉至细微路径粒度，摆脱单个 QP 绑死整条物理流的限制。 - 扩展维度：实现带宽扩容、RTT 时延跨度适应以及网络弹性的多维协同演进。 --- ### [11:23](https://www.youtube.com/watch?v=-KjG16c5fWU&t=683s) Meta RoCE 演化中的工程权衡与硬件资源妥协 主持人 James Zeng 提出疑问：宏大的架构愿景落地绝非一蹴而就，背后必然充斥着艰难的设计权衡。Sandeep Nagaraja 回顾 Meta RoCE（简称 Meta-RoCE）的研发历程时坦言，除了纯技术难题，最大的挑战其实源于外部应用场景与技术格局的剧烈动态演变。最初系统主要面向通用规模化设计，但随着业务发展，多路径（multipath）与多平面（multiplane）网络织网的需求骤然提前，迫使团队优先支持复杂拓扑，将长距离传输（long distance）在大规模下的拥塞控制与可靠性机制提上日程。 在具体硬件实现层面，Meta 团队与 AMD 展开了深入的技术博弈。最初的硬件方案分配了更多的 QP 数量，而细微路径的数量相对较少；但面对现实的网络拥塞场景，团队果断做出取舍，选择削减支持的 QP 总量，以换取更多的细微路径资源，从而更精准地掌控各种网络异常场景。AMD 的 Balakrishnan Raman 进一步补充指出，网卡上的内存空间（memory constraint）是极其严苛的硬限制，双方在参数规模与片上资源之间反复拉锯。此外，如何合理暴露并精简拥塞控制参数也是关键权衡之一：既要提供高度优化的内置拥塞算法，又要留出充裕的参数空间，允许自定义算法进行细粒度微调。 - 架构优先级调整：优先支持多路径与多平面架构，延后单一维度的网络规模扩张。 - QP 与路径数量权衡：缩减 QP 规模，增加细微路径容量，以提升对复杂网络场景的调度适应能力。 - 片上资源瓶颈：在网卡有限的片上内存限制下平衡状态存储与路径追踪开销。 - 拥塞参数设计：兼顾开箱即用的原生算法性能与面向未来定制算法的可调谐性（tuning）。 --- ### [13:31](https://www.youtube.com/watch?v=-KjG16c5fWU&t=811s) AI 工作负载特异性与测试验证层面的范式转移 针对 Meta-RoCE 与诞生于传统超算（HPC）环境的 RoCEv2 的本质差异，Keysight Technologies 的 Kingshuk Mandal 深入剖析了 AI 时代网络协议验证的独特性。在 AI 工作负载中（尤其是排除掉对外推理服务的核心训练场景），网络流量以东西向（East-West traffic）为主，高度依赖机器间的紧密协同。这类业务对延迟抖动与时钟精度有着极为严苛的要求。 Kingshuk 指出，尽管在底层代码实现上“C 语言写出来的代码在任何地方都是 C 语言”，但网络协议在宏观设计与验证范式上面临着天壤之别。普通互联网流量侧重于突发式的南北向浏览交互，对单包时延的敏感度较低；而 AI 通信对精度的挑战提升到了全新量级。为确保 Meta-RoCE 这种前沿协议在极限拥塞、微秒级时延抖动下的鲁棒性，Keysight 开发了专门面向 AI 网络的测试套件（如测试套件 KAI），以满足远比传统网络更加严酷的验证标准。 - 流量模式差异：AI 训练场景几乎完全由东西向机器间流量主导，与传统南北向互联网流量机制迥异。 - 严苛敏感性：AI 集合通信对微秒级延迟、时序精度及丢包具备极高的敏感度。 - 测试技术重塑：在验证环境中引入更高精度的网络压力仿真与多维度指标监控。 --- ### [15:22](https://www.youtube.com/watch?v=-KjG16c5fWU&t=922s) 拥抱现有网络基础设施：零门槛交换机复用 面对现场关于“升级至 Meta-RoCE 是否需要推倒重演现有交换机与路由器”的实际担忧，Arvind Srinivasan 给出了极为明确的定心丸：Meta 绝不会因为引入了一种新的传输协议或网卡，就去彻底重构整个数据中心网络架构。Meta-RoCE 在设计之初就立足于完全兼容并复用现存的基础设施，不强制要求交换机具备任何专有私有功能。 尽管在早期愿景讨论中曾提及，如果交换机硬件能够原生支持直接数据包喷洒（packet spray）等多路径特性将会对系统有所助益，但 Arvind 和 Sandeep 明确强调，协议不会对此产生强依赖。Meta-RoCE 在最基础层面上仅要求网络设备具备业界通用的标准功能：支持基于标准 ECMP（等价多路径路由）的转发能力，以及标准的 ECN（显式拥塞通知）标记能力。“只要交换机支持这两项基本能力，Meta-RoCE 就能完美运行；如果交换机能提供更多高级特性固然锦上添花，但我们绝不指望它。” | 评估维度 | 交换机基础功能（必需） | 交换机进阶特性（可选/不依赖） | |:--- |:--- |:--- | | **路由转发机制** | 标准 ECMP（等价多路径路由） | 交换机级直接数据包喷洒（Direct Packet Spray） | | **拥塞反馈机制** | 标准 ECN（显式拥塞通知）打标 | 交换机内部遥测（INT）或私有流控协议 | | **设备兼容性要求** | 市面所有标准白盒/商用交换机均支持 | 专有可编程交换架构芯片 | | **基础设施影响** | 零改动复用现有集群 Fabric | 无需进行硬件级整体翻新迭代 | --- ### [16:42](https://www.youtube.com/watch?v=-KjG16c5fWU&t=1002s) 开源规范发布与 OCP 峰会技术深度披露 在问答环节的尾声，James Zeng 邀请嘉宾剧透即将在几个月后的 OCP（Open Compute Project）全球峰会上呈现的内容。Sandeep Nagaraja 宣布，Meta 正在全力推进 Meta-RoCE 协议规范的完全开源工作。此举旨在与开源社区建立深度的双向互动，解答行业疑问，并在长期维度上推动跨厂商硬件生态的敏捷、平滑开发。 Sandeep 透露，团队自项目启动之初就同步打造了一套完整的“参考实现”（reference implementation）。这套参考实现不仅在前期作为 Meta 与 AMD 联合研发、验证生产级硬件实现的黄金标尺，未来更将直接开源，作为整个技术社区理解并适配该协议的核心指南。在即将到来的 OCP 峰会上，团队将带来更为细致的技术汇报，包括对 Meta-RoCE 协议机制的逐层深度解析、针对开源参考实现的实机演示（Demo），以及指导开发者如何基于该方案进行二次开发的落地实践指南。随后，主持人 James Zeng 向参与分享的 Meta、Keysight 及 AMD 专家表达了由衷谢意，圆满结束了本次联合技术问答。 ## Notable quotes > [00:03](https://www.youtube.com/watch?v=-KjG16c5fWU&t=3s) 究竟是什么促使 Meta 决定彻底抛弃现有的 RoCEv2，转而从零打造一套全新的 Meta RoCE？为什么大家不能继续沿用传统的 RoCEv2 呢？ — James Zeng > [00:03](https://www.youtube.com/watch?v=-KjG16c5fWU&t=3s) 依照 InfiniBand 贸易协会（IBTA）发布的传统 RoCE 标准，协议本身根本无法支持多路径传输（multipathing）——在传统规范下，一旦出现数据包乱序（out of order），整个传输连接基本上就直接瘫痪崩溃。 — Arvind Srinivasan > [01:05](https://www.youtube.com/watch?v=-KjG16c5fWU&t=65s) 要想真正兑现多路径的收益，系统必须做出并严格信守“四大承诺” — Kingshuk Mandal > [05:45](https://www.youtube.com/watch?v=-KjG16c5fWU&t=345s) 面对一个规范不断变动、却又必须按期交出可用硬件实现的协议，到底是一种怎样的体验？ — James Zeng > [05:45](https://www.youtube.com/watch?v=-KjG16c5fWU&t=345s) 起初每次收到协议修改邮件时，心里难免会嘀咕“天哪，怎么又来一次变更”；但当你明白每一次推倒重来都是为了把系统打磨得更强大时，整个团队就真正开始享受这一创新蜕变的过程了。 — Kingshuk Mandal > [05:45](https://www.youtube.com/watch?v=-KjG16c5fWU&t=345s) Balky，能不能在你的硬件芯片里再多抠出一点点资源来？ — Sandeep Nagaraja > [09:18](https://www.youtube.com/watch?v=-KjG16c5fWU&t=558s) 整套系统中究竟谁才真正懂得工作负载的意图（workload intent）？ — Arvind Srinivasan > [09:18](https://www.youtube.com/watch?v=-KjG16c5fWU&t=558s) 将所有智能全部推向边缘端点（Push all the intelligence to the edge），保持网络传输管道（Tiny network pipes）尽可能绝对精简 — James Zeng > [10:09](https://www.youtube.com/watch?v=-KjG16c5fWU&t=609s) 拥塞控制是在“细微路径”级别上独立实施的，而非停留在传统的队列对（Queue Pair, QP）层级。 — Sandeep Nagaraja > [13:31](https://www.youtube.com/watch?v=-KjG16c5fWU&t=811s) C 语言写出来的代码在任何地方都是 C 语言 — Kingshuk Mandal > [15:22](https://www.youtube.com/watch?v=-KjG16c5fWU&t=922s) 只要交换机支持这两项基本能力，Meta-RoCE 就能完美运行；如果交换机能提供更多高级特性固然锦上添花，但我们绝不指望它。 — Arvind Srinivasan ## Mentioned Meta, Keysight Technologies, AMD, James Zeng, Arvind Srinivasan, Kingshuk Mandal, Sandeep Nagaraja, Balakrishnan Raman, RoCEv2, Meta RoCE, InfiniBand 贸易协会 (IBTA), ECMP, ECN, Queue Pair (QP), OCP (Open Compute Project) ## Video description Moderator - James Zeng, Meta Panelists - Arvind Srinivasan, Meta Kingshuk Mandal, Keysight Technologies Sandeep Nagaraja, Meta Balakrishnan Raman, AMD Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=-KjG16c5fWU)

[0:03] James Zeng: 究竟是什么促使 Meta 决定彻底抛弃现有的 RoCEv2，转而从零打造一套全新的 Meta RoCE？为什么大家不能继续沿用传统的 RoCEv2 呢？ [0:03] Arvind Srinivasan: 依照 InfiniBand 贸易协会（IBTA）发布的传统 RoCE 标准，协议本身根本无法支持多路径传输（multipathing）——在传统规范下，一旦出现数据包乱序（out of order），整个传输连接基本上就直接瘫痪崩溃。 [1:05] Kingshuk Mandal: 要想真正兑现多路径的收益，系统必须做出并严格信守“四大承诺” [5:45] James Zeng: 面对一个规范不断变动、却又必须按期交出可用硬件实现的协议，到底是一种怎样的体验？ [5:45] Kingshuk Mandal: 起初每次收到协议修改邮件时，心里难免会嘀咕“天哪，怎么又来一次变更”；但当你明白每一次推倒重来都是为了把系统打磨得更强大时，整个团队就真正开始享受这一创新蜕变的过程了。 [5:45] Sandeep Nagaraja: Balky，能不能在你的硬件芯片里再多抠出一点点资源来？ [9:18] Arvind Srinivasan: 整套系统中究竟谁才真正懂得工作负载的意图（workload intent）？ [9:18] James Zeng: 将所有智能全部推向边缘端点（Push all the intelligence to the edge），保持网络传输管道（Tiny network pipes）尽可能绝对精简 [10:09] Sandeep Nagaraja: 拥塞控制是在“细微路径”级别上独立实施的，而非停留在传统的队列对（Queue Pair, QP）层级。 [13:31] Kingshuk Mandal: C 语言写出来的代码在任何地方都是 C 语言 [15:22] Arvind Srinivasan: 只要交换机支持这两项基本能力，Meta-RoCE 就能完美运行；如果交换机能提供更多高级特性固然锦上添花，但我们绝不指望它。

[0:03] Arvind Srinivasan: just kind of dies [1:05] Kingshuk Mandal: promises [1:05] Kingshuk Mandal: elephant flow [1:05] Kingshuk Mandal: mice flows [4:12] Sandeep Nagaraja: scale-out [4:12] Sandeep Nagaraja: scale-across [9:18] Kingshuk Mandal: tiny path [9:18] Kingshuk Mandal: hierarchical congestion control [17:06] smoother and faster development

[0:03] Hey, great talk uh about the meta rocky. [0:06] So maybe we'll start I will start with [0:08] you Arvin like what makes meta decide [0:11] that we need to build something [0:13] completely new? Why we can't just like [0:16] keep using uh Rocky V2? [0:18] >> Yeah, it's a great question. So as when [0:20] we started to scale our clusters, we [0:23] soon found out that they're going to be [0:24] hot spots in the network because you [0:26] need to build larger scale networks and [0:28] then we started looking at Rocky more [0:30] deeply and if I just look at traditional [0:32] Rocky the standard Rocky as published by [0:34] IBDA it doesn't allow you to do [0:37] multipathing anytime you get packet out [0:39] of order just kind of dies. So then we [0:41] started looking okay what do we do to [0:43] kind of add this feature into the TR and [0:45] Rocky we soon found out well the headers [0:47] just doesn't allow you to do it. we need [0:49] to do something different. Okay, then [0:52] that's what led us to to the point, [0:54] okay, we needed a out of order delivery. [0:56] We need out of order uh uh packets. [0:59] Rocky just doesn't work. Let's do [1:01] something different. Then there were all [1:02] implementation challenges from there [1:04] onwards. [1:05] >> That's great. And then maybe next [1:07] question for you uh I multipass I saw in [1:12] uh said in the in just now and then in [1:14] the uh in the talk is a core feature. [1:18] Um I think you also mentioned that you [1:21] need to test based both the happy pass [1:23] and then the the unhappy pass. Um can [1:26] you walk us through a little bit more [1:28] detail about how do you validate like [1:30] multipass uh performance? [1:32] >> Okay. Uh that's a big one. Multipath [1:36] sounds like a single feature but it's [1:38] it's one of the significant feature and [1:41] now to implement multipath successfully [1:44] you need to make multiple promises and [1:47] you need to keep them. So let me explain [1:49] it with an with an example. Let's say [1:52] there is an elephant flow and you want [1:53] to deliver it over certain different [1:55] multipaths. The first promise which you [1:58] need to make and keep is you are really [2:00] able to create multipath with enough I [2:02] would say entropy differentiation. You [2:05] can break an elephant flow into multiple [2:07] mice flow but if you are not creating [2:09] enough entropy differentiation your flow [2:12] will reach the destination but again [2:14] through a single path not through [2:15] multipath. So that's the first promise [2:17] which you need to keep and validate. All [2:19] right let's say you have done that and [2:21] your flow has reached the destination. [2:23] Well, the destination is acknowledging [2:25] you. The second promise is that all your [2:28] acknowledges also has to come through [2:30] multipath. Your flow reaches the [2:33] destination through multipath but the [2:34] acknowledgements are coming through a [2:36] single path. Your RT and etc. All those [2:38] things will get messed up. So that's the [2:40] second promise and we need to validate [2:42] the same thing as well. Third what I can [2:46] think of is which I touched upon during [2:48] the talk is about your bookkeeping. Is [2:50] your bookkeeping part multipath? During [2:53] the talk, I gave an example that the RTO [2:55] is getting shared and it's not really [2:56] truly multip multipath bookkeeping [2:59] happening there. So there's a set of [3:00] promises related to this bookkeeping as [3:02] well. And the fourth section that I can [3:05] think of is like say purpose of [3:08] multipath. One of the thing is [3:09] efficiency as well as redundancy. So if [3:12] one path goes down, are you able to [3:14] switch to the next path? If one path [3:16] goes down or your load going higher, are [3:18] you able to add more active paths into [3:20] your multipart multipath pool. So that's [3:23] another set of promises you need to [3:25] validate. So netn net there are many [3:27] promises needs to be kept to have a [3:29] successful multipath and those are the [3:31] promises that we validate. [3:33] >> Mhm. Okay. Maybe a follow-up question [3:35] like uh I think you uh in your talk you [3:39] mentioned like there's two paths but in [3:41] the actual test [3:43] how many paths do you actually [3:45] physically need to test like this to to [3:48] verify that things works well [3:51] >> well [3:51] >> two three four or even more [3:53] >> no I think it's it's even even more I [3:56] think so far what we have done I think [3:58] we have tested up to eight but metari [4:01] supports uh way more way more path But [4:03] as we go more towards scalable scalable [4:06] situations, yes, we'll increase the [4:08] speed and the feed and the dimensions of [4:10] the paths uh in future. [4:12] >> Okay, great. Um we just talked about [4:14] multipass but in fact like one of the uh [4:18] key advantage of metaroki is that it uh [4:21] it handles both uh scale out and scale [4:24] across that means that we also can do [4:27] like a long distance right um maybe this [4:30] question for you uh Santiv can you [4:32] elaborate a little bit about like how me [4:35] adapts to the long distance um [4:37] communication what differentiates it [4:39] from different protocols [4:41] >> yeah sure I [4:43] uh to what Kingshuk just mentioned, we [4:45] are able to uh scale across uh across [4:49] long distance RTDs and we've tested up [4:51] to uh 1.5 milliseconds to what uh Balky [4:55] had presented in the talk and we're able [4:57] to achieve this by dynamically modifying [5:00] our uh congestion windows and path count [5:03] right so we have tested actual number of [5:06] individual paths up to eight to what [5:07] kings point but within this within the [5:10] available network we can create more [5:11] paths U along the same uh network fabric [5:15] etc. So we went up to 72 paths there [5:18] that allowed us to spray packets more [5:20] across longer distances and achieve [5:22] this. Um so we were able to do this in a [5:25] of course in a lab network with fiber [5:27] spools etc etc. uh but soon to what Omar [5:31] was suggesting etc uh our data centers [5:33] are growing and are going to be longer [5:35] distances and we need to have end to end [5:38] RDMA communication across these [5:39] distances uh and metari is looking ahead [5:42] in those terms and u trying to solve [5:44] those [5:45] >> mhm great and then um and we all know [5:48] that metari is like a new protocol we [5:50] just talked about it in detail there's [5:52] more we're going to talk about in OCP [5:54] but like bucket like you found AMD and [5:57] then you work with Meta working on this [6:00] metari implementation. How does it feel [6:02] to work on some protocols that is still [6:04] evolving like I I imagine a lot of [6:07] things is changing but we need some [6:10] working implementation. [6:13] Yeah. So um so obviously it's an [6:16] evolving protocol. So quite a bit of [6:18] things were changing. So the [6:20] programmability helped a lot with [6:22] respect to implementing an evolving [6:24] protocol. Also there are challenges with [6:26] respect to you know there are a lot of [6:28] features with the metari protocol and [6:30] also it supports a lot of operations. Um [6:34] so you have to have a balance there [6:36] right because you want to first prove [6:38] that whatever important aspects of the [6:40] metari protocol are evaluated. Uh so we [6:43] obviously uh we implemented the we [6:46] wanted to focus on the congestion [6:47] control aspects of the metari protocol [6:49] the multipaththing. Um so we took a [6:51] subset of operations and made sure those [6:54] aspects are uh properly addressed and [6:56] then we kept adding more features uh [6:59] multiplane and things like that. [7:02] Yeah. So u so it's it's it's a challenge [7:05] where like I mean the programmability [7:07] helped a lot in you know handling this. [7:09] >> Just yeah just to add on him you asked [7:12] about how does it feel. So firstly it [7:15] feels to be great to be part of [7:16] something which is evolving and new. So [7:18] that's an excitement. Yes, it has been a [7:21] moving moving target in our experience [7:23] as well. But the thing is when you [7:25] realize that why that move is and at the [7:28] end if you realize that move is for a [7:30] betterment well initially you might say [7:32] oh god one more change but when you [7:34] realize that's a change for better you [7:36] start enjoying it. So I would say that's [7:38] what that's what my our experience has [7:40] been. [7:41] >> Yeah. Go ahead. Yeah. [7:41] >> No just to add to that I think uh we all [7:43] know how fast-paced uh AI landscape is [7:46] right. it's changing. Uh customers are [7:48] our customers are demanding something [7:50] different every uh at every turn. U so [7:53] that's where some of the challenges was [7:55] coming right. We have the we have this [7:56] new protocol which itself is evolving [7:58] and getting perfected but also the [8:01] landscape itself was changing right the [8:03] scale the requirement we would jigsaw [8:05] the features here and there. Ultimately [8:07] hardware only has so much resources. [8:09] every time uh something changes I would [8:12] go back to Balky and like Balky can you [8:13] find a little bit more resource in your [8:15] hardware somewhere and he he would take [8:16] out his specs and okay I'll think about [8:18] it [8:18] >> but maybe I'm I I I want to ask you a [8:21] question is there any time that hey we [8:24] have some limited resources or maybe we [8:26] have some hardware constraints and that [8:30] uh in the obvious way influence how we [8:32] design metari [8:34] >> yeah so uh so obviously we need to you [8:37] know with the metari protocol I mean [8:39] there is always a balance of how much [8:40] state you want to maintain but at the [8:42] same time you also want to handle all [8:44] the congestion aspects and things of the [8:46] metari protocol. So we uh we wanted to [8:50] you know support a very high scale [8:51] because meta wanted to have a very high [8:53] QPR scale and at the same time yeah [8:55] there is also this you know challenge of [8:58] how much state you can maintain. So [9:00] those are all the constraints that we [9:02] you know went back and forth with the [9:04] meta and we designed we actually we [9:06] scaled up quite a bit. we have still 4K [9:09] uh Q per scale and with 8K you know [9:12] parts that can be you know supported [9:14] with that that much of 4K QPS. [9:18] >> Okay, maybe Arvin I want to ask you uh [9:20] the next question. So I think you have [9:22] this concept of like let's push all the [9:25] uh intelligence to the edge right and [9:28] that's uh let's keep the transport as [9:30] simple as uh to keep the network as [9:32] simple as possible. Um you even have [9:35] this term like tiny network pipes in [9:37] your talk. Um what is the specific like [9:40] transport mechanism that you will need [9:43] in order to realize that a great [9:46] question. So the whole thing was driven [9:47] by where should intelligence really look [9:49] live and it all comes down to who knows [9:52] the intent of what the workload is [9:54] looking for right. So the intent only [9:57] the endpoints know the intent only the [9:58] endpoint understand the workload but can [10:01] we get enough information from the [10:03] fabric so that the endpoints can make [10:04] the decision. So then what do you need [10:06] from the fabric? Well we can't make the [10:07] fabric pro we don't want to go down the [10:09] path of programmable fabric that's too [10:11] much to take on infrastructure is [10:13] another thing that we want to go. So [10:15] then the point was what can you extract [10:17] from the fabric so that the end points [10:19] can make and that was this concept of [10:21] paths or tiny pipes that I talked about. [10:23] So we just want to get enough [10:25] information for this guys to make load [10:27] balancing decision. Oh then okay concept [10:30] of a path okay can I scale if I need to [10:32] know RT [10:34] RT is one thing or I need ECN I need to [10:36] you know that's that's all it ended up [10:38] being building it up and became a [10:40] building block with we can stamp it as [10:42] we go along and it also helped us scale [10:45] and when you start scaling we scale on [10:47] bandwidth you scale on RT the scale [10:48] across is an RT scale if you want to [10:51] more put more more bandwidth more so [10:53] it's like you can have multi-dimensional [10:55] scale that we went [10:57] along the way came resilience also that [10:59] comes along the way. So it was a kind of [11:00] buildup of all of that [11:02] >> and just to add to what Arvin just said [11:05] this tiny path one of the I think big [11:07] innovation here is that the congestion [11:09] control is on the tiny path it's not at [11:11] the QR level so there is a hierarchical [11:14] congestion congestion control and you [11:16] can control it at the tiny path level [11:18] adds a lot of I would say flexibility [11:20] and improves the efficiency quite [11:22] significantly [11:23] >> but I'm I'm pretty sure this design is [11:26] not like straight like we have a vision [11:28] and then we just implemented There must [11:30] be a lot of like design decisions that [11:32] you need to make hard hard tradeoffs. [11:34] Maybe Sep like one question for you is [11:37] like what would you consider to be the [11:39] hardest design decisions that you need [11:40] to make in your meta meta rocket [11:43] journey? I think uh uh outside of the I [11:48] would say technical I think the biggest [11:50] challenge like I already mentioned was [11:51] the uh uh the landscape itself changing [11:54] right we initially began [11:58] um doing for scale and but then we [12:00] realized we have to support more [12:03] multiath and multiplane fabric scale can [12:05] come later uh we were thinking of long [12:08] distance uh but then we had to think of [12:10] more network scenarios for congestion [12:12] and reliability over long distance etc. [12:14] So we had to constantly make uh uh [12:17] trade-off decisions with available [12:19] hardware etc. [12:21] uh we initially had uh uh more QPS and [12:26] less paths for example right and then we [12:28] had to make a trade-off decision to have [12:30] fewer keep QPS but more paths that would [12:32] allow us to um manage the network [12:35] scenarios better right so these were [12:37] interesting uh uh uh sort of uh [12:40] decision- making that both AMD and us [12:43] along with Arvin etc we had to sort of [12:45] discuss if this is the right thing to do [12:48] uh Mali did you have more examples like [12:51] the ones that we had to make trade-offs. [12:54] >> Yeah. So, uh with respect to trade-offs, [12:56] Yeah. So, um so yeah, obviously the that [12:59] scale um the memory constraint uh so we [13:03] had to you know go back and forth uh [13:05] with respect to that that that [13:07] particular trade-off. Also um um with [13:11] respect to you know the congestion [13:14] control parameters itself like what are [13:16] all the parameters that we want to you [13:18] know have it tuned so that you can you [13:20] know have the a built-in congestion [13:22] control also you can have a custom [13:24] congestion control that can tune those [13:26] parameters. So these are all some of the [13:28] you know constraints that we went [13:29] through. [13:31] >> Okay. Um I think one of the key um [13:36] origin of metari as avenue you said is [13:39] that we are looking at this AI workload [13:41] right and then we need to design a [13:43] protocol that fits this AI uh fits this [13:46] AI AI workload and that is very [13:49] different from traditional like meta uh [13:52] like rocky v2 was originated from the uh [13:55] supercomputer like uh HPC community. Um [13:58] maybe a question for you uh king should [14:00] like um like how is validating this mega [14:04] rocket in this environment [14:06] uh different from uh from from other [14:09] other protocols? I think in your talk [14:11] you mentioned like this KI right like [14:14] you have this sweet can you elaborate a [14:16] little bit more about how you design how [14:18] you envision that uh testing protocols [14:20] in the AI era will be different? Well, [14:23] it's it's it's quite significantly [14:25] different uh just because of the domain. [14:27] We are not talking about internet [14:29] browsing traffic. We are talking about [14:31] an AI workload traffic where most [14:33] majority if you take out inference for [14:36] the time being majority is is east west [14:39] traffic and it's very machine sensitive [14:41] and uh latency and the precision becomes [14:44] uh becomes so significant. So that [14:46] itself is a big big learning that if you [14:48] have to test a technology for this [14:50] particular domain that's brings in I'd [14:53] say requirements are more stricter. Uh [14:55] the protocols are is pro protocols are [14:58] there are challenging protocols every [14:59] everywhere but this set of set of [15:02] protocols handles a different level of [15:04] precision challenge that we do not see [15:06] in general other internet type of uh [15:09] type of pro protocol here. So I guess [15:11] that's the that's the significant [15:12] different that we that we at the code [15:14] level where C program is C program [15:17] everywhere. Uh but from design's [15:19] perspective yes this is quite quite [15:21] different. [15:22] >> Mhm. Okay. Um I think we have some [15:25] people ask question like hey we already [15:28] have a lot of like switches and routers [15:30] deployed. If I want to switch to uh [15:33] metari can we reuse those switches [15:36] routers like question for you. [15:39] No, we don't expect any uh you know we [15:43] have to work we work with existing [15:44] infrastructure that is the uh thing we [15:47] don't expect to redo redo our fabric [15:49] just because we added a new transporter [15:51] a new nick so no we do not expect uh [15:54] overhaul of any kind of uh switches or [15:57] anything yeah that's our expectation [15:59] that's our goal towards it but you I [16:02] think in your talk you do mention you [16:04] did mention that we hopefully the switch [16:07] will support direct packet spray you [16:09] right so multi passing so [16:12] >> and uh so uh our protocol will work [16:14] without any touches any anything uh [16:16] touch we don't need any special feature [16:18] in the switch we rely on basic standard [16:20] features that exist in every possible [16:21] switch exist in the in the market [16:23] >> u yes if switches does more work it will [16:27] help us but we are not counting on it [16:28] let's put it that way [16:30] >> as long as the switch is supporting this [16:32] ECN marking and then ECMP based [16:35] forwarding metro is good [16:36] >> yeah that's [16:37] >> nothing else at the fundamental level is [16:39] needed there. If it does more, great. [16:41] But [16:42] >> okay. So can you maybe one of you that [16:45] that we are approaching on time maybe [16:47] one of you can give us a sneak peek of [16:48] what will happen in the OCP in a few [16:51] months. [16:52] >> Um sure I think uh we have we will open [16:55] source the specification right uh we are [16:57] in process of doing that. uh our goal [17:00] there is to have interact with more of [17:02] the community and enable answer [17:04] questions but even even in the long term [17:06] enable uh smoother and faster [17:08] development right so as we presented [17:11] very in the very beginning we had in [17:13] mind the reference implementation which [17:15] would not only help us work with AMD and [17:18] develop uh the uh real implementation [17:21] but also it was always kept in mind that [17:23] we will open source this someday and the [17:25] reference implementation will guide the [17:27] community right so at OCP [17:29] uh we're more than happy to uh walk you [17:31] through the details of this. We have a [17:34] presentation very similar to what was [17:35] done here but goes more in depth with [17:38] the protocol. We also have a specific [17:40] talk that dives deep into the reference [17:42] implementation how it can be used a demo [17:45] on that etc. So uh it'll it'll be more [17:47] elaborate and interactive. Yeah. [17:49] >> Great. And well, thank you all for our [17:53] speakers to um uh to for great talks and [17:56] the Q&A. And thank you everyone here.