OpenAI Deploys MRC and SRv6 Protocols to Eliminate AI Network Outages
@Scale
OpenAI's Multipath Reliable Connection and static SRv6 routing maintain cluster throughput during hardware failures by spraying packets across host-selected paths and bypassing network drops within a single round-trip time.
## Background
Published on YouTube by @Scale on August 27, 2026, this presentation features OpenAI's Jitendra Padhye presenting at the SCCC event. The video description highlights OpenAI's three-pronged approach to eliminating tail latency in large-scale AI pretraining using a new RDMA transport protocol called MRC paired with static SRv6 source-routing.
## TL;DR
Jitendra Padhye outlines how OpenAI overcomes tail-latency bottlenecks and fabric failures during large-scale synchronous AI training across tens of thousands of GPUs. Traditional RoCE networks bound connections to single paths, causing ECMP hash collisions and capped utilization. MRC (Multipath Reliable Connection) solves this by spraying packets across all available healthy network paths, leveraging out-of-order memory placement via virtual addresses, and using static SRv6 source routing. This host-driven design allows training jobs running across 75,000 GPUs to seamlessly ride out silent switch failures and link flaps without interrupting workloads.
## Atmosphere & style
The presentation blends technical depth with approachable, self-deprecating humor, anchored by Padhye's lighthearted opening joke about his son's interpretation of 'MRC'. The tone is pragmatic, grounded, and authoritative, reflecting real-world operational insights from managing frontier AI infrastructure. Listening to the talk feels like an insider look into modern hyperscale engineering, where high-stakes cluster stability is transformed from a stressful on-call nightmare into a routine, resilient system behavior.
## Key takeaways
- [00:03](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=3s) MRC stands for Multipath Reliable Connection, a data center scale-out transport protocol developed by OpenAI, Microsoft, AMD, Broadcom, and Nvidia to replace Infiniband and RoCE.
- [00:42](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=42s) Synchronous AI training efficiency across thousands of GPUs is governed entirely by worst-case tail latency rather than average performance.
- [05:41](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=341s) Standard RoCE fabrics bound to single static paths rely on load-unaware ECMP, creating hash collisions that cap theoretical fabric utilization at around 66%.
- [07:22](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=442s) MRC eliminates network hotspots by spraying packets across all available healthy fabric paths.
- [07:22](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=442s) To handle out-of-order packet arrivals resulting from packet spraying, MRC embeds Virtual Memory Addresses (VA) in headers so receiving GPU NICs can place payloads directly into memory.
- [10:53](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=653s) Static source routing is implemented via SRv6 using micro-SIDs, leaving the network switches completely passive without dynamic routing control planes.
- [10:53](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=653s) When intermediate link failures occur, switches drop packets without in-network rerouting, allowing end hosts to detect drops and adjust paths within 1 RTT.
- [16:52](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=1012s) End hosts maintain active QPs, backup QPs, and a demoted set for bad paths to instantly reroute around packet loss without stalling workloads.
- [16:52](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=1012s) Network operators can apply regex denylists on source hosts to direct traffic away from targeted hardware ahead of scheduled maintenance.
- [18:11](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=1091s) In production environments with 75,000 GPUs, MRC successfully rode out silent 'zombie' router failures and optical transceiver link flaps without interrupting training runs.
## Detailed walkthrough
Here is the detailed walkthrough of the first part of the presentation:
[00:03](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=3s) Introduction and the Meaning of MRC
Jitendra Padhye from OpenAI opens his talk following an inspiring introduction by previous speaker Omar, joking that his teenage son provided a reality check during a dry run by suggesting "MRC" actually stands for "the most ridiculous concept." Padhye clarifies that MRC stands for **Multipath Reliable Connection**, a specialized scale-out transport protocol for AI training within the data center created through a massive joint effort between OpenAI and industry partners including Microsoft, AMD, Broadcom, and Nvidia. Designed to replace traditional Infiniband and RoCE (RDMA over Converged Ethernet) setups, MRC targets data center-level scale-out interconnects rather than host-internal scale-up fabrics. Its core purpose is to make giant AI training clusters robust against fabric failures and eliminate network bottlenecks.
- MRC stands for **Multipath Reliable Connection**.
- It is a data center scale-out transport protocol designed to replace RoCE and Infiniband.
- Developed collaboratively by OpenAI, Microsoft, AMD, Broadcom, and Nvidia.
- Relies on three foundational architectural pillars: **packet spraying**, **source routing**, and **multi-planar networks**.
[00:42](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=42s) The Tail Latency Problem in Synchronous AI Training
Padhye breaks down the mechanics of large-scale ML training, emphasizing that thousands of GPUs operate synchronously step-by-step. Each training iteration alternates between local GPU computation and collective data exchanges (such as AllReduce). Because GPUs cannot progress to step $N+1$ until every single GPU finishes step $N$, the overall system efficiency is entirely dictated by tail latency. In standard cloud workloads, P95 or P99 metrics might suffice, but in synchronous AI pretraining, any single congested link or packet stall freezes the entire cluster. If a failure occurs, the whole step must be discarded and retried.
- ML pretraining requires synchronized lock-step execution across thousands of GPUs.
- Overall efficiency depends strictly on worst-case tail latency rather than average latency or P95/P99 metrics.
- A single congested link stalls all GPUs, preventing the cluster from starting the next step.
- The sole objective of the transport network is to "keep the job stepping."
[05:41](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=341s) Limitations of Standard RoCE Fabrics
Standard RoCE Queue Pairs (QPs) assign all packets within a flow to a single network path because RoCE cannot natively tolerate out-of-order packet delivery or packet loss. In a multi-tier Clos network, running many QPs simultaneously inevitably causes hash collisions when using traditional, load-unaware Equal-Cost Multi-Path (ECMP) routing. When multiple QPs randomly collide on the same physical link, that link becomes a bottleneck while neighboring links sit idle, capping theoretical fabric utilization to roughly 66%.
- Standard RoCE QPs bind a connection to a single static path.
- Standard ECMP routing is load-unaware, creating hash collisions when multiple QPs pick the same path.
- Hash collisions create localized hot spots, limiting total network throughput even when idle capacity exists elsewhere.
- Theoretical fabric utilization bounds for uncoordinated single-path ECMP sit around **66%**.
[07:22](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=442s) MRC Packet Spraying and Out-of-Order Packet Handling
MRC solves hash collisions by taking the single QP requested by the application and spraying its constituent packets across all available healthy paths in the network. Because packets take different paths, they arrive out of order due to tiny differences in wire lengths or transient delays. Standard RoCE degrades under out-of-order arrivals because its headers lack direct destination buffer offsets, triggering unnecessary retransmissions. MRC solves this by embedding the explicit Virtual Memory Address (VA) into every packet header, allowing the receiving GPU NIC to immediately place incoming out-of-order payloads directly into the correct location in memory.
- MRC sprays packets from a single QP across all available fabric paths to eliminate hotspots.
- Packet headers carry direct Virtual Memory Addresses (VA) to handle out-of-order arrivals gracefully.
- The receiving NIC places out-of-order payloads into memory immediately as they arrive.
- A completion signal is generated only once all packets forming the full buffer (e.g., packets 1 through 5) have landed.
[10:53](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=653s) Static Source Routing via SRv6
Instead of relying on dynamic in-network ECMP, MRC leverages **SRv6 (Segment Routing over IPv6)** to implement static source routing. The source host maintains full knowledge of the topology and explicitly encodes the exact path of switches and routers into the packet header using micro-SIDs (16-bit router identifiers). As a packet traverses the network, each switch reads its designated micro-SID, shifts the destination address left by 16 bits, and forwards the packet to the next predetermined hop. The fabric network remains entirely passive and static; if a link along the route fails, the switch simply drops the packet, leaving retransmission and path selection entirely to the end hosts.
- Packets use SRv6 with up to six **16-bit micro-SIDs** embedded inside a 32-bit locator structure.
- Switches perform static forwarding and bit-shifting without maintaining dynamic control planes or state tables.
- If an intermediate link fails, switches drop the packet immediately without attempting in-network rerouting.
- End hosts detect drops and deterministically steer traffic away from bad paths within **1 round-trip time (RTT)**.
- SRv6 allows end hosts to run deterministic loopback probes (e.g., source $\rightarrow$ T0 $\rightarrow$ T1 $\rightarrow$ T0 $\rightarrow$ source) to continuously monitor network path health.
[16:52](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=1012s) Dynamic Path Management and Preventive Maintenance via Denylists
The core beauty of static source-routing combined with Multi-path RDMA Transport (MRC) lies in the host’s total awareness of the network environment. The source host maintains full knowledge of the static physical topology and dynamically tracks which paths are currently alive and performing well. To illustrate how path selection operates in practice, consider a network topology offering 128 potential end-to-end paths:
1. **Active Paths:** The host creates 32 active Queue Pairs (QPs) to immediately spray and balance traffic across 32 available paths.
2. **Backup Pool:** It maintains 32 backup QPs in reserve while leaving the remaining 64 potential paths unassigned.
3. **Loss Handling & Probing:** If a path experiences packet loss, the host immediately demotes that QP to a designated "bad path" set for continuous probing and monitoring.
4. **Instant Promotion:** A spare QP from the backup pool is promoted to active duty instantly—without waiting for pre-testing—ensuring unmanaged flow degradation never stalls the host.
Beyond reactive recovery, this approach enables proactive network management. Because path selector values and micro-ETs (Ethertypes/encapsulations) are fully mapped, network operators can schedule maintenance events without impacting active workloads. Engineers can program regular expressions directly into source hosts to define "denylists." When specific switches or links are slated for maintenance, these regex patterns ensure that traffic completely avoids the affected infrastructure ahead of time, preventing any artificial drops or latency spikes during router servicing.
[18:11](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=1091s) Real-World Resiliency: Riding Out Zombie Routers and Link Flaps
In production environments, MRC has repeatedly demonstrated its ability to prevent network anomalies from disrupting massive AI training runs. During a training job utilizing approximately 75,000 production GPUs, a Tier 1 (T1) switch experienced a "zombie" failure—a scenario where the router remains powered on and attempts to forward traffic, yet silent corruption or drop behavior occurs with zero indication in standard telemetry. Despite impacting 25% of the active QPs and dropping roughly half a million packets before system-wide detection, MRC's end-to-end telemetry allowed hosts to independently identify failing path selector (PS) values and move them into the bad path set.
```
[Host Spraying Traffic]
/ | \
[Path A] [Path B] [Path C (Zombie T1)]
│ │ │
▼ ▼ x (Silent Drops)
(Active) (Active) │
▼
[Host Detects Drop]
│
▼
[Quarantine PS Value to Bad Set]
│
▼
[Promote Backup QP Instantly]
```
Although the graph was plotted at a minute granularity to obscure exact recovery figures, MRC shifts paths on the scale of a single Round-Trip Time (RTT). Full throughput recovered near-instantly, allowing the job to continue uninterrupted while the faulty switch eventually rebooted. A similar incident occurred on a Tier 0 (T0) switch due to a flapping optical transceiver across four links. Under traditional architectures, losing a T0 or T1 switch triggers job-hurting tail latency or complete pretraining failures; with MRC and static SRv6 routing, it manifests as nothing more than a minor, unnoticeable blip.
- **On-Call Impact:** Operating large-scale AI fabrics becomes significantly less stressful for network engineers, as jobs automatically sustain high throughput even when underlying links flap or drop unexpectedly.
- **Hardware Integration & Vendors:** The architecture is supported in ASIC and NIC hardware from major vendors, including NVIDIA, AMD, and Broadcom.
- **Speed Profiles:** MRC is deployed and operational across both 400G and 800G fabric speeds.
- **OpenAI Production Scale:** Deployed across multi-tier, multi-plane production clusters configured in both $4 \times 200\text{G}$ and $8 \times 100\text{G}$ setups.
- **Debugging & Tooling:** Traditional network diagnostics such as `ping` and `traceroute` continue to work alongside specialized tools like Cluster Mapper to debug degraded hardware while jobs run unimpeded.
- **Research Paper:** Detailed technical specifications regarding compact data structures, congestion control, and multi-plane fabric designs are published in the MRC paper available on arXiv.
<FollowUp label="Want to explore how MRC's congestion control algorithms interact with static SRv6 source-routing?" query="Explain the congestion control mechanisms used in OpenAI's MRC and how they interact with static SRv6 source-routing."/>
## Notable quotes
> [00:03](https://www.youtube.com/watch?v=gTqeU9cCxKQ&t=3s) the most ridiculous concept — Jitendra Padhye
Jitendra Padhye of OpenAI presents Multipath Reliable Connection (MRC), a scale-out transport protocol developed with Microsoft, AMD, Broadcom, and Nvidia to replace RoCE and InfiniBand in AI training. MRC addresses the extreme tail-latency sensitivities of synchronous supercomputing by coupling packet spraying with host-managed SRv6 static source routing.
Synchronous Training Vulnerabilities and RoCE Bottlenecks (0:42)
Synchronous machine learning pretraining alternates between local GPU computation and collective exchanges, such as AllReduce, across thousands of GPUs. Because no worker can begin step N+1 until every GPU completes step N, overall cluster efficiency is governed strictly by worst-case tail latency rather than average or high-percentile (P95/P99) performance. Any localized link congestion or packet delay halts the entire cluster, risking discarded steps.
Conventional RDMA over Converged Ethernet (RoCE) ties each Queue Pair (QP) to a single static path. In multi-tier Clos networks, load-unaware Equal-Cost Multi-Path (ECMP) routing causes random hash collisions where multiple QPs congest a single physical link while alternative routes remain idle. Padhye notes that this uncoordinated allocation caps theoretical fabric utilization at approximately 66%.
Packet Spraying with Direct Memory Placement (7:22)
To eliminate hash collisions, MRC sprays the individual packets of a single application QP across all available healthy paths in the network fabric. Because variable physical link lengths and transient delays cause packets to arrive out of order, standard RoCE implementations would suffer severe degradation and trigger redundant retransmissions due to a lack of buffer offset information.
MRC overcomes this limitation by encoding the explicit Virtual Memory Address (VA) directly into each packet header. The destination GPU NIC reads this address and places out-of-order payloads directly into memory upon arrival. A completion signal is delivered to the host only after every packet comprising the buffer has landed.
Static Source Routing Through SRv6 Micro-SIDs (10:53)
Rather than relying on dynamic in-network routing decisions, MRC implements static source routing using Segment Routing over IPv6 (SRv6). The sending host maintains full visibility of the physical topology and encodes up to six 16-bit micro-SIDs inside a 32-bit locator structure within each packet header. Intermediate switches operate statelessly, executing a 16-bit left shift on the destination address and statically forwarding the packet without maintaining control planes or routing state tables.
If an intermediate link fails, the switch immediately drops the packet instead of attempting local rerouting. End hosts independently detect packet loss via continuous deterministic loopback probes and steer traffic away from degraded paths within one round-trip time (RTT).
Host-Driven Path Recovery in Production Environments (16:52)
MRC manages path availability by organizing network routes into dynamic pools. In an illustrative 128-path topology, the host activates 32 QPs across active paths, holds 32 backup QPs in reserve, and leaves 64 unassigned. If packet loss is detected, the host demotes the impaired path to a monitored bad-path set and immediately promotes a reserve QP to maintain throughput. For planned maintenance, operators can inject regular-expression denylists into source hosts to proactively route traffic around specific switches without packet loss.
In an OpenAI production run across roughly 75,000 GPUs, a Tier 1 switch suffered a 'zombie' failure, silently dropping around 500,000 packets across 25% of active QPs without generating switch-level telemetry. MRC host telemetry isolated the failing path selectors and promoted backup paths within a single RTT, preventing job disruption. Similar fault isolation successfully mitigated Tier 0 optical transceiver link flaps across 400G and 800G fabrics.
Padhye asserts that shifting routing and failover responsibilities from switch hardware to host-driven software eliminates single-point fabric failures in large-scale AI clusters. Supported by ASICs and NICs from Nvidia, AMD, and Broadcom, the architecture is operational in OpenAI's 400G and 800G production systems, with detailed congestion control mechanisms and compact data structures documented in OpenAI's published research paper.
[0:03] Jitendra Padhye: the most ridiculous concept
[0:03] Jitendra Padhye: Multipath Reliable Connection
[0:42] Jitendra Padhye: keep the job stepping
Original transcript
[0:03] All right, thank you guys. Uh, good
[0:05] morning everyone. Um, I'm Chitendra Pad.
[0:08] I work at OpenAI and um, it's going to
[0:11] be very hard to top the wonderful,
[0:13] wonderful introduction that Omar gave.
[0:15] Um, it was so inspiring. I was sitting
[0:16] at the back listening and I had
[0:17] goosebumps. Um, so I'm going to talk
[0:20] about a much smaller vision. Now we
[0:22] going to talk about transport protocols.
[0:23] You heard Omar say this is the new
[0:25] transport protocol called MRC uh that we
[0:29] have developed at OpenAI along with our
[0:31] partners um and SRV6. So what does MRC
[0:34] stand for? Let me tell you a little bit
[0:36] of a story. Um I gave a I made my
[0:40] teenage son listen to the dry run this
[0:43] talk. Big mistake. Don't do that. He
[0:46] listened to the talk and he said, "Dad,
[0:48] MRC clearly stands for the most
[0:50] ridiculous concept." So uh hopefully you
[0:54] guys will be more kind. Hopefully you
[0:55] guys will be more kind. Okay. Uh okay.
[0:58] So MRC stands for multipath reliable
[1:00] connection by the way. Uh it's a
[1:02] protocol transfer protocol for uh AI
[1:05] training that we had developed with a
[1:08] large team. Uh I'm here merely as a
[1:12] representative. You can see how many
[1:13] authors are on our archive paper. And
[1:15] that's again just a small subset of
[1:17] people who worked on it. Some of these
[1:19] guys are in the audience. Okay. So at
[1:21] OpenAI we work with Microsoft, AMD,
[1:24] Broadcom, Nvidia and this is really um
[1:27] keeping up with the spirit of Atscale.
[1:29] This is a joint you know this is a joint
[1:31] work this is collaboration we work with
[1:33] the community and the protocol is
[1:35] available from multiple vendors as you
[1:37] will see. Okay,
[1:39] so TDR one slide if you're going to
[1:42] listen to only one slide. What is MRC?
[1:44] Multipath reliable connection. It's a
[1:47] new transport protocol uh for scale out
[1:51] networks for AI training. As Omar
[1:53] mentioned, this is within the data
[1:54] center. This is not um scale up. This is
[1:59] a replacement for the job that was
[2:01] traditionally done by Infiniband and
[2:03] Rocky. Uh why did we do it? because we
[2:07] want it to be robust to fabric failures.
[2:10] Obviously, like everybody else, we want
[2:12] better job uptime. We want to build
[2:14] bigger clusters. All of these things.
[2:16] And hopefully as the talk goes, you guys
[2:18] will see how we accomplish all these
[2:19] objectives, right? There are two or
[2:22] three key technologies that come into
[2:24] play. Packet spreading,
[2:28] source routing, and then multiplaner
[2:30] networks. I'll talk about the first two.
[2:32] I'll allude only a little bit to the
[2:34] last one uh just because we don't have
[2:36] enough time but I'm around. Happy to
[2:38] talk more anytime you guys want. Okay.
[2:42] So here's the outline of the talk.
[2:46] You guys are all familiar with ML
[2:48] training but just to kind of put the
[2:50] talk in context and for the sake of
[2:52] completeness sort of an ELI level uh
[2:55] overview of what ML training is.
[2:59] Then the two key technologies uh that
[3:01] make MRC possible uh packet spraying and
[3:04] source routing. I'll give you I'll show
[3:08] you how MRC um achieves these goals sort
[3:11] of show you MRC in action. Couple of
[3:12] examples uh from our training runs and
[3:16] then conclude with current status. Okay.
[3:19] So at a very high level and this is
[3:22] nothing new to anybody in this audience.
[3:25] uh ML training is essentially thousands
[3:27] of GPUs uh synchronous training working
[3:30] one step at a time. Okay. Uh every GPU
[3:34] think of it maybe as they all reduce or
[3:36] some other sort of collective. Every GPU
[3:38] does some comput computation locally
[3:41] then they all exchange data with each
[3:44] other. Uh and those are the light blue
[3:46] boxes. Uh and the key thing is that you
[3:49] can't start the next step until the
[3:51] previous step is finished. Okay. So your
[3:54] step time and therefore your efficiency
[3:56] overall is governed by the tail latency.
[4:00] Okay, if the network is congested, if
[4:02] one GPU can't extend data fast enough,
[4:04] what have you, the entire step stalls
[4:07] and you can't start the next one. That's
[4:09] the key thing. That's the key takeaway.
[4:11] Um I come from Microsoft where I ran M,
[4:13] you know, Microsoft's public cloud
[4:14] network. That was not the constraint.
[4:16] This is the main learning here, right?
[4:17] It's tail latency matters. P95, P99 not
[4:20] useful tail latency. Uh and if you are
[4:23] failure that's even worse because then
[4:25] you basically have to repeat the whole
[4:26] step right go back to the previous step
[4:28] start from the next one. Uh and that's
[4:30] the key thing to take away. Okay. So
[4:34] what does this mean? This means that the
[4:37] network has one job and one job only.
[4:39] Keep the job stepping. That right below
[4:42] is our opens icon that we post in Slack
[4:45] when the job has stopped stepping and it
[4:47] starts stepping again. So that's our
[4:49] icon right there. Right. And so If
[4:52] that's the job of the network to keep
[4:54] the job stepping a transport protocol
[4:57] effectively has to do two things. It has
[5:00] to use all healthy capacity right and
[5:04] that's to make sure that no GPU is
[5:06] bottleneck available all healthy
[5:08] capacity and then failures are going to
[5:12] happen it's a big fabric as you heard
[5:13] big numbers hundreds of thousands of
[5:15] GPUs how many routers how many links
[5:18] quickly and automatically route around
[5:19] it right and both quickly and
[5:22] automatically are keywords obviously
[5:24] human intervention at this time scale is
[5:26] out of the window and you have to even
[5:28] automatic has to network as fast as
[5:30] possible. Okay, so this is how um we
[5:33] think about the goals of transport
[5:34] protocols and this is what MRC is trying
[5:37] to achieve. Okay, now this is where all
[5:41] the meta rocky people are probably going
[5:42] to throw rocks at me. Um I'm going to
[5:46] claim that regular Rocky can't achieve
[5:47] this goal. I mean they might agree but
[5:49] they'll claim that you know they they
[5:51] can do better and you'll hear about it
[5:52] in the next two talks. Okay, so the
[5:54] regular Rocky QP you guys all know this
[5:56] it uses a single path. Okay. Uh Rocky is
[6:00] can't stand packet reordering, can't
[6:02] stand packet loss. Uh and if you have
[6:05] just one QP, this is fine, right? This
[6:07] is a standard two-tier glass network.
[6:10] And uh if there is just one QP, fine.
[6:13] Anyway, your bottleneck is from the
[6:14] server to the tower or the leaf uh
[6:18] switch anyways. And so if you have one
[6:19] path, you haven't lost anything. It's
[6:21] all good, right? Um all packets in the
[6:25] QB follow the same path. Okay? And but
[6:29] what happens when you have multiple QPS?
[6:31] When you have multiple QPS, if you have
[6:34] enough of them, sooner or later there is
[6:36] going to be a hash collision. Again,
[6:37] this is something that you guys are all
[6:39] familiar with. Uh that is because we are
[6:41] relating the fabric ECMP choose the path
[6:45] and it's not in the classic rocky
[6:48] fabric. The ECMP is not load aware. So
[6:51] it's going to choose path without
[6:52] knowing current load or future load or
[6:54] anything like that. And once you have
[6:56] hash collisions, you might have two QPS
[6:59] or more QPS sharing a particular link.
[7:01] That's going to limit your throughput
[7:02] even though the rest of the fabric is
[7:04] lying idle. And in this particular case,
[7:07] you can't use all available capacity by
[7:08] definition. Okay. In fact, there's a
[7:10] beautiful result out there that if you
[7:11] do the right calculations, the expected
[7:13] bound is around 66% for fabric
[7:15] utilization. So this is a well-known
[7:17] result. Okay. So this is all good. So
[7:20] what's the answer to this? Very simple.
[7:22] Packet spring. Okay. So the key here is
[7:27] that MRC opens only one QP. So the
[7:29] application is not aware. The
[7:30] application is still going to open one
[7:32] QP except we are going to spray all the
[7:37] packets from this QP onto all available
[7:40] paths or at least multiple available
[7:41] paths. All is a bit much. But the first
[7:44] hop obviously there is no choice. I'm
[7:46] just assuming a single plane arrow here.
[7:48] First hop there is no choice. So you go
[7:50] up to the T0 or the leaf switch and then
[7:53] you typically have choice of multiple
[7:55] spine routers or T1 routers and you
[7:57] spray the packets and you use all
[8:00] available links. Okay. And then on the
[8:02] way down obviously again you have no
[8:03] choice so you come down on that path.
[8:04] Okay. Um so essentially every MRCQP uses
[8:08] all available paths and essentially all
[8:11] available healthy links and that's what
[8:14] that's what when the uh the packet
[8:16] spring. What happens when you're two
[8:18] QBs? Well, that's great. That QP is also
[8:21] using our level of blocks. Okay. So, the
[8:24] end result here is that by definition
[8:27] there are no hotspots in the network
[8:28] like we saw for the Rocky case earlier
[8:33] because you are spraying packets you are
[8:35] using all available capacity and as long
[8:38] as there is capacity in your network
[8:39] there are no hotspots. Right? So, this
[8:42] is very simple. That's what avoids
[8:44] essentially the hash collision problem
[8:47] in that that you have in the rocky
[8:49] world. Okay,
[8:52] this is easier said than done. Of
[8:54] course, everybody's aware that packet
[8:56] spraying creates problems and now we
[8:58] have to deal with it. The first is
[9:00] obviously reordering and that's going to
[9:03] happen no matter what simply because
[9:05] sometimes the length of your wires is a
[9:07] little bit different even if you have no
[9:08] congestion in your network. And then the
[9:11] second one is more sort of engineering
[9:13] decision. Um it's easy to say you spray
[9:15] packets. Wow, there are many choices.
[9:17] You could still rely on the fabric for
[9:19] ECMP. Then you might have the same
[9:21] problems again or you can do something
[9:23] else. I'll tell you how we solve both
[9:25] those problems. Okay, so the first out
[9:27] of order arrivals. This is fairly
[9:29] straightforward. Um a rocky QP is not
[9:32] tolerant of out of order arrivals as we
[9:34] all know. uh it doesn't tolerate because
[9:37] the packet headers don't carry the VA
[9:39] address doesn't know when out of where
[9:41] to place the out of order packet as a
[9:43] result it will typically lead to
[9:45] retransmissions and then maybe you know
[9:47] congestion signals and so forth um in
[9:50] MRC we solve this problem very simply
[9:52] each packet carries the address where
[9:54] it's supposed to go in the GPU buffer so
[9:56] as soon as the packet arrives it's
[9:58] placed in the right place so in this
[10:00] particular example that I have shown
[10:01] here packet one arrives after that
[10:03] packet three arrives So packet three
[10:06] normally rocky wouldn't know where to
[10:08] place it but because the address exists
[10:09] MRC knows where to place it it's placed
[10:11] in the right place and then two and so
[10:13] forth eventually finally when packet
[10:16] five arrives the buffer is complete and
[10:19] then they can generate a completion okay
[10:21] so essentially by making every packet
[10:23] carry the address where it should go in
[10:25] the memory out of order packet arrival
[10:27] can be handled fairly straightforwardly
[10:29] okay and because no networking talk is
[10:31] ever complete without a wireshark
[10:32] diagram here is a wireshark screenshot.
[10:35] Uh this is how it looks like on the
[10:37] wire, right? There's re error and that
[10:39] header simply carries the V address. As
[10:41] you can see, that's a right middle
[10:43] packet. Uh normally this address would
[10:45] be only in the right first and so that's
[10:46] that. Okay, so nothing fancy about it.
[10:50] This is great, right?
[10:53] Here is the key thing. How are you going
[10:54] to choose the path of each packet? So we
[10:56] had two choices. If you look at the
[10:57] original MRC spec or the UC spec, uh
[11:00] packets were originally you can do ECMP
[11:04] but then you still have dependency on
[11:05] the fabric and you are not going to be
[11:08] able to route around the failures.
[11:09] You're at the mercy of the fabric. What
[11:11] we do instead is we source route the
[11:14] packets using SRV6. SRV6 is a well-known
[11:17] technology, right? And the idea here is
[11:19] that the host knows the base topology
[11:23] and it essentially puts the source
[11:25] routed path of the packet and the fabric
[11:27] is completely dumb. The fabric simply
[11:30] forwards the packet in a static manner
[11:32] and if you try to shoot a packet on a
[11:35] path where the links are down, so be it.
[11:37] Packet is dropped and it's the host jobs
[11:40] to deal with it. The fabric does not
[11:42] intervene. Everything is end to end. So
[11:44] if you think you know you guys might
[11:45] have heard of the end to end principle,
[11:47] this is sort of the really data center
[11:49] version of it. Okay. So quick SRV6
[11:53] primer in case you guys are not familiar
[11:55] with it. Uh when the packet leaves the
[11:57] source uh it contains the complete
[12:00] destination address. It consists of a
[12:02] 32-bit locator. What that is? We can
[12:04] ignore that for now. And then it
[12:06] consists of a series of microsids up to
[12:08] six uh 16 bits. And these are
[12:12] essentially router identifiers. Okay, so
[12:14] when the packet leaves there is a source
[12:16] T0. So obviously it's going to go up to
[12:18] source T0. Uh there is typically no
[12:20] choice here. Uh then you have potential
[12:24] choices of several leaves. So you can
[12:25] put one of those microetss up there.
[12:28] Then you have a choice of only one
[12:29] destination T0. It's done now. The
[12:31] packet is coming down. Uh and then the
[12:33] nick the destination itself has its own
[12:36] microite. Okay. So that's what the
[12:38] address looks like when the packet
[12:40] leaves the source nick. And at each hop
[12:45] the address is shifted. So the T0 gets
[12:47] the packet. Okay, this is my microet.
[12:49] This is my this is the packet I'm
[12:51] supposed to forward. Fine. It drops that
[12:53] shifts everything left by 16 bits and
[12:56] then oh okay this is the T1 I'm supposed
[12:58] to send the packet to. It sends the
[13:00] packet to that particular T1 because it
[13:01] knows what the mapping is. So
[13:02] essentially static routing at each hop.
[13:04] Okay, you have no control play, no
[13:06] nothing. Everything is programmed
[13:07] statically up front. So the fabric now
[13:10] really has no role to play apart from
[13:12] simply forwarding the packet. What
[13:14] happens if the link to this particular
[13:16] T1 is down? I drop the packet. Simply
[13:19] the router simply drops the packet and
[13:21] now the source has to realize that the
[13:23] packet has been dropped. It has
[13:24] generated a potentially bad path and on
[13:28] the next retransmission maybe do
[13:30] something else. Okay. So everything is
[13:32] end to end. That's the key idea.
[13:35] All right. So to do this, we'll walk
[13:39] through this a little bit quickly. Uh
[13:42] for each destination, the source knows
[13:44] what are all possible paths available in
[13:46] the network. Okay. Uh obviously all of
[13:48] this information is compacted. I'm just
[13:50] going to make it sound like you are
[13:52] really doing everything right. So you
[13:53] have a set of path selectors. Uh each
[13:56] path selector maps to a particular path.
[13:58] Uh you can think of it as effectively a
[14:00] number. Uh and then once you know the
[14:04] source and the destination, you know how
[14:06] many paths there are. You can think of
[14:07] them as numbers and then each path
[14:10] selector
[14:12] because there is a structure to the
[14:14] microsets that we assign. The numbers
[14:16] work out in such a way that they can be
[14:18] bits from that number can be copied to
[14:20] generate the appropriate microsets.
[14:23] That's you know because there is a
[14:24] structure to the network. We are not
[14:25] building random networks here. Right?
[14:27] Okay. So at the QP setup time, MRC
[14:29] chooses a subset of those uh eligible
[14:32] path selectors, right? And so for
[14:35] example, and it builds the address at
[14:37] the same time. So for example, if you
[14:40] have this particular setup, let's say
[14:42] it's a 8x100 nick, you will have a
[14:44] choice I I've been saying that you have
[14:47] choice of only one T0, but obviously
[14:48] it's a multiple network. So you'll have
[14:50] a choice of eight T0. You pick one. So
[14:52] that's essentially a three-bit choice.
[14:54] So right so you pick the right three
[14:56] bits from your path selector and you
[14:58] copy them there. That gives you the
[15:00] first T0 micro you might have a choice
[15:02] of whatever 128 T1s or whatever it might
[15:05] be. Now you have seven bits you pick
[15:07] those. The destination you can go from
[15:10] that's fixed and the uh destination T0
[15:13] is also fixed. So here you have created
[15:16] a path right. uh so selected PS supplies
[15:19] the plane and the nick and the result is
[15:22] that you have effectively a complete IP
[15:26] address there is no ECMP involved okay
[15:28] that's it it's very simple and the
[15:31] benefits here is as I've been saying
[15:34] repeatedly the network stays very simple
[15:36] the switches just forward traffic
[15:38] everything is static right our
[15:40] configuration is very simple it allows
[15:42] us to verify that at setup time right uh
[15:46] we know whether the router is correctly
[15:47] configured. There is no need to ask any
[15:48] questions. Very easy verification,
[15:50] right? End host can deterministically
[15:52] steer traffic. I'll show a little
[15:54] example within one roundtrip time
[15:55] because now you know that there is a
[15:56] loss. You know what was the path
[15:58] selector value you use for that packet.
[16:00] Why don't use it again. That's it.
[16:01] You're done. Or put it on the back
[16:03] burner, probe it separately, what have
[16:05] you, right? Um
[16:07] the most interesting thing here is that
[16:09] it allows you to do because of SRV6
[16:12] something that I always wanted to do. So
[16:14] when I was back at Microsoft, you know,
[16:16] in Azure, it was hard for us to prove
[16:18] our network effectively because we used
[16:19] DCMP, right? If you generate synthetic
[16:21] traffic, you always had to guess where
[16:23] the packet went. So we resorted to
[16:24] tricks like IP and IP encapsulation. You
[16:26] go from a nick to a T0 decap go to a
[16:30] particular T1 and so forth. But here you
[16:32] can do whatever you want. You can probe
[16:33] yourself, right? You can go source to T0
[16:37] to T1 back to the same T0 back to
[16:39] yourself. You can probe ping yourself if
[16:43] you want. You can go T 0 T1 some other
[16:45] T0 some other T1 back to your T0 back
[16:47] down here. Right? So this is beautiful.
[16:49] Allows you to build a very clear picture
[16:52] of the network. The host at all time not
[16:55] only knows the static topology but
[16:56] potentially even what's alive in the
[16:58] network. So that's the beauty. Okay. So
[17:02] the source the way this works is it has
[17:04] a set of let's say you had let's just
[17:06] pretend that you had 128 potential paths
[17:08] in the network. What you would do is you
[17:10] would create 32 active QPS, 32 backup
[17:13] QPS, then leave the other 64 aside and
[17:16] you start by spreading traffic on 32
[17:18] paths. Let's say you have a loss. That's
[17:21] okay. You take that QP, chuck it into
[17:22] the bad path. You keep proving it. You
[17:25] know, you can prove it in many number of
[17:26] ways. Uh and then bring one spare that
[17:29] you had from the backup. No need to even
[17:31] test it. It was in backup. You bring it.
[17:33] If it experiences loss, you can chuck
[17:35] that into backup path also. Right? And
[17:37] you can continue like this. So this is
[17:39] what it does and um because you even
[17:43] form an idea of what is bad in the
[17:45] network right you can tell ahead of time
[17:47] let's say you have a maintenance event
[17:48] coming up so many switches are going to
[17:50] go down but their path selector values
[17:52] are known their microetss are known you
[17:55] can effectively program in regular
[17:56] expressions in all the sources so that
[17:58] those switches are never used so your
[18:00] you know your traffic is not affected by
[18:02] the routers that are under maintenance
[18:03] or by the links that are under
[18:04] maintenance. So these are called
[18:05] denless. That's also another beautiful
[18:07] thing you can do with this. Okay. So now
[18:11] let me show you glimpses of MRC in
[18:13] action and how beautifully this all
[18:15] works. Okay. So this one, yeah, it's a
[18:18] smalish job about 75,000 production
[18:21] GPUs. Uh we had a T1 that went zombie.
[18:25] I'm sure all of you guys have dealt with
[18:27] zombie routers. The router stayed up and
[18:28] started forwarding packets and you know
[18:30] no indication in the telemetry that
[18:32] anything was wrong with the router. MRC
[18:35] happily removed the path throughput
[18:37] recovered the switch rebooted. Yeah, we
[18:39] had 25% of the QPS that were affected
[18:42] but MRC fully recovered because as you
[18:44] saw in that diagram earlier well it it
[18:46] didn't know that a particular T1 was
[18:48] bad. It just realized that certain PS
[18:50] values were bad and those were not used.
[18:52] They all went into the bad set and that
[18:54] was done by every route every host
[18:56] independently. So the network just
[18:58] recovered no issues. Okay. There were
[19:00] about half a million packets that were
[19:01] dropped by the bat switch by the time
[19:03] everybody figured out what was going on.
[19:04] But so be it, right? This is what it
[19:06] look like, right? On the on the x-axis
[19:08] there is time. And this graph is
[19:10] deliberately plotted at a minute
[19:12] granularity because you didn't want to
[19:14] give away how quickly we recover, but
[19:15] it's on the scale of one round trip
[19:17] time. Obviously, uh time zero is when
[19:19] the switch rebooted. It had gone bad
[19:22] about 10 minutes earlier. Uh so that's
[19:24] when you see a little bump in the packet
[19:26] loss and there a little dip in the
[19:27] throughput
[19:29] within a few roundtrip times. All hosts
[19:32] that were sending traffic through this
[19:34] had recovered chucked along the switch
[19:37] rebooted and then it came back as
[19:39] normal. Who knows what it was doing
[19:40] right? Yeah it's great nothing happened.
[19:43] This is the beautiful part right and as
[19:45] a network engineer my on call has become
[19:48] really really painless because of MRC.
[19:51] So like I don't have to worry about how
[19:53] much of the narrow is up as long as
[19:54] substantial number of links are up. I
[19:56] don't care if the links are flapping. I
[19:58] mean I do care. We don't want links
[19:59] flapping but nonetheless nonetheless
[20:02] right you know um it's it's really
[20:05] amazing to watch MRC doing this. Uh
[20:08] here's another example. We had a T0
[20:10] switch uh where the switch transceiver
[20:12] was flapped. Uh four links uh went bad.
[20:16] MRC removed the path and full throughput
[20:18] return. Again the same thing you can see
[20:21] there's just a tiny blip and we carried
[20:24] on. Previously a T0 going down a T1
[20:28] going down is potentially a job hurting
[20:30] event because of MRC it just translates
[20:33] to a small blip and full recovery.
[20:35] That's the power of packet spreading.
[20:37] That's the power of SRV6. That's the
[20:39] power of having entirely an end to-end
[20:42] approach to doing this. Okay. And so
[20:45] with this um there's a lot more
[20:47] multiplaner fabric design addressing we
[20:50] developed fairly compact data structures
[20:52] to do all this. There is condition
[20:54] control that we can talk about if you
[20:55] guys are interested at some point. The
[20:58] last bit that gets always unsaid support
[21:00] tools right it's easy to say hey this is
[21:02] my transport protocol and this is a
[21:04] service wonderful these are all
[21:05] standards we're paying trace route right
[21:07] our standard debugging tools still have
[21:09] to work right even though we I said that
[21:11] network blips don't matter we still have
[21:13] to debug it so we build these tools
[21:15] cluster mapper I talked about it uh
[21:17] deployment is a somewhat of an issue so
[21:19] our paper that's on archive uh has all
[21:21] these details uh so happy to talk more
[21:24] about this if you guys have questions
[21:26] okay so I'll leave you with the status
[21:28] um it's implemented it's available from
[21:30] Nvidia AMD and Broadcom again many of
[21:33] those guys many of the engineers who
[21:34] helped build uh helped us build this
[21:37] they are in the audience uh uh available
[21:40] for both 400 and 800 and at open AI it's
[21:43] deployed at a very large scale uh both
[21:45] at 4x200 and 8x100
[21:48] uh two-tier multiplane production
[21:50] clusters has been working wonderfully so
[21:53] thank you
