Google Unveils Virgo Network Fabric for Multi-Building AI Scaling

@Scale

Google is introducing Virgo, a multiplane two-layer scale-out network fabric utilizing skinny links and CSIG congestion signaling, to enable non-blocking AI compute clusters across data center campuses.

AI-assisted summary of Live from SCCC: Virgo: Scale-out Data Center Network for AI | Jeongkeun "JK" Lee from Google — original source. Check the original for full context.

Source published: 2026-09-13

Detailed notes

## Background Published on YouTube by @Scale on August 31, 2026, this presentation features Jeongkeun "JK" Lee from Google speaking at the Santa Clara Convention Center. According to the page description, the talk introduces Virgo—a new backend scale-out fabric for east-west AI communication designed for deterministic latency, resilience, and high goodput—which will power Google Cloud's A5x bare-metal instances running the NVIDIA Vera Rubin NVL72 platform and serve as a key building block for next-gen TPUs.

## TL;DR Jeongkeun "JK" Lee presents Virgo, Google's purpose-built, 2-layer scale-out network fabric engineered specifically to handle the extreme resilience and latency demands of massive AI workloads. He details Virgo's multi-plane architecture powered by 200G SerDes skinny links, which ensures high resilience and non-blocking bisection capacity while decoupling east-west traffic from the Jupiter front-end network. Lee highlights adaptive routing, SRv6 source routing, and sub-millisecond telemetry required to catch transient congestion micro-bursts. Finally, he outlines CSIG (Congestion Signaling), an in-band telemetry protocol standardized through the Ultra Ethernet Consortium, alongside Google's roadmap to scale RDMA fabrics to metro distances.

## Atmosphere & style The presentation carries an authoritative yet humble tone, driven by Lee's reflective appreciation for the open technical community and his pride in Google's collaborative engineering milestones. It feels like an insider look into frontier data-center architecture, blending pragmatic engineering rigor with a forward-looking vision for AI infrastructure. The delivery is lucid, grounded, and technically rich, creating a atmosphere of shared technical breakthrough as past networking concepts find new life in modern AI deployments.

## Key takeaways - [00:03](https://www.youtube.com/watch?v=WuzpHLYnre0&t=3s) Jeongkeun "JK" Lee introduces Virgo, Google's brand-new scale-out data center network fabric purpose-built to address the bandwidth and latency demands of AI workloads. - [00:18](https://www.youtube.com/watch?v=WuzpHLYnre0&t=18s) Google's networking evolution spans four main eras: Internal Services (Jupiter, Orion, Apollo), Streaming (BBR, QUIC), Cloud (PSP, Falcon offload), and the current AI Era requiring campus-wide invisible failure resilience. - [02:18](https://www.youtube.com/watch?v=WuzpHLYnre0&t=138s) AI network design prioritizes five core principles: campus-supercomputer integration, lock-step hardware lifecycles with accelerators, inherent fabric resiliency, high-resolution telemetry, and rapid low-layer deployment. - [04:37](https://www.youtube.com/watch?v=WuzpHLYnre0&t=277s) Google AI data center architecture relies on three specialized layers: Scale-Up (NVLink/ICI for intra-pod), Scale-Out / Virgo (dedicated backend accelerator interconnect), and Front-End / Jupiter (storage and general compute). - [05:40](https://www.youtube.com/watch?v=WuzpHLYnre0&t=340s) Decoupling the Virgo scale-out fabric from the Jupiter front-end network enables independent annual hardware evolution for GPUs and TPUs without infrastructure disruption. - [06:50](https://www.youtube.com/watch?v=WuzpHLYnre0&t=410s) Virgo adopts a 2-layer collapsed flat topology paired with a multiplane architecture using native SerDes 'skinny links' to eliminate 3rd-tier Clos switches and minimize latency. - [08:03](https://www.youtube.com/watch?v=WuzpHLYnre0&t=483s) The multiplane skinny link design provides graceful degradation; in an 8-plane fabric, losing a single plane retains 87.5% of total bisection bandwidth. - [10:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=626s) Virgo leverages hardware adaptive routing and SRv6 source routing to maximize multipath utilization and allow bare-metal cloud customers to dictate packet paths. - [11:31](https://www.youtube.com/watch?v=WuzpHLYnre0&t=691s) Sub-millisecond telemetry visibility is required because standard 30-second average metrics mask severe periodic micro-bursts that saturate switch buffers. - [13:55](https://www.youtube.com/watch?v=WuzpHLYnre0&t=835s) Google introduced Congestion Signaling (CSIG), an in-band telemetry mechanism with a fixed-size header tag that carries only bottleneck link metrics back to the sender. - [15:51](https://www.youtube.com/watch?v=WuzpHLYnre0&t=951s) CSIG telemetry specifications have been published at SIGCOMM and are being standardized within the Ultra Ethernet Consortium (UEC). - [16:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=986s) Future Virgo developments focus on extending scale-out RDMA connectivity across metro distances and enabling multi-vendor switch interoperability.

## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=WuzpHLYnre0&t=3s) Welcome and Google’s Data Center Networking History

Jeongkeun "JK" Lee from Google opens his talk at the @Scale conference by reflecting on his long relationship with the event, noting that his first attendance back in 2016 gave him a deep appreciation for the open industry perspectives shared on stage. Expressing immense gratitude, he explains that he is humbly representing numerous engineering teams at Google who are actively building and deploying **Virgo**, Google's brand-new scale-out data center fabric purpose-built for AI workloads.

Looking back across Google's history, Lee highlights several major inflection points in data center networking innovations that led up to this moment: * **The Internal Services Era:** Scaled global consumer applications like Google Search and Gmail using the **Jupiter** data center architecture alongside the **Orion** SDN control plane, and pioneered **Apollo** (the first production deployment of optical circuit switching in data centers). * **The Streaming Era:** Optimized real-time, low-jitter video delivery through novel transport and congestion control schemes such as **BBR** and **QUIC**. * **The Cloud Era:** Shifted performance offloads directly into hardware, introducing **PSP** encryption and the **Falcon** reliable transport protocol offloaded to NIC hardware. * **The AI Era (2026):** AI model sizes and bandwidth demands per chip are growing exponentially, requiring hyper-strict latency bounds and fabrics spanning multiple buildings and campuses where failure resilience must be transparent to the workload.

> "At this scale we know that everything breaks, and we should make these failures invisible to the AI workload."

---

[02:18](https://www.youtube.com/watch?v=WuzpHLYnre0&t=138s) Core Design Principles for Scale-Out AI Fabrics

Lee outlines five core architectural principles guiding the development of Virgo, contrasting them with traditional generic networking designs:

1. **Campus-Supercomputer Integration:** The network must be co-designed as a dedicated component of a campus-scale AI supercomputer for a single specialized workload, rather than a generic multi-tenant infrastructure. 2. **Lock-Step Hardware Lifecycle:** Switches, optical transceivers, and fiber infrastructure are lifecycle-bound to the specific AI accelerators (GPUs/TPUs) they support; traditional requirements like backward compatibility are no longer primary constraints. 3. **Inherent Resiliency:** Resiliency must be baked directly into the fabric layout so that single link or switch failures never crash a multi-node machine learning training job. 4. **High-Resolution Telemetry:** Fine-grained observability is essential so transport protocols (like congestion control) and control planes can extract maximum throughput and availability. 5. **Rapid Deployment & Low Layer Count:** Multi-billion dollar AI clusters must be delivered to customers on tight timelines. Minimizing switching layers reduces total rack count and fiber complexity, dramatically speeding up cluster turn-up.

---

[04:37](https://www.youtube.com/watch?v=WuzpHLYnre0&t=277s) The Three Networking Layers of Google AI Data Centers

Google’s modern AI data center architecture separates networking responsibilities into three distinct, specialized layers that collectively act as a unified compute domain:

* **Scale-Up Fabric (Orange):** High-bandwidth, sub-microsecond interconnect operating within a pod or rack scale. Connects accelerator chips directly using NVIDIA **NVLink** for GPU systems, or Google's Inter-Connect Interface (**ICI**) powered by optical switching for TPU systems. * **Scale-Out Fabric / Virgo (Yellow):** A dedicated, backend accelerator-to-accelerator network providing non-blocking cluster-wide connectivity across multiple pods and buildings. * **Front-End Fabric / Jupiter (Blue):** Connects the compute domain to distributed storage systems and general-purpose compute. Optimized for TCO, shared access, and incremental expansion across clusters within a building.

Lee emphasizes that decoupling the backend scale-out fabric (Virgo) from the front-end network (Jupiter) allows independent evolution cycles. This is vital for supporting different annual iterations of GPUs and TPUs without triggering systemwide disruptions during upgrades. Furthermore, Virgo can be engineered for 100% non-blocking bisection capacity while Jupiter remains cost-optimized for storage traffic.

---

[06:50](https://www.youtube.com/watch?v=WuzpHLYnre0&t=410s) Virgo Topology: Two-Layer Collapsed Design & Multiplane Architecture

Zooming in on the Virgo topology, Lee highlights three primary architectural structural features: a collapsed 2-layer flat topology, multi-plane architecture with independent control domains, and cross-building campus expansion.

By maximizing switch radices, Google collapsed traditional 3-tier folded Clos networks into a flat **2-layer non-blocking topology**, significantly cutting down hops, latency, and jitter.

To scale the fabric across thousands of chips, Virgo introduces a **multiplane design** built on **skinny links**: * **Skinny Links:** Every network link operates at the native SerDes lane speed (e.g., 200 Gbps per lane in current deployments), rather than aggregating SerDes into wider 800G ports. This maximizes effective switch radix and enables massive 2-stage scaling. * **Plane Calculation:** The number of parallel network planes ($N$) equals the total bandwidth of an accelerator divided by the SerDes speed. For instance, a GPU pushing **1.6 Tbps** over 200G SerDes yields **8 skinny links**, resulting in an 8-plane network fabric. * **Graceful Degradation:** If one plane fails in an 8-plane system, the node retains **87.5% ($7/8$) of its total bisection bandwidth**, avoiding the catastrophic 50% or 100% bandwidth loss typical of conventional dual-homed network designs. * **Fully Meshed Multi-Rail Integration:** Unlike strict rail-aligned architectures, every accelerator chip connects across all $N$ network planes. In a 4-GPU system with 4 rails and 8 network planes, the fabric is fully meshed, allowing both intra-rail and cross-rail traffic to traverse the full bisection bandwidth in just 2 switch hops without requiring a 3rd switching tier.

---

[10:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=626s) Multipath Routing: Adaptive Routing & SRv6

To maximize path utilization and lower tail latency across the multiplane topology, Virgo relies on advanced packet-level load-balancing mechanisms:

* **Adaptive Routing:** Hardware-based switch-level intelligence that dynamically steers packets around local switch congestion and quickly reroutes around downstream link failures. * **SRv6 Source Routing:** Allows bare-metal cloud customers (who supply their own host networking stacks, such as an RDMA stack) to directly monitor, tag, and dictate the exact end-to-end network path for every individual packet.

---

[11:31](https://www.youtube.com/watch?v=WuzpHLYnre0&t=691s) High-Resolution Telemetry: Sub-Millisecond Visibility

Because AI training generates massive, tightly synchronized line-rate micro-bursts, conventional network telemetry misses critical transient congestion.

> "While the blue line still claims 10% load, the traffic bursts are redlining the switch buffer every few seconds."

To combat this, Virgo integrates two high-resolution telemetry channels: 1. **Management Plane Telemetry:** Switch CPUs stream fine-grained performance metrics (such as sub-millisecond port transmission utilization) directly to monitoring dashboards and control systems. 2. **Data Plane Telemetry (CSIG):** In-band congestion signals embedded in data packets for real-time host-transport consumption.

Lee presents a live telemetry dashboard comparison demonstrating the necessity of fine-grained sampling: * **30-Second Average View:** Shows a clean, healthy network operating at nominal bandwidth with massive headroom. * **Sub-Millisecond View:** Reveals severe, periodic micro-bursts where traffic spikes to **100% buffer saturation** every few seconds despite average utilization showing only ~10%.

---

[13:55](https://www.youtube.com/watch?v=WuzpHLYnre0&t=835s) Congestion Signaling (CSIG) and UEC Standardization

Lee introduces **CSIG** (Congestion Signaling), an in-band telemetry mechanism designed as a practical, lightweight evolution of In-band Network Telemetry (INT).

While traditional INT stacked metadata headers from every hop—causing packet overhead to explode—CSIG maintains a **fixed-size header tag** carrying only the bottleneck link's information: * As a packet traverses switches (e.g., Hop 1 → Hop 2 → Hop 3), each switch updates the CSIG tag **only** if its local congestion state (e.g., available bandwidth normalized by port capacity or max hop delay) is worse than the upstream value already in the header. * The destination host receives the exact bottleneck metric alongside hints indicating whether congestion stemmed from first-hop multipath issues or last-hop incast congestion. * The receiver reflects this bottleneck telemetry back to the sender in ACK packets, allowing host transport layers to instantly throttle or adjust flows.

Lee shares a personal moment of pride, remembering that INT was his first P4 application while at Barefoot Networks, and expressing delight that CSIG builds upon those concepts into a cleaner, production-ready fabric standard.

- CSIG is being standardized within the **Ultra Ethernet Consortium (UEC)**. - Detailed technical papers on CSIG were presented by Google researchers at **SIGCOMM**.

---

[16:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=986s) Next Steps: Metro Scale-Out, RDMA Long-Haul, and Interoperability

Looking ahead, Lee details Google's roadmap for scaling Virgo beyond single data center buildings:

* **Metro-Scale Expansion:** Extending Virgo fabrics across metro regions spanning hundreds of kilometers. * **Long-Haul RDMA:** Partnering within the UEC to solve the steep challenges of running high-performance RDMA over long-distance optical links using extended CSIG congestion feedback. * **Multi-Vendor Interoperability:** As fabrics span distributed campuses, integrating switch silicon from multiple hardware vendors requires strict protocol unification and open standards collaboration.

Closing his presentation, Lee encourages the audience to join the UEC community efforts and scan a provided QR code to explore Google’s published research on AI data center and WAN networking.

> "Some innovations we as a community have worked on so far for the last decade, like INT or source routing... were overlooked initially, but now they are finding real proving ground in AI."

## Notable quotes > [00:03](https://www.youtube.com/watch?v=WuzpHLYnre0&t=3s) At this scale we know that everything breaks, and we should make these failures invisible to the AI workload. — Jeongkeun "JK" Lee

> [11:31](https://www.youtube.com/watch?v=WuzpHLYnre0&t=691s) While the blue line still claims 10% load, the traffic bursts are redlining the switch buffer every few seconds. — Jeongkeun "JK" Lee

> [16:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=986s) Some innovations we as a community have worked on so far for the last decade, like INT or source routing... were overlooked initially, but now they are finding real proving ground in AI. — Jeongkeun "JK" Lee

## Mentioned Jeongkeun "JK" Lee, Google, Virgo, Jupiter, Orion, Apollo, BBR, QUIC, PSP, Falcon, NVIDIA, NVLink, Inter-Connect Interface (ICI), SRv6, CSIG, In-band Network Telemetry (INT), Ultra Ethernet Consortium (UEC), SIGCOMM, NVIDIA Vera Rubin NVL72, Google Cloud A5x, @Scale, Barefoot Networks

[Source](https://www.youtube.com/watch?v=WuzpHLYnre0)

中文简报

谷歌推出Virgo网络架构，助力跨楼宇AI扩展谷歌推出Virgo，这是一种采用细链路（skinny links）和CSIG拥塞信令的多平面双层横向扩展网络架构，旨在跨数据中心园区构建无阻塞AI计算集群。 Jeongkeun “JK” Lee介绍了谷歌专为满足海量AI工作负载对弹性和延迟的极高需求而定制的双层横向扩展网络架构Virgo。他详细阐述了基于200G SerDes细链路的Virgo多平面架构，该架构在将东西向流量与Jupiter前端网络解耦的同时，确保了高弹性和无阻塞的对分带宽能力。Lee重点介绍了捕获瞬态拥塞微突发所需的自适应路由、SRv6源路由以及亚毫秒级遥测技术。最后，他概述了通过超级以太网联盟（Ultra Ethernet Consortium）标准化的带内遥测协议CSIG（拥塞信令），以及谷歌将RDMA网络扩展至城域距离的路线图。 Google 推出专为 AI 工作负载打造的横向扩展数据中心网络架构 Virgo，为加速器之间提供确定性延迟与高可用性互联。 (0:03) Google 现代 AI 数据中心划分为三个层级：Scale-Up（基于 NVLink 或 ICI 的机架级互联）、Scale-Out（Virgo 后端加速器网络）以及 Front-End（基于 Jupiter 的存储与通用计算网络）。 (4:37) Virgo 采用 2 层扁平折叠拓扑与基于原生 SerDes 细链路（skinny links）的多平面设计；以 8 平面为例，单个平面故障仍能保留 87.5% 的对分带宽。 (6:50) 网络结合硬件自适应路由与 SRv6 源路由，并借助定长 CSIG（拥塞信号标记）实现亚毫秒级拥塞监控，目前相关标准正推向 Ultra Ethernet Consortium (UEC)。 (10:26)

Reading summary

Google engineer Jeongkeun Lee introduced Virgo, a purpose-built scale-out data center network fabric designed to satisfy the strict latency, bandwidth, and resilience demands of campus-scale AI supercomputers. Lee argued that supporting modern AI clusters requires abandoning traditional generic multi-tenant networking in favor of a flat, resilient fabric decoupled from storage and front-end infrastructure. Decoupling Scale-Out Fabrics from General Infrastructure (4:37) Google organizes its modern AI data centers into three specialized networking tiers: an ultra-low-latency scale-up fabric within pods (using NVLink for GPUs or ICI optical switching for TPUs), the front-end Jupiter fabric handling storage and general-purpose compute, and the backend Virgo scale-out fabric connecting accelerators across pods and buildings. Lee explained that isolating Virgo from Jupiter provides 100% non-blocking bisection capacity for accelerator traffic while allowing the front-end network to stay cost-optimized. This separation also decouples hardware lifecycles, enabling Google to deploy annual accelerator generations without triggering wider infrastructure disruptions. Virgo’s architecture is driven by principles specific to AI supercomputing rather than traditional enterprise hosting. Switches, optics, and fiber are co-designed in lockstep with specific accelerator generations, eliminating backward-compatibility constraints. Additionally, Google minimizes switching layer counts to reduce fiber complexity and rack counts, which Lee noted significantly accelerates cluster turn-up times for multi-billion dollar AI deployments while masking individual component failures from the training workload. Two-Layer Multiplane Topology and Skinny Links (6:50) To lower latency, hop counts, and jitter, Google collapsed the traditional three-tier folded Clos network into a flat, two-layer non-blocking topology. Virgo achieves massive two-stage scaling by deploying "skinny links" that operate at native SerDes lane speeds, such as 200 Gbps per lane, rather than bundling SerDes into wider 800 Gbps ports. This technique maximizes switch radix and establishes independent parallel network planes. The number of planes is determined by dividing accelerator bandwidth by the SerDes speed, creating an eight-plane network for a 1.6 Tbps accelerator operating over 200 Gbps SerDes. According to Lee, this multiplane design prevents the catastrophic 50% or 100% bandwidth collapse typical of conventional dual-homed networks; losing a single plane in an eight-plane fabric leaves 87.5% of the bisection bandwidth intact. Every accelerator chip connects across all planes in a fully meshed multi-rail integration, allowing both intra-rail and cross-rail traffic to traverse the full bisection bandwidth within two switch hops without requiring an extra switching tier. Managing Micro-Bursts with In-Band Congestion Signaling (11:31) Because AI training generates synchronized line-rate micro-bursts, standard network telemetry based on 30-second averages obscures severe network stress, reporting modest 10% average utilization while switch buffers periodically redline at 100% capacity. To provide sub-millisecond visibility, Virgo pairs streaming management plane telemetry from switch CPUs with data plane Congestion Signaling (CSIG). Lee highlighted CSIG as a practical evolution of In-band Network Telemetry (INT). While original INT stacked metadata at each hop and swelled packet sizes, CSIG maintains a fixed-size header tag carrying only the bottleneck link's telemetry. Intermediate switches update the tag only if their local congestion state exceeds the recorded value. Receivers mirror this bottleneck data back in ACK packets, allowing sender transport layers to dynamically throttle transmission while clarifying whether bottlenecks arise from first-hop multipath issues or last-hop incast congestion. Multipath Routing and Metro-Scale Expansion (10:26) Virgo routes traffic across its multiplane topology using hardware-based adaptive routing, which allows switches to dynamically steer packets around congested ports and failed links. For bare-metal cloud customers running custom host networking stacks, Virgo supports SRv6 source routing, enabling hosts to directly monitor, tag, and dictate precise end-to-end packet paths across the fabric. Looking beyond individual facilities, Lee outlined Google's roadmap to extend Virgo fabrics across metro regions spanning hundreds of kilometers. Running high-performance RDMA over long-distance optical links relies on extending CSIG congestion feedback, an effort Google is standardizing within the Ultra Ethernet Consortium (UEC). Lee stressed that multi-vendor switch silicon interoperability and unified open standards will be critical as AI fabrics expand across distributed campuses. Lee concluded that earlier networking concepts like INT and source routing have found an essential proving ground in AI supercomputing. While Google is actively standardizing CSIG within the Ultra Ethernet Consortium to enable multi-vendor interoperability and long-haul RDMA, scaling ultra-low-latency fabrics across metropolitan distances remains an ongoing industry challenge.

精炼阅读摘要

Google 网络工程师 Jeongkeun "JK" Lee 在 @Scale 大会上介绍了专为人工智能工作负载设计的向外扩展数据中心架构 Virgo，强调通过软硬件协同设计、扁平化拓扑与高精度遥测，使底层硬件故障对上层 AI 任务完全透明。 AI 时代驱动的基础设施与设计原则转向 (0:03) Google 数据中心网络经历了从内部服务（Jupiter 与 Apollo）、流媒体优化（BBR 与 QUIC）到云端硬件卸载（PSP 与 Falcon）的演进。Lee 指出，面对 2026 年 AI 时代芯片带宽与模型规模的指数级增长，网络基础设施的设计重点转向满足极度严苛的延迟限制，以及在跨园区、多建筑扩展时实现透明的故障恢复能力。为此，Virgo 确立了五项核心设计原则：将网络与园区级 AI 超级计算机进行一体化协同设计、使交换机与光模块与特定芯片保持紧密的硬件生命周期协同、在物理架构中内生容错弹性、采用高分辨率遥测以最大化吞吐量，并通过精简网络层级来加速数以十亿美元计的 AI 集群交付部署。分层架构与两层扁平多平面拓扑 (4:37) 在数据中心内部，Google 将网络解耦为三个独立演进的层面：底层是基于 NVLink 或光交换 ICI 的 Pod 内极低延迟 Scale-Up 网络；中间层是提供全集群无阻塞连接的专用后端 Scale-Out 网络 Virgo；顶层则是面向分布式存储与通用计算的前端 Jupiter 网络。后端 Virgo 与前端 Jupiter 分离，使 AI 计算节点能够按年快速升级 GPU 或 TPU，而无需承担全系统范围的改造风险。通过最大化交换机基数，Google 将传统的 3 层 Clos 网络折叠为两层非阻塞拓扑，大幅降低了转发跳数与尾延迟抖动。细链路多平面设计与网状多轨集成 (6:50) 为了在数千颗芯片规模下维持两层拓扑，Virgo 采用了基于“细链路”（Skinny Links）的多平面架构。系统不进行端口聚合，而是直接以 SerDes 的原生速率运行链路（例如当前部署中的每通道 200 Gbps），从而最大化交换机基数。以单颗 GPU 产生 1.6 Tbps 带宽为例，系统会拆解为 8 条 200 Gbps 细链路并接入 8 个独立的网络平面；当其中一个平面发生故障时，节点仍能保留 87.5%（7/8）的双向对分带宽，避免传统双归属网络中直接损失 50% 甚至全部带宽的情况。同时，Virgo 引入了全网状多轨集成，使每颗加速器芯片均接入全部网络平面，跨轨与轨内流量均能在两跳交换机内完成传输。多路径自适应路由与亚毫秒级流量遥测 (10:26) 针对 AI 训练中高度同步的高突发流量，Virgo 结合了基于交换机硬件的自适应路由以绕开拥塞链路，并为裸机云客户提供基于 SRv6 的源路由能力，允许主机网络栈自主标记和指定端到端网络路径。由于平均化指标容易掩盖瞬时过载，Lee 展示了两种遥测维度的对比：在 30 秒平均视图中网络表现为负载仅约 10% 的健康状态，但在亚毫秒视图下，周期性微突发每隔数秒就会导致交换机缓冲区完全达到 100% 饱和。Virgo 通过交换机 CPU 直接导出亚毫秒级端口利用率，以实现对瞬时拥塞的精细化监控。 CSIG 拥塞标记与超以太网标准化 (13:55) 为替代逐跳累积元数据导致包头膨胀的传统带内网络遥测（INT），Google 研发了轻量化的拥塞信号机制 CSIG。CSIG 在数据包中使用固定大小的报头标记，仅当交换机检测到自身拥塞状况（如基于端口容量归一化的可用带宽或最大跳延迟）比报头已有数值更严重时，才更新该字段。目标主机接收到瓶颈链路信息以及多路径或尾部拥塞提示后，会在 ACK 确认包中将指标回传给发送端，使主机传输层能够瞬时调节流量。目前 Google 正在超以太网联盟（UEC）推动 CSIG 的标准化，相关技术细节已在 SIGCOMM 会议上发表。 Google 下一步计划将 Virgo 架构从单一数据中心拓展至跨越数百公里的城域环境，重点解决跨长途光链路运行 RDMA 的拥塞控制难题，并在超以太网联盟框架下联合多家芯片厂商实现多供应商互操作性。

Detailed notes — zh

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布，内容为 Google 的 Jeongkeun 'JK' Lee 在圣克拉拉会议中心（SCCC）举办的 @Scale 大会上的现场演讲。演讲主要介绍了 Google 专为 AI 东西向横向扩展打造的新一代数据中心 Fabric 架构——Virgo，探讨其面向确定性延迟、韧性容灾、高精度遥测以及多建筑规模扩展的设计理念，并指出其将为 Google Cloud 搭载 NVIDIA Vera Rubin NVL72 的 A5x 实例及下一代 TPU 提供底层网络支撑。 ## TL;DR JK Lee 系统阐述了 Google 从搜索时代到 2026 年 AI 时代网络架构的演进，重点介绍了专为超大规模 AI 算力设计的 Virgo 横向扩展 Fabric。Virgo 采用扁平两层拓扑、Multi-Plane 解耦与单通道 200G SerDes 的细链路（Skinny Links）机制，打破传统向下兼容包袱，实现优雅降级与全网状对分带宽。同时，Virgo 深度集成了自适应路由与 SRv6 源路由，并将传统 INT 升级为定长报头的 CSIG 带内拥塞信令，配合亚毫秒级流式遥测解决了微突发盲区，最后展望了跨城际长距 RDMA 互联与 UEC 多厂商互操作性的未来方向。 ## Atmosphere & style 演讲展现出浓厚的一线工程极客底色与严谨务实的系统设计哲学，JK 的表述沉稳自信、条理清晰，既有对技术史演进的宏观把握，又有深挖底层网络机制的热忱。在展示 30 秒平均粒度与亚毫秒级流式遥测对比时，直观暴露微突发打满缓冲区的现场演示极具冲击力，瞬间打破了传统监控的平稳假象。随着回顾早年 P4/INT 创新在如今 AI 大模型浪潮中全面复兴，整场演讲流露出一种技术经历沉淀后重获新生的激昂感与行业使命感。 ## Bulletin - [00:03](https://www.youtube.com/watch?v=WuzpHLYnre0&t=3s) Google 正式发布专为 AI 算力横向扩展打造的后端网络 Fabric「Virgo」，重点支撑新一代 TPU 以及搭载 NVIDIA Vera Rubin NVL72 的 Google Cloud A5x 裸金属实例。 - [06:50](https://www.youtube.com/watch?v=WuzpHLYnre0&t=410s) Virgo 采用扁平化两层无阻塞拓扑与基于原生 SerDes 线速（200 Gbps）的 Skinny Links 机制，通过多平面（Multi-Plane）解耦实现优雅降级，单平面失效仅损失 1/N 算力带宽。 - [13:47](https://www.youtube.com/watch?v=WuzpHLYnre0&t=827s) 针对 AI 纳秒级微拥塞，Google 推出固定包头长度的带内拥塞信令协议 CSIG，克服了传统 INT 堆叠包头膨胀的缺陷，已被正式纳入 UEC 标准草案。 - [16:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=986s) Virgo 架构未来将向跨城际数百至数千公里的远距离 RDMA 扩展，并积极推动多交换芯片厂商之间的协议互操作性。 ## Key takeaways - [00:03](https://www.youtube.com/watch?v=WuzpHLYnre0&t=3s) Google 数据中心网络历经互联网时代（Jupiter/OCS）、流媒体时代（BBR/QUIC）和云时代（PSP/Falcon），演进至 2026 年以专用超算网络为特征的 AI 时代。 - [02:00](https://www.youtube.com/watch?v=WuzpHLYnre0&t=120s) 在跨机房乃至跨园区的超大 AI 集群中底层故障常态化，网络设计首要目标是将软硬件故障对上层训练计算完全透明化。 - [02:46](https://www.youtube.com/watch?v=WuzpHLYnre0&t=166s) Virgo 确立了四项核心原则：单一超算场景定制、生命周期与 AI 硬件深度绑定并摒弃传统向下兼容包袱、内置高精度遥测、精简网络层级以加速交付。 - [04:37](https://www.youtube.com/watch?v=WuzpHLYnre0&t=277s) Google 坚持将前端存储网络（Jupiter）与后端横向扩展 Fabric（Virgo）物理和逻辑解耦，实现独立迭代演进并隔绝故障扩散。 - [06:50](https://www.youtube.com/watch?v=WuzpHLYnre0&t=410s) 利用单交换芯片的超高端口分度，Virgo 仅用两层交换矩阵便构建起跨机房乃至园区的超大规模扁平化无阻塞互联。 - [07:40](https://www.youtube.com/watch?v=WuzpHLYnre0&t=460s) 采用单通道 200 Gbps 的 Skinny Links 取代捆绑大管道，配合独立的 Fabric 平面，使 8 平面系统单点故障时仍保有 87.5% 的网络吞吐。 - [09:30](https://www.youtube.com/watch?v=WuzpHLYnre0&t=570s) 相较于传统的 Rail-Alignment 架构，Virgo 采用全网状（Full Mesh）交织连接，无需第三层交换机即可实现任意加速卡跨 Rail 通信吃满全带宽。 - [10:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=626s) Virgo 在转发面原生提供毫秒级自主规避拥塞的自适应路由，并针对裸金属用户提供基于 SRv6 的端到端源路由精确控制能力。 - [12:40](https://www.youtube.com/watch?v=WuzpHLYnre0&t=760s) 实测演示证明，传统 30 秒采样粗粒度监控会严重掩盖瞬时微突发（Micro-burst），而亚毫秒级流式遥测能直接暴露缓冲区打满至 100% 的真实拥塞。 - [13:47](https://www.youtube.com/watch?v=WuzpHLYnre0&t=827s) 作为 INT 的进化形态，CSIG 采用定长报头仅记录路径最恶劣瓶颈，通过 ACK 反射机制精准区分首跳哈希不均与最后一跳 Incast 拥塞。 - [15:40](https://www.youtube.com/watch?v=WuzpHLYnre0&t=940s) CSIG 机制已进入超级以太网联盟（UEC）规范草案，相关研究成果已发表于计算机网络顶会 SIGCOMM。 - [16:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=986s) Virgo 正在攻克跨城际都市圈数百至数千公里的长距离 RDMA 传输难题，并重点解决多芯片厂商交换设备在同一织网内的跨生态互通。 ## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=WuzpHLYnre0&t=3s) 演进背景：从搜索时代到 AI 时代的网络拐点来自 Google 的 Jeongkeun "JK" Lee 登台向与会者致谢。他回忆起自己早在 2016 年就第一次参加 @Scale 大会，自那时起便一直非常推崇该会议公开分享产业一线视角的传统。今天，他怀着十分谦逊的心情，代表 Google 背后负责研发与正在进行大规模落地的众多团队，正式介绍专为 AI 打造的全新横向扩展数据中心网络架构——Virgo。回顾 Google 数据中心网络的技术创新史，团队走过了数个根本性的分水岭： 1. **互联网与搜索时代**：为了支撑 Search 和 Gmail 等全球级服务的爆发式扩张，Google 开创了 Jupiter 数据中心架构，并搭配著名的 Orion 软件定义网络（SDN）控制平面；同时推出了 Apollo，开创在数据中心内使用光路交换（OCS, Optical Circuit Switching）技术的先河。 2. **流媒体时代**：面对低抖动、高质量实时视频流的分发诉求，Google 先后研发出 TCP BBR、QUIC 等全新的拥塞控制与传输协议。 3. **云时代**：网络性能向硬件底层迁移卸载，例如实现线速安全的 PSP 加密，以及将高可靠传输协议 Falcon 深度卸载到 NIC 硬件。 4. **2026 AI 时代**：AI 工作负载带来了一场颠覆性的结构转型。当前 AI 模型的体量呈指数级暴增，强力拉动每颗加速器芯片所需的网络带宽同步飞涨，并对延迟提出了极其苛刻的上限约束。更关键的是，现代 AI 超级计算机的网络跨越了园区内的多个机房建筑（multi-building / multi-campus）。在这种史无前例的超大规模下，"一切组件都会发生故障"（everything breaks），网络架构必须做到将底层软硬件故障对上层 AI 训练与计算任务彻底隐形。 ```mermaid flowchart LR accTitle: Google 数据中心网络架构演进历程 accDescr: 展示从互联网时代到 AI 时代 Google 核心网络技术的演化 subgraph Timeline ["网络技术演化历程"] direction LR S1["互联网时代 Jupiter / Orion / Apollo OCS"] --> S2["流媒体时代 TCP BBR / QUIC"] S2 --> S3["云时代 PSP 加密 / Falcon NIC 卸载"] S3 --> S4["2026 AI 时代 Virgo 超级计算 Fabric"] end ``` --- ### [02:46](https://www.youtube.com/watch?v=WuzpHLYnre0&t=166s) Virgo 核心设计原则：从通用设施到专用超算架构为了应对 AI 时代的严苛挑战，Google 在构思 Virgo 时确立了一套清晰的核心设计原则： 1. **专为单一场景定制的园区级超算网络**：横向扩展网络必须作为整个园区超级计算机的有机组成部分进行一体化设计，只针对 AI 这一特定场景做到极致优化，而不再是过去那种兼顾各种杂项负载的通用基础设施。这意味着交换机、光模块与光纤等所有硬件规格，都必须与所承载的 AI 硬件及超算体系严丝合缝地步调一致（in lockstep）。 2. **生命周期深度绑定，摒弃传统向下兼容包袱**：横向扩展网络的生命周期与 AI 算力集群高度重合。因此，传统网络中那些苛刻的向下兼容（backward compatibility）已不再是强需求。弹性与韧性（resiliency）必须内置在架构中，绝对不能因为单个链路或交换机硬件损坏而导致整个 AI 训练任务发生崩溃中断。 3. **内建超高精度遥测（Telemetry）**：必须为整套 Fabric 配备高精度遥测能力，使拥塞控制协议和集群管理系统能够榨干网络的最后一滴性能，并保持最高可用性。 4. **精简网络层级，保障极速交付**：全球对 AI 算力的渴求空前迫切，耗资数十亿美元的 AI 集群必须以最快速度交付给客户。最小化交换层级至关重要，因为机架和光纤连接数量的大幅减少，直接意味着物理开通与上线周期的成倍缩短。 --- ### [04:37](https://www.youtube.com/watch?v=WuzpHLYnre0&t=277s) 数据中心三层网络布局与前后端网络解耦在宏观拓扑结构上，Google 数据中心构建了三套各自独立、职责专一，但协同运行在同一算力域下的网络层级： ```mermaid flowchart TD accTitle: 数据中心三层网络互联架构 accDescr: 展示前端存储通信、后端纵向扩展与横向扩展的三层网络分离架构 subgraph FrontEnd ["前端与全局网络"] JUPITER["Jupiter 前端网络 (访问分布式存储与通用算力 / 跨集群共享)"] end subgraph ScaleOut ["集群级 Scale-Out Fabric"] VIRGO["Virgo 横向扩展网络 (专用后端 / 100% 无阻塞 / 极速互联多 Pod)"] end subgraph ScaleUp ["Pod 内 Scale-Up 互联"] NVLINK["GPU: NVLink 互联"] TPU_ICI["TPU: 光交换 ICI 互联"] end JUPITER <--> ScaleOut ScaleOut <--> ScaleUp ``` - **纵向扩展网络（Scale-up）**：位于机架/Pod 内部，以极高带宽和极低延迟直接互联加速芯片。针对 GPU 方案采用 NVIDIA NVLink；针对 TPU 方案则采用基于光路交换的芯片间互联通道（ICI, Inter-Chip Interconnect）。 - **横向扩展后端网络（Scale-out / Virgo）**：专用后端 Fabric，负责在集群级别为跨 Pod 的大规模加速芯片群提供海量全互联带宽。 - **前端网络（Front-end / Jupiter）**：Google 历经验证的核心网络，充当连接分布式存储及通用计算资源的高速公路，可被同一栋机房内的多个 AI 集群无缝共享。 Google 坚决将前端网络与后端专用网络进行物理及逻辑解耦，原因在于： - **独立的演进节奏（Independent Cadence）**：前后端可以独立迭代。由于 Google 每年都要推陈出新、快速引入多代不同规格的 GPU 与 TPU 硬件，专用网络必须与具体的 AI 芯片规格精准匹配，不受通用网络周期的拖累。 - **故障隔离与无感升级**：升级或维护其中一套网络域时，完全规避了波及全集群系统级中断的风险。 - **设计目标与优化方向截然不同**：后端的 Virgo 追求 100% 绝对无阻塞的互联吞吐性能；而前端的 Jupiter 则更侧重于总拥有成本（TCO）的优化以及渐进式弹性扩容能力。 --- ### [06:50](https://www.youtube.com/watch?v=WuzpHLYnre0&t=410s) 扁平化两层拓扑、Multi-Plane 设计与 Skinny Links 机制深入到 Virgo 的拓扑设计，JK 梳理了三大核心技术特征： 1. **压缩为扁平两层无阻塞拓扑**：彻底抛弃以往复杂的多层交换结构，最大限度利用单颗交换芯片的超高端口分度（Switch Radix），仅用两层交换矩阵便构建起庞大的无阻塞网络，从而将传输延迟和时延抖动降到极限。 2. **多平面（Multi-Plane）解耦与独立控制域**：在加速器机架接入网络时采用多平面方案，并在每个平面部署各自独立的控制域，将任何硬件故障彻底隔离在单一平面内。 3. **跨机房与跨园区级无缝扩展**：Virgo 的互连能力不仅覆盖单栋建筑，更可原生扩展至园区内的多栋建筑之间。 ```mermaid flowchart TD accTitle: Virgo Multi-Plane 与 Skinny Links 架构 accDescr: 解释基于 SerDes 速率的细链路如何构建多网络平面及降级弹性 subgraph Accelerator ["单颗 AI 芯片 (例如 1.6 Tbps 总带宽)"] CHIP["GPU / TPU 加速芯片"] end subgraph SkinnyLinks ["Skinny Links (细链路：200G SerDes 线速)"] L1["Link 1 (200G)"] L2["Link 2 (200G)"] L3["Link 3 (200G)"] L4["Link 4 (200G)"] L5["Link 5 (200G)"] L6["Link 6 (200G)"] L7["Link 7 (200G)"] L8["Link 8 (200G)"] end subgraph FabricPlanes ["Virgo 独立 Fabric 平面 (N=8)"] P1["Plane 1"] P2["Plane 2"] P3["Plane 3"] P4["Plane 4"] P5["Plane 5"] P6["Plane 6"] P7["Plane 7"] P8["Plane 8"] end CHIP --> L1 & L2 & L3 & L4 & L5 & L6 & L7 & L8 L1 --> P1 L2 --> P2 L3 --> P3 L4 --> P4 L5 --> P5 L6 --> P6 L7 --> P7 L8 --> P8 ``` 关于当今行业广泛采用的“多平面（Multi-Plane）”架构，Virgo 从底层采用了名为 **细链路（Skinny Links）** 的机制： - **定义**：无论连接加速服务器还是交换机互联，链路均以底层 SerDes 通道的原生最低线速运行（在最新一代技术中为单通道 200 Gbps），而非将多条通道捆绑为粗管道。这最大化释放了单交换芯片的端口分度（Radix），奠定了两阶段（Two-stage）超大规模组网的基础。 - **按需扩展平面数量**：每个芯片支持的 Skinny Link 数量决定了平面的数量 $N$。以单颗提供 1.6 Tbps 后端带宽的 GPU 为例，在 200G SerDes 速率下，每颗芯片天然衍生出 8 条 Skinny Links，直连 8 个物理独立的 Fabric 平面。如果未来单芯片带宽持续翻倍而交换机容量一时受限，系统只需横向平推增加平面数即可，扩展性非常优美。 - **平滑优雅降级（Graceful Degradation）**：如果 $N$ 个平面中的某一个出现故障，芯片仍能保有 $(N-1)/N$ 的可用带宽。例如在 8 平面系统中掉了一个平面，芯片依然保留 87.5% 的网络吞吐。这彻底终结了传统网络中单链路或单交换机宕机瞬间吞吐断崖式下跌 50% 甚至归零（100% 损失）的灾难。 - **与传统 Rail-Alignment 的本质区别**：传统 Rail-Only 设计中芯片通常按固定导轨索引严格绑定；而在 Virgo 中，无论加速卡位于哪个物理 Rail，每颗芯片都会与全部 $N$ 个网络平面进行全网状互联（Full Mesh）。例如一台拥有 4 颗 GPU 的主机构建为 4 个 Rail 体系，与后端的 8 个网络平面全面交织打通。这使得不仅 Rail 内部流量畅通无阻，跨 Rail 的任意通信也能在扁平的两层网络中直接吃满全部对分带宽，免去了强加第三层昂贵交换矩阵的负担。 --- ### [10:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=626s) 多路径路由与高分辨率网络遥测方案在大规模 Fabric 调度中，多路径路由（Multipath Routing）是维持极低长尾延迟的核心。Virgo 在交换机层面原生提供两种数据包级负载均衡技术： - **自适应路由（Adaptive Routing）**：交换机具备本地智能，能够动态感知拥塞和局部硬件故障，并毫秒级自主规避；同时能对下游及远端故障做出极速绕路反应。 - **基于 SRv6 的源路由（SRv6 Source Routing）**：针对裸金属云客户，用户可自带主机网络栈（例如 MR-C 栈），利用 SRv6 直接对每一个数据包进行端到端全路径精确监控与干预。尽管兼具两层扁平架构与动态路由，AI 训练时各节点高度同步触发的“线速突发”（line-rate bursts）依然会在纳秒级产生可怕的微拥塞（micro-congestion），而传统遥测技术对这类短时毛刺完全视而不见。为此，Google 将**高分辨率遥测**确立为 Virgo 的第一等公民（first-class citizen），打通软硬两层感知链路： 1. **控制与管理平面遥测**：由交换机 CPU 流式推送高精度的性能指标（例如端口发送队列的实时利用率等），供全局控制器进行集中式性能调优、链路诊断与故障排查。 2. **数据平面线速遥测（CIC 机制）**：将同样极高分辨率的网络健康与拥塞指标，以内联带内信号方式注入数据平面，通过名为 **CIC** 的带内拥塞信令机制实时送达终端主机网络栈与传输协议层，使端侧传输层能够立刻采取精准流控措施。 ### [12:40](https://www.youtube.com/watch?v=WuzpHLYnre0&t=760s) 高分辨率流式遥测演示：打破传统监控的假象 JK 开始展示高分辨率流式遥测（Streaming Telemetry）的现场演示。监控大屏左侧呈现工作负载的整体视图，右侧则是交换机级别的微观视图。在传统的标准监控模式下，数据以 30 秒为采样粒度进行汇总展示，此时整个网络状态看起来极其健康、平稳，似乎保留了极为充裕的带宽余量（Headroom）。JK 随即指出，这种看似健康的视图实际上具有严重的欺骗性，因为粗粒度的平均统计完全抹平了瞬时的微突发（Micro-burst）。紧接着，JK 将监控切换到了亚毫秒（Sub-millisecond）级分辨率。原本风平浪静的健康基线瞬间消失，屏幕上骤然冒出一根根代表 100% 满载饱和的绿色立柱。尽管传统的蓝色平均指标依然声称链路负载只有区区 10%，但在真实的线路上，剧烈的流量突发几乎每隔几秒钟就会将交换机缓冲区推至极限边缘（Redlining）。JK 感慨于能够直接洞察物理线路级别的极细粒度网络行为，并强调 Google 正在利用这种超高精度的遥测洞察来全面加固其底层基础设施与日常运维系统。 - 传统 30 秒粒度监控会将网络突发展现为平均 10% 负载，掩盖了真实的微突发堵塞。 - 亚毫秒级视图能够暴露每隔数秒即打满缓冲区至 100% 饱和状态的微突发事件。 - 该高精细度流式遥测技术现已用于 Google 内部基础设施的架构加固与生产日常运维。 --- ### [13:47](https://www.youtube.com/watch?v=WuzpHLYnre0&t=827s) 从 INT 到 CSIG：带内拥塞信标与数据平面闭环在管理平面获取到指标之后，下一步是如何让这些数据直接暴露给数据平面协议，以便终端主机的网络协议栈能够即时消费。JK 介绍了关键机制——CSIG（Congestion Signaling，带内拥塞信号标记）。CSIG 能够直接在数据包报头中携带交换机织网中检测到的瓶颈点信息。JK 解释道，如果这听起来像早期的 INT（In-band Network Telemetry，带内网络遥测），那是因为它确实源于此；但与 INT 相比，CSIG 是一个更加务实、高效的进化版本。INT 过去采取逐跳堆栈元数据（Stacking）的方式，导致报头开销随着跳数膨胀；而 CSIG 则采用固定长度的报头格式，只记录整个路径上唯一的瓶颈点指标。以基于端口容量归一化的可用带宽为例（其数值实际上是端口发送利用率的倒数），数据包在从发送端穿透至接收端的过程中途径多跳交换机。每台途径的交换机只有在发现自身的拥塞状况比数据包当前记录的数值“更差”时，才会覆写更新 CSIG 标签。这样一来，接收端收到的就是整条链路真正的瓶颈信息，同时还伴随着拥塞成因的线索——例如是由首跳上行多路径（Multipathing）不均引起的，还是由最后一跳汇聚（Incast）引发的。随后，接收端通过 ACK 确认包将这些拥塞特征反射回发送端，实现精准的流控与拥塞控制。 ```mermaid flowchart LR accTitle: CSIG 数据路径瓶颈更新与 ACK 反射机制 accDescr: 展示数据包在经过各跳交换机时如何条件性更新 CSIG 标签，并在接收端通过 ACK 包将拥塞特征反馈给发送端。 subgraph DataPath["数据转发路径 (Data Path)"] S["发送端 (Sender)"] H1["第 1 跳交换机 (Hop 1 Switch)"] H2["第 2 跳交换机 (Hop 2 瓶颈点)"] H3["第 3 跳交换机 (Hop 3 Switch)"] R["接收端 (Receiver)"] end S -->|"数据包 (初始 CSIG)"| H1 H1 -->|"写入自身拥塞指标"| H2 H2 -->|"检测到更恶劣状态 覆写为最差瓶颈指标"| H3 H3 -->|"自身状态优于瓶颈 保持报头透传"| R R -.->|"ACK 反射瓶颈与拥塞成因 (首跳多路径 / 尾跳 Incast)"| S ``` 这一演进令 JK 倍感自豪，因为多年前他在 Barefoot Networks 工作期间，主导开发的第一个 P4 语言应用正是 INT。如今看到作为 INT 升华形态的 CSIG 在 Virgo 织网中实现交换机与网卡（NIC）间的端到端全链路支持，他感到极其激动。在超级以太网联盟（Ultra Ethernet Consortium, UEC）的通力合作下，CSIG 已被正式纳入 UEC 规范草案；而 Google 团队也在刚过去的 SIGCOMM 顶会上发表了关于该协议的详细论文。 - CSIG 解决了 INT 堆叠元数据导致包头膨胀的痛点，采用定长数据包头只记录沿途最恶劣的瓶颈数据。 - CSIG 报头能够灵活承载多种度量指标，如逐跳最大排队时延（Delay）或归一化可用带宽。 - 该机制能向端节点提供细粒度诊断线索，区分首跳上行哈希不均与尾跳 Incast 拥塞。 - CSIG 已经进入 UEC（Ultra Ethernet Consortium）标准化流程，技术细节由 Google 团队发表于 SIGCOMM。 --- ### [16:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=986s) 总结与展望：长距互联、多厂商协同与 AI 时代的网络复兴在演讲的收尾阶段，JK 归纳了 Virgo 作为 Google 新一代横向扩展织网的核心价值：它专为极大规模下的高性能与高可靠性而生，贯穿管理平面与数据平面高分辨率遥测能力。目前，Virgo 正在为 Google 新一代 GPU 与 TPU 基础设施提供底层支撑，包括基于 NVIDIA Vera Rubin 架构的 NVL72 平台（Google Cloud A5x 裸金属实例）以及最新的 TPU 系统。展望未来演进，Google 正致力于将 Virgo 的覆盖范围横向推向跨城际都市圈（Metros），实现相距数百甚至数千公里的远距离横向扩展。正如 Omar 在大会 Keynote 中所强调的，在长距离连接（Long Haul）上实现高吞吐、低延迟的 RDMA 通信是一项极其艰巨的挑战。JK 坚信，基于 CSIG 及其扩展版本所提供的精细拥塞感知，将是攻克长距 AI 互联难题的核心钥匙。此外，随着算力集群在物理地域上的分布式扩展，数据中心织网混用多家芯片厂商交换机的概率大幅提升，推动协议统一与多厂商设备互操作性（Interoperability）已成为业界的迫切需求。 JK 回顾道，当下是投身网络技术领域最令人兴奋的黄金时代。在过去十年中，网络社区构想并打磨了诸如 INT、数据中心源路由（Source Routing）等诸多前沿创新；其中不少技术在诞生初期曾一度被外界忽视，如今却在 AI 基础设施这一严苛的真实考场中找到了不可替代的用武之地。最后，JK 呼吁整个技术社区在 UEC 及开源生态中深化合作，共同推进 AI 大规模扩展时代的网络变革，并在幻灯片展示了获取 Google AI 网络架构与 WAN 方案资料的二维码。 - Virgo 织网不仅支撑大规模 TPU 系统，同时也是 Google Cloud 搭载 NVIDIA Vera Rubin 平台的 A5x 裸金属实例的核心网络底座。 - 研发路线图正向跨数百至数千公里的城际长距（Long-haul）RDMA 扩展迈进。 - 织网规模扩展催生了不同硬件厂商交换机混部的需求，协议统一和跨厂商互通成为重要协作方向。 - INT 与数据中心源路由等过去十年的网络创新，正在现代 AI 规模化通信架构中迎来全面的应用复兴。 ## Notable quotes > [00:03](https://www.youtube.com/watch?v=WuzpHLYnre0&t=3s) 一切组件都会发生故障 — Jeongkeun "JK" Lee > [02:46](https://www.youtube.com/watch?v=WuzpHLYnre0&t=166s) 横向扩展网络必须作为整个园区超级计算机的有机组成部分进行一体化设计，只针对 AI 这一特定场景做到极致优化，而不再是过去那种兼顾各种杂项负载的通用基础设施。 — Jeongkeun "JK" Lee > [02:46](https://www.youtube.com/watch?v=WuzpHLYnre0&t=166s) 弹性与韧性（resiliency）必须内置在架构中，绝对不能因为单个链路或交换机硬件损坏而导致整个 AI 训练任务发生崩溃中断。 — Jeongkeun "JK" Lee > [04:37](https://www.youtube.com/watch?v=WuzpHLYnre0&t=277s) 后端的 Virgo 追求 100% 绝对无阻塞的互联吞吐性能；而前端的 Jupiter 则更侧重于总拥有成本（TCO）的优化以及渐进式弹性扩容能力。 — Jeongkeun "JK" Lee > [06:50](https://www.youtube.com/watch?v=WuzpHLYnre0&t=410s) 无论连接加速服务器还是交换机互联，链路均以底层 SerDes 通道的原生最低线速运行……而非将多条通道捆绑为粗管道。 — Jeongkeun "JK" Lee > [06:50](https://www.youtube.com/watch?v=WuzpHLYnre0&t=410s) 如果 N 个平面中的某一个出现故障，芯片仍能保有 (N-1)/N 的可用带宽。 — Jeongkeun "JK" Lee > [10:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=626s) 高分辨率遥测确立为 Virgo 的第一等公民（first-class citizen） — Jeongkeun "JK" Lee > [12:40](https://www.youtube.com/watch?v=WuzpHLYnre0&t=760s) 在传统的标准监控模式下，数据以 30 秒为采样粒度进行汇总展示，此时整个网络状态看起来极其健康、平稳，似乎保留了极为充裕的带宽余量 — Jeongkeun "JK" Lee > [12:40](https://www.youtube.com/watch?v=WuzpHLYnre0&t=760s) 粗粒度的平均统计完全抹平了瞬时的微突发（Micro-burst）。 — Jeongkeun "JK" Lee > [13:47](https://www.youtube.com/watch?v=WuzpHLYnre0&t=827s) CSIG 解决了 INT 堆叠元数据导致包头膨胀的痛点，采用定长数据包头只记录沿途最恶劣的瓶颈数据。 — Jeongkeun "JK" Lee > [16:26](https://www.youtube.com/watch?v=WuzpHLYnre0&t=986s) 当下是投身网络技术领域最令人兴奋的黄金时代。 — Jeongkeun "JK" Lee ## Mentioned Jeongkeun "JK" Lee, Google, Google Cloud, @Scale, Virgo, Jupiter, Orion, Apollo OCS, TCP BBR, QUIC, PSP, Falcon, NVIDIA, Vera Rubin, NVL72, A5x, TPU, NVLink, ICI, Skinny Links, SRv6, CIC, CSIG, INT, Barefoot Networks, P4, UEC (Ultra Ethernet Consortium), SIGCOMM, Omar, RDMA ## Video description Jeongkeun "JK" Lee from Google presents "Virgo: Scale-out Data Center Network for AI " live at the Santa Clara Convention Center. To address the unique requirements of AI networking, we leverage our proven Jupiter network for north-south traffic and are introducing Virgo, a new fabric for east-west scale-out communication. Virgo is a dedicated accelerator-to-accelerator fabric optimized for massive horizontal scale. It is engineered for deterministic latency and maximum resilience to provide high goodput for the ML workload. Virgo will enable the new Google Cloud's A5x bare-metal instances, which run the NVIDIA Vera Rubin NVL72 platform. It is also a key building block for next-gen TPUs. This talk will discuss Virgo's design principles for resilience, observability, deployability, and multi-building scale. Learn more about @Scale here: atscaleconference.com [Source](https://www.youtube.com/watch?v=WuzpHLYnre0)

Source quotes

[0:03] Jeongkeun "JK" Lee: 一切组件都会发生故障 [2:46] Jeongkeun "JK" Lee: 横向扩展网络必须作为整个园区超级计算机的有机组成部分进行一体化设计，只针对 AI 这一特定场景做到极致优化，而不再是过去那种兼顾各种杂项负载的通用基础设施。 [2:46] Jeongkeun "JK" Lee: 弹性与韧性（resiliency）必须内置在架构中，绝对不能因为单个链路或交换机硬件损坏而导致整个 AI 训练任务发生崩溃中断。 [4:37] Jeongkeun "JK" Lee: 后端的 Virgo 追求 100% 绝对无阻塞的互联吞吐性能；而前端的 Jupiter 则更侧重于总拥有成本（TCO）的优化以及渐进式弹性扩容能力。 [6:50] Jeongkeun "JK" Lee: 无论连接加速服务器还是交换机互联，链路均以底层 SerDes 通道的原生最低线速运行……而非将多条通道捆绑为粗管道。 [6:50] Jeongkeun "JK" Lee: 如果 N 个平面中的某一个出现故障，芯片仍能保有 (N-1)/N 的可用带宽。 [10:26] Jeongkeun "JK" Lee: 高分辨率遥测确立为 Virgo 的第一等公民（first-class citizen） [12:40] Jeongkeun "JK" Lee: 在传统的标准监控模式下，数据以 30 秒为采样粒度进行汇总展示，此时整个网络状态看起来极其健康、平稳，似乎保留了极为充裕的带宽余量 [12:40] Jeongkeun "JK" Lee: 粗粒度的平均统计完全抹平了瞬时的微突发（Micro-burst）。 [13:47] Jeongkeun "JK" Lee: CSIG 解决了 INT 堆叠元数据导致包头膨胀的痛点，采用定长数据包头只记录沿途最恶劣的瓶颈数据。 [16:26] Jeongkeun "JK" Lee: 当下是投身网络技术领域最令人兴奋的黄金时代。

Source quotes

[0:03] Jeongkeun "JK" Lee: At this scale we know that everything breaks, and we should make these failures invisible to the AI workload. [11:31] Jeongkeun "JK" Lee: While the blue line still claims 10% load, the traffic bursts are redlining the switch buffer every few seconds. [16:26] Jeongkeun "JK" Lee: Some innovations we as a community have worked on so far for the last decade, like INT or source routing... were overlooked initially, but now they are finding real proving ground in AI.

Original transcript

[0:03] Hey, thanks for having me here. Um, 2016 [0:06] was my first time attending the ask [0:10] event and since then I've really [0:12] appreciated the unique industry [0:14] perspectives openly shared in this event [0:16] series. So that makes me very grateful [0:19] about this opportunity talk about Virgo [0:22] the new scale out data center fabric for [0:24] AI. I'm humbly representing many teams [0:27] at Google behind the development and [0:30] then at scale deployment of VGO fabrics [0:32] happening right now. [0:35] So we know that we are living in the [0:36] midst of AI era but reflecting back at [0:39] Google we have crossed several [0:42] fundamental inflection points in the [0:44] data center networking innovations. [0:46] Starting from at the bottom internal era [0:49] to scale the global services like a [0:52] search and Gmail we started the new [0:55] Jupiter data center architecture which [0:58] is paired with well-known Orion SDN [1:00] control plane. We also pioneered Apollo [1:04] which is the first time use of optical [1:07] circuit switching technology in the data [1:09] center. [1:11] Next the streaming era. Uh to deliver [1:14] the real time high quality video stream [1:17] with the low jitter we came up with the [1:19] new congestion control schemes like [1:21] TCPBR and quick followed by in cloud era [1:25] we move the performance into the [1:27] hardware for example PSP encryption and [1:30] then falcon reliable transport offloaded [1:33] to the nick hardware. [1:36] Now in 2026 AI workloads demand a [1:39] fundamental shift. [1:43] The right on the side we know that um [1:45] the AI model size has been in increasing [1:48] exponentially which also drives the [1:51] required network bandwidth per chip [1:53] increasing at the same same speed. It [1:56] also comes with very strict delay bound [2:00] and we know that the networking for this [2:02] kind of AR supercomput is spread across [2:04] multiple buildings and campuses. At this [2:07] scale we know that everything breaks and [2:10] we should make these failures invisible [2:12] to the AI workload. [2:18] So with that um here is what we cover [2:20] today. First I will start with some core [2:23] design princip principles that guided [2:25] our architectural decisions with some [2:28] details about VGO topology and how its [2:31] position with with the the rest of the [2:33] other network components in the AI [2:34] supercomputer [2:36] followed by some two new telemetry [2:38] solutions baked in Virgo. Then I will [2:41] conclude with next steps. [2:46] So let's first dive into some core [2:48] design principles. [2:50] First, the scale out network should be [2:52] designed as a part of the campus level [2:56] supercomputer optimized for the one use [2:58] case instead of being a very generic [3:01] infrastructure that serves many [3:02] different use cases. [3:05] That means that the hardware for the [3:09] scale out networks whether this is [3:11] switches and optics and fibers should [3:13] should be specked in lock step with the [3:16] AI supercomputer and AI hardware that it [3:19] supports. [3:20] Secondly, the scale out network itself [3:23] the life cipher life cycle of the scale [3:26] out network is highly coupled with AI [3:28] cluster. [3:30] So consequently the traditional [3:31] requirements like a backward [3:33] compatibility is not a strong [3:34] requirement anymore [3:37] and the resiliency should be baked in [3:40] the network design in a way such that a [3:42] training job should not crash for a [3:44] single link or switch failures. [3:47] Once the fabric is built for the [3:49] resiliency, we need high resolution [3:52] telemetry so that protocols like a [3:55] congestion control and then management [3:56] protocols can get the draw the maximum [4:00] performance and availability out of this [4:02] fabric. [4:05] Finally, we know that the worldwide [4:07] demand for the AI compute is super high. [4:10] The network design should support timely [4:12] delivery of this multi-billion dollar AI [4:14] cluster to the customers or end users. [4:18] In that sense, minimizing the number of [4:20] switching layers is critical because [4:23] fewer number of racks and fibers as we [4:25] heard um means that faster turn up. [4:30] With that, let's see how this translates [4:32] into our network architecture. [4:37] So our data center features three [4:39] distinct and specialized networking [4:42] layers that operate operate as a one [4:44] compute domain. [4:46] First orange at the bottom is our scale [4:50] up network connecting [4:53] accelerated chips with high bandwidth [4:55] and low latency in a pass scale. [4:59] Nvidia has a MV link for GPU scale up [5:02] and also we do have um optical switching [5:05] based ICI for TPU scale up. [5:09] Next is a Virgo as a scale out network [5:12] in the or yellow boxes. These are [5:14] dedicated back end providing the high [5:17] bandwidth bandwidth uh interconnect for [5:20] the multiple parts at the cluster level. [5:24] We will dive more into this um VGO [5:26] network in the next slide. [5:28] Finally, we have a Jupiter, our proven [5:31] front-end network in between. [5:33] It is easily there to provide a reliable [5:36] and fast access to the distributed [5:39] storage or generic computer resources [5:44] and it can be easily shared across uh [5:46] multiple AI clusters in the same [5:48] building as shown in the picture. [5:53] Then why do we separate the front end [5:55] and backend networks? [5:57] First, it enables us the independent [5:59] cadence, which means that we can evolve [6:02] each network domain independently, [6:04] ensuring the scale out fabric is matched [6:07] to the AI hardware it supports. [6:10] This is increasingly important at Google [6:13] because we have to support multiple [6:15] generations and flavors of GPU products [6:17] and also TPU products coming out every [6:20] year. [6:23] Secondly, we can also avoid systemwide [6:25] disruptions by upgrading updating them [6:28] separately. [6:30] Finally, the front and back end they [6:32] serve two different goals. Virgo [6:34] delivers 100% non-blocking capacity [6:37] while the front end Jupiter can be [6:39] optimized for TCO and incremental [6:42] expansion. [6:45] Then let's zoom in onto the Virgo [6:47] topology in the next slide. [6:50] There's a lot going on this slide. Um, [6:53] take a peek. I will walk you through [6:55] from number one to three. [6:58] First, we collapsed the many network [7:00] layers into flat two layer non-blocking [7:03] topology which is possible by maximally [7:06] leveraging the switch radics [7:09] and minimizing the number of layers also [7:12] helps with minimizing latency and jitter [7:15] as we know. [7:17] And secondly in connecting the [7:20] accelerator racks to the fabric we [7:22] employ the multiplaner design and in [7:25] that we also deploy independent control [7:28] domains per plane to isolate failures. [7:34] And finally the vergo spans not only in [7:36] the uh one building but also it it can [7:39] be expanded across multiple buildings in [7:42] a campus. [7:44] Then let let's look at how the [7:46] multiplane works which is relatively new [7:49] concept in the industry and we figure [7:51] that almost everyone is using this [7:53] architecture in the the latest scale out [7:55] network. [7:59] So in Virgo we use something called [8:01] skinny links as a starting point which [8:04] means that every link in the fabric [8:06] whether it's connecting to the [8:08] accelerator machine or in between [8:10] switches they run at the lowest possible [8:13] speed which is the surges lane speed for [8:16] example 200 gigs uh uh bit per second in [8:19] the latest technology [8:22] and this enables to us to maximize the [8:25] switcher radics enabling also two-stage [8:28] scaling Here [8:31] the number of planes is a function of [8:33] the number of skinny links available per [8:35] accelerator. [8:38] For example, if a GPU product is driving [8:42] 1.6 teraby per second bandwidth and with [8:45] a 200 gig service, this means that we [8:48] have eight skinny links per chip. [8:51] And this gives a pretty nice scaling [8:53] property. If the purchase bandwidth [8:55] increases for a given product and higher [8:59] capacity switches are not available for [9:02] some some reason then you can consider [9:04] actually adding more planes because the [9:07] number of planes is proportional to the [9:09] the bandwidth demand per chip. [9:13] It also helps with the graceful [9:15] degradation. Let's say if one of the end [9:18] planes is down you still have a n minus [9:20] one out of n bandwidth available. For [9:22] example, 87.5% [9:25] bandwidth still there if one of eight [9:27] planes is down for a given chip, which [9:30] is a big improvement compared to the [9:32] traditional design when a single link or [9:35] um switch failure cost 50% or 100% [9:39] bandis cut per chip. [9:42] And you may notice that this multiplane [9:44] design is different from rail alignment. [9:48] Here every chip regardless of the um [9:52] rail index it connects to all different [9:54] N network planes. In other words, if we [9:58] have a four GPUs per machine, that means [10:00] that we have four rail system and then [10:02] eight planes from the network fabric, [10:04] they are fully meshed together. [10:08] In that way, not only the intraail [10:10] communications but also crossra traffic [10:13] can enjoy the full bisection bandwidth [10:15] within two stages without needing [10:17] another third third layer of switching. [10:26] Next let's take a little bit about the [10:28] multipath routing which is also hot [10:30] topic. Um in VGO we support two pack [10:34] level load balancing schemes. First [10:37] adaptive routing known as a switches are [10:41] have smartness in terms of steer around [10:43] the local congestion and failures. It [10:46] also reacts to [10:49] remote and downstream failures in a very [10:51] fast manner. [10:54] Secondly, we also support SRV6 source [10:57] routing in VGO switches so that the [11:00] barometer cloud customers who can bring [11:02] their own host stack like MRC stack [11:05] can monitor and control end to end data [11:08] path of every packet. [11:12] This way VGO provides maximum throughput [11:15] and minimum tail latency. But we know [11:17] that still micro bursts or [11:20] failure induced congestions can still [11:22] happen in the fabric. So then that [11:25] brings us to telemetry. [11:31] So AI training is well known to create [11:34] massive and synchronized line rate burst [11:38] and that means that the micro congestion [11:40] can still happen but which is often [11:43] invisible to existing telemetry [11:45] solutions. [11:47] So we took telemetry as a first class [11:49] citizen in Virgo design and we [11:51] introduced two telemetry high high [11:54] resolution telemetry solutions. First [11:57] one is the streaming of high resolution [11:59] matrix from the switch CPU so that the [12:02] control and the manage management plane [12:04] can use of it of them for per [12:07] performance tuning, troubleshooting and [12:08] debugging. There are multiple different [12:11] metrics but on the left hand side we [12:13] show the port transmission utilization [12:16] as an example [12:19] and the same set of high res le [12:21] resolution metrics are also available in [12:23] the data plane for the end host to [12:26] consume for the for the visibility and [12:28] also uh transport protocols which is [12:32] happening through CIC as a inband [12:34] condition signaling mechanism that I [12:36] will detail in the next next slide. [12:40] So let's first look at the demo of the [12:42] high resolution streaming telemetry. [12:46] Okay, here is a telemetry dashboard with [12:49] workload view on the left hand side and [12:50] then switch level views on the right [12:52] hand side. It starts with the standard [12:55] monitoring view and at 30 second [12:58] resolution the network looks very [12:59] healthy. It it shows a massive bandwidth [13:02] headroom [13:04] but this view is misleading. They erase [13:07] the m micro burst. So let's take a look [13:09] at what's actually happening on the wire [13:13] by toggling to the sub millisecond view. [13:17] The health baseline is gone and the [13:20] green pillars are the 100% saturation [13:22] events. While the blue line still claims [13:25] that 10% load, the traffic bursts are [13:29] wet lining the switch buffer every few [13:31] seconds. As you can see here, [13:34] isn't it great to have such a fine grain [13:36] view and the wire level? So we are using [13:39] this kind of high resolution insights to [13:41] harden Google's infrastructure and also [13:44] daily operations. [13:47] Next we will see how these metrics made [13:49] available for data plane protocols for [13:51] the anhost tech to consume. [13:55] This become is becoming possible through [13:58] through something called cig. Hopefully [14:00] you heard some of you heard about this [14:03] is a inband congestion signaling which [14:06] carries the bottleneck point information [14:08] from the fabric switches in the data [14:10] packet. [14:12] Um if it sounds somewhat familiar uh to [14:16] you about uh embed network telemetry int [14:19] then you are right actually the cig is a [14:22] even more practical and efficient better [14:24] version of int because int used to carry [14:28] all the power hop data uh metadata in a [14:31] stacking manner but CC carries only the [14:33] bottleneck information [14:36] in a constant packet header format which [14:38] is shown in the diagram. [14:41] So in this example we use available [14:44] bandwidth relative to the port speed but [14:47] actually CC can support many different [14:49] metrics. For example the port hop [14:51] maximum delay [14:54] here the available bandwidth normal [14:56] normalized by capacity is actually the [14:58] inverse of the port TX utilization we [15:01] have shown in the previous uh slide. [15:05] So what's happening here is that as a [15:07] packet traverses from the sender to [15:09] receiver it goes through three different [15:11] hops and each hop device updates the CC [15:15] tag only when it sees its own value is [15:18] worse than the value carried by the [15:21] upstream switches. [15:23] In that way the receiver gets only the [15:26] bottling information. It also comes with [15:29] some hint about whether the congestion [15:31] happening due to the uh first hub unlink [15:34] multipaththing problem or the last hub [15:36] incast [15:39] and that information is reflected back [15:41] to the sender and in the act packet. [15:45] This is a quite exciting moment for me [15:47] because um INT was my kind of first P4 [15:50] application uh I was able to develop [15:53] when I was at barefoot and I'm happy to [15:56] say that the CIC even is a better [15:59] version of INT is fully supported in [16:02] Virgo in in end to end matter between [16:05] switches and nicks [16:07] and thanks to the community effort at [16:09] UEC C6 is going to be a part of the [16:12] standard UC standard and there There is [16:15] a quite detailed paper from my [16:17] colleagues uh at SECOM last week. So [16:20] please check it out if you want to learn [16:22] more about the details. [16:26] Okay, [16:28] to wrap up, Virgo is a new scaleout [16:32] fabric at Google is designed to deliver [16:35] performance at scale and reliability. [16:38] It is also empowered by high resolution [16:40] telemetry solutions available in the [16:42] management plane also in data plane. [16:46] It is currently enabling the new [16:49] generation of GPU and TPU products [16:51] including Nvidia Vera Rubin platforms [16:53] and then TPU at systems. [16:58] As next steps, we are working to scale [17:01] Virgo across metros hundreds of [17:04] thousands of kilometers apart. And as [17:06] were highlighted by Omar in his keynote, [17:10] delivering high performance RDMA over [17:13] the long haul connection is particularly [17:15] challenging and we believe that um the [17:18] congestion information carried by Case [17:20] and by it extended version of it can [17:23] actually help to address the challenges. [17:26] We are working together with the [17:27] partners at UC. So feel free to join the [17:30] community effort there. [17:33] And as we have our uh fabric scaling [17:36] across multiple distributed locations, [17:40] the chance for the uh fabric to actually [17:43] make use of multiple different vendor [17:45] switches is is in increasing. So that [17:49] means that protocol unification and [17:50] interrupt between different vendor [17:52] systems is also quite important and it [17:56] it is calling for another opportunity [17:58] for collaboration. [18:01] Um taking a step back it's very exciting [18:04] time to be in the networking field. Some [18:07] innovations we as a community has worked [18:09] on so far for the last decade like INT [18:12] or source routing in the data center. [18:16] some of them overlooked initially but [18:18] now they are finding real proving ground [18:21] in AI. So let's continue to work [18:24] together to make a better version of AI [18:26] scale across um or any other new use [18:29] cases. With that, thanks thanks again [18:32] for having me here. And here is a QR [18:35] code if you want to learn a little bit [18:36] more about Google's AI networking, [18:38] whether data center or when.