Google Deploys Virgo Network Fabric to Scale AI Supercomputers

@Scale

Google engineered its Virgo scale-out network fabric with multi-plane skinny links and fixed-overhead telemetry to connect up to one million AI chips across campus data centers without training job disruptions.

AI-assisted summary of Virgo: Scale-out Data Center Network for AI | Jeongkeun "JK" Lee, Google — original source. Check the original for full context.

Source published: 2026-09-13

Detailed notes

## Background This talk by Jeongkeun "JK" Lee from Google was published on YouTube by @Scale on August 27, 2026. The description notes that it introduces Virgo, a dedicated scale-out accelerator-to-accelerator network fabric engineered for deterministic latency, maximum resilience, and massive horizontal scale to power next-gen TPUs and NVIDIA Vera Rubin platforms.

## TL;DR Google engineer JK Lee details Virgo, a dedicated scale-out data center network fabric designed to meet the extreme multi-building demands of modern AI workloads. He outlines four core design principles—integrated supercomputer design, built-in resiliency, high-resolution telemetry, and deployment speed—that shaped Virgo's 2-layer flat architecture and multi-plane "skinny link" topology. The talk highlights how Consolidated In-Band Congestion Signaling (CEC) and sub-millisecond telemetry expose invisible micro-bursts to maintain high goodput. Looking ahead, Lee invites industry collaboration through the Ultra Ethernet Consortium to standardize adaptive routing and scale RDMA across multi-metro long-haul campuses.

## Atmosphere & style The presentation carries an authentic, enthusiastic, and highly technical tone that balances deep architectural rigor with nostalgic industry reflections. Lee brings a grounded, engineer-to-engineer presence to the stage—symbolized by his classic 2018 conference t-shirt—making complex network concepts accessible through vivid side-by-side dashboard comparisons. Watching the talk feels like an insider look into hyper-scale infrastructure design, driven by a genuine passion for how once-niche telemetry innovations are now solving critical real-world AI bottlenecks.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=5s) Google's networking evolution spans four main inflection points: the Internet Era (Jupiter, Orion, Apollo), the Streaming Era (QUIC, TCP BBR), the Cloud Era (Andromeda, PSP, Falcon), and the modern AI Era. - [00:41](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=41s) AI network design requires treating the entire building or campus as an integrated supercomputer co-designed tightly with specific accelerator generations. - [00:41](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=41s) The core design mandate for modern AI networks is to build native resiliency so underlying infrastructure failures remain completely invisible to AI training workloads. - [06:13](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=373s) Google's AI data center network is split into three distinct layers forming a flat compute domain: the Scale-in domain (ICI for intra-pod), the Virgo Scale-out fabric (inter-pod), and the Jupiter front-end fabric (storage and WAN). - [08:36](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=516s) Virgo uses high-radix switches to create a flat, 2-layer non-blocking topology, reducing switch hops and packet latency. - [08:36](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=516s) Virgo implements multi-plane isolation using 'skinny links' at lane speeds down to 30 Gbps, ensuring an 8-plane architecture retains 87.5% bandwidth even if an entire plane fails. - [08:36](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=516s) The multi-plane skinny-link architecture enables non-blocking Virgo clusters to scale up to 1,000,000 TPU/GPU accelerator chips in a single logical domain across campus sites. - [13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) Consolidated In-Band Congestion Signaling (CEC) aggregates telemetry metrics across hops (using MIN/MAX functions) to maintain a constant header overhead while pinpointing exact bottlenecks. - [13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) CEC is being standardized through the Ultra Ethernet Consortium (UEC) and featured at SIGCOMM 2026 based on Google's production deployment experience. - [15:47](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=947s) Sub-millisecond telemetry monitoring is essential because traditional 30-second average metrics hide severe 100% link utilization micro-bursts caused by synchronized AI workloads. - [17:39](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=1059s) Future development focuses on expanding Virgo to multi-metro fabrics over hundreds or thousands of kilometers, adapting CEC for long-haul, high-RTT RDMA performance. - [17:39](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=1059s) Google is calling on the industry to establish multi-vendor standardization around adaptive routing, fast route-change notifications, and in-band telemetry protocols.

## Detailed walkthrough Here is a detailed, faithful walkthrough of the transcript, capturing the technical specifics, architectural principles, data points, and overall narrative texture.

---

[00:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=5s) Introduction and Historical Context: A Decade of Networking Inflection Points

Jeongkeun "JK" Lee introduces himself, representing his engineering team at Google. Reflecting on his career trajectory, he recalls starting as a researcher at HP Labs before joining Barefoot Networks—widely known for the P4 programming language and Tofino switch ASICs. He proudly notes that he is wearing a t-shirt from the 2018 Networking @Scale conference, expressing deep gratitude for the open industry perspectives shared at these events over the years.

Moving to the present era of 2026, Lee maps out Google's networking journey through four fundamental inflection points leading up to today's massive AI demands:

1. **The Internet Era:** Built to power global services like Search, Maps, and Gmail. Google had to re-architect data center networks from the ground up, giving birth to the Jupiter fabric, paired with Orion for SDN control, and pioneering Apollo optical circuit switching (OCS) for flexible, intra-building expansion. 2. **The Streaming Era:** Driven by the need to deliver high-quality video with minimal jitter to millions of end devices. This motivated new transport and congestion protocols like QUIC and TCP BBR. 3. **The Cloud Era:** Demanded extreme multi-tenancy, high reliability, and strict security guarantees. Google engineered Andromeda, gRPC for microservices, and moved network functions directly into hardware—such as PSP for inline hardware encryption and Falcon for reliable transport. 4. **The AI Era:** Characterized by exponential growth in foundational model parameter sizes. This makes AI training exceptionally network-intensive, requiring exponentially higher network bandwidth alongside extremely strict delay bounds.

Because AI supercomputers now span across multiple data center buildings and whole campuses, traditional scale constraints cause "basically everything to break." The core mandate for modern network design is to make these inevitable underlying infrastructure failures completely invisible to AI workloads.

- **Speaker background:** JK Lee (Google; formerly HP Labs, Barefoot Networks). - **Conference artifact:** Wearing a classic 2018 @Scale event t-shirt. - **Key Google historical milestones:** Jupiter (fabric), Orion (SDN), Apollo (OCS), QUIC, TCP BBR, Andromeda, gRPC, PSP (encryption), Falcon (transport).

---

[00:41](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=41s) Four Core Principles for AI Network Design

To tackle the extreme conditions of AI supercomputing, Lee presents four foundational principles that guided the engineering and execution of Virgo—Google's new scale-out data center network:

1. **Integrated Supercomputer Design:** The network must be engineered as part of the entire building or campus functioning as a single, unified supercomputer optimized specifically for one or two AI use cases, rather than acting as generic, multi-purpose infrastructure. Hardware components (switches, optics, fiber) are co-designed and specced in lockstep with each accelerator generation (GPUs/TPUs). Consequently, traditional requirements like backward/forward compatibility or flexible expansion (which Jupiter and Apollo solved) are no longer primary constraints. 2. **Bake-in Resiliency:** Training jobs must not crash or restart due to a single link or switch failure. Resilience must be built natively into the physical topology and control loops. 3. **High-Resolution, High-Quality Telemetry:** Hardware resiliency alone is insufficient. Transport protocols, congestion control, load balancing in the data plane, and management-plane debugging all rely heavily on timely, granular telemetry to extract maximum performance and availability from the physical network. 4. **Execution and Deployment Speed:** Networking must never be the bottleneck delaying the deployment of multi-billion-dollar AI supercomputers. This requires minimizing switching layers (which reduces latency while cutting raw material requirements) and designing fiber/rack layouts that align directly with real-world building construction phases. Modeling physical dimensions and physical space constraints in software beforehand prevents costly first-time deployment issues on-site.

- **Design Philosophy:** Dedicated AI fabric > general-purpose infrastructure. - **Cycle Alignment:** Network hardware lifecycle is tightly bound to accelerator (GPU/TPU) generations. - **Deployment Efficiency:** Fewer switching layers = lower latency + fewer materials to physically deploy.

---

[06:13](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=373s) Virgo Architecture: The Three-Layer Flat Compute Domain

Google shifted away from traditional general-purpose network structures toward a flat, specialized, low-latency architecture. Under this model, the data center operates as three distinct, specialized network layers forming one unified compute domain:

``` +-----------------------------------------------------------------+ | JUPITER FABRIC | | (Front-End: Storage, General Compute, Global WAN & Multi-Site) | +-------------------------------+---------------------------------+ | +-------------------------+-------------------------+ | | +-----v---------------------------+ +---------v-----------------------+ | VIRGO SCALE-OUT FABRIC | | VIRGO SCALE-OUT FABRIC | | (Inter-Pod Accelerator Network)| | (Inter-Pod Accelerator Network) | +-----------------+---------------+ +-----------------+---------------+ | | +-----------------v---------------+ +-----------------v---------------+ | SCALE-IN DOMAIN (ICI) | | SCALE-IN DOMAIN (ICI) | | (Intra-Pod TPU/GPU Circuit) | | (Intra-Pod TPU/GPU Circuit) | +---------------------------------+ +---------------------------------+ ```

1. **Scale-in Domain (Orange):** High-bandwidth, ultra-low-latency fabric engineered for tightly coupled intra-pod accelerator communication (e.g., optical switching-based Inter-Chip Interconnects / ICI used in TPU pods). 2. **Scale-out Accelerator Fabric (Yellow - Virgo):** A dedicated fabric optimized for massive horizontal scaling across accelerator pods. 3. **Front-End Fabric (Jupiter):** The proven front-end network providing fast, reliable access to distributed storage, general compute resources, and the global WAN. Jupiter ensures data access doesn't bottleneck training or serving, and allows multiple AI clusters within the same building to share front-end capacity efficiently.

This architectural decoupling between front-end and back-end offers three strategic benefits: independent evolution/upgrades of each domain without systemwide disruption; maximum non-blocking bisectional bandwidth for training while optimizing front-end TCO; and tight lockstep co-design with new ML accelerator hardware.

- **Scale-in (ICI):** Intra-pod, ultra-low latency. - **Scale-out (Virgo):** Inter-pod, massive horizontal scale across pods. - **Front-end (Jupiter):** Storage, management, WAN access, cross-site cluster connection.

---

[08:36](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=516s) Multi-Plane Design and Skinny Links

Zooming into the Virgo architecture details, Lee highlights how the network achieves extreme scale without adding extra topology layers:

- **Flat 2-Layer Topology:** By using high-radix switches, Virgo collapses multi-tier trees into a flat, non-blocking 2-layer topology. This drastically reduces packet transit latency by cutting the number of required switch hops. - **Skinny Links & Multi-Plane Isolation:** To maximize switch radix, logical links are configured using "skinny links" at lane speeds as low as 30 Gbps (SerDes). The number of available planes ($N$) is a function of the skinny links per accelerator. For example, a Network Interface Card (NIC) with 1.6 Tbps bandwidth using 200G/30G lane speeds provides 8 skinny links per chip, creating an 8-plane architecture. - **Graceful Degradation:** If an entire network plane fails, $N-1$ planes remain fully operational. In an 8-plane design, a single plane outage leaves 87.5% of total bandwidth available to every chip—a massive improvement over legacy architectures where a single link/switch failure could drop per-chip bandwidth by 50% to 100%. - **Scale Limits:** Multi-plane skinny-link designs allow non-blocking domains to scale $N \times$ larger without needing a 3rd switching tier. Virgo can scale up to **1,000,000 TPU/GPU chips** (specifically TPU v6e/v8-class targets) within a single logical cluster across campus sites. - **Multipath Load Balancing:** - *Switch-driven Adaptive Routing:* Extremely fast local reaction to switch failures/congestion, plus rapid downstream failure notifications. - *SRv6 Source Routing:* Supported for bare-metal customers bringing custom host networking stacks to distribute traffic evenly at packet-level granularity across all available fabric paths.

- **Bandwidth Resiliency:** 8-plane design guarantees 87.5% capacity retention during a full plane failure. - **Max Cluster Scale:** Up to 1 million accelerator chips in a single logical cluster. - **Routing Mechanisms:** Adaptive routing (hardware-level) + SRv6 (host-level packet distribution).

---

[13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) High-Resolution Telemetry and Consolidated In-Band Congestion Signaling (CEC)

AI workloads are notorious for bulk synchronization phases, creating extreme burstiness. Because congestion and failures cause costly training interruptions, Virgo elevates telemetry to a "first-class citizen" operating in two modes:

1. **High-Resolution Streaming Telemetry:** Live switch port bandwidth utilization and metric streaming sampled at sub-millisecond intervals. 2. **Consolidated In-Band Congestion Signaling (CEC):** High-resolution congestion metrics embedded directly inside data plane packets.

Lee explains that CEC is an advanced evolution of In-Band Network Telemetry (INT)—a technology he personally worked on during his time at Barefoot Networks. Unlike standard INT, which appends per-hop metadata to the packet header (causing header bloat that can exceed the payload size over deep paths), CEC aggregates metrics across hops using `MIN` or `MAX` mathematical aggregation functions.

As a result, CEC maintains a **constant header overhead** regardless of how many hops a packet traverses, while still providing exact bottleneck location data. Supported end-to-end across Virgo switch ASICs and NICs, CEC is currently being standardized within the **Ultra Ethernet Consortium (UEC)**, with an upcoming **SIGCOMM 2026** paper detailing its design and production deployment history at Google.

- **Metric Aggregation:** Uses `MIN`/`MAX` functions to prevent packet header inflation. - **Header Overhead:** Fixed/constant size regardless of hop count. - **Standardization & Research:** Adopted by UEC; detailed in a SIGCOMM 2026 paper.

---

[15:47](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=947s) Telemetry Demo: Unmasking Invisible Micro-Bursts

Lee shares a side-by-side dashboard demo illustrating why standard monitoring tools fail in AI environments:

- **Standard View (30-second averages):** The monitoring dashboard shows a healthy network with wide headroom, claiming an average port utilization of roughly 10%. - **Sub-Millisecond View:** Toggling the telemetry view to sub-millisecond resolution reveals massive green saturation spikes hitting **100% link capacity** every few seconds. - **The Finding:** 30-second—and even 100-millisecond application-level—averages completely erase micro-bursts, creating "invisible congestion" during synchronized AI workload phases. Google utilizes this sub-millisecond visibility to harden fabric control loops and tune next-generation switch hardware.

``` Standard 30s View: [--- 10% Avg Load ---] -> (Looks healthy, hides problems) Sub-millisecond View: [||| 100% SPIKE |||] -> (Redlining switches continuously) ```

- **Demo Takeaway:** Standard telemetry masks micro-bursts that destabilize collective communication routines during model training.

---

[17:39](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=1059s) Future Roadmap and Industry Call to Action

Looking ahead, Lee outlines the next development frontiers for Virgo and leaves the community with key takeaways:

1. **Multi-Metro Fabric Scaling:** Expanding Virgo fabrics across data center campuses separated by hundreds or thousands of kilometers. Running high-performance RDMA over long-haul connections introduces major challenges due to multi-millisecond round-trip times (RTT). Google is collaborating with industry partners to adapt CEC for sub-RTT congestion and failure signaling. 2. **Multi-Vendor Interoperability:** A direct call to action for the networking community to standardize telemetry protocols and fast route-change notifications for adaptive routing across diverse hardware vendors.

In closing, Lee expresses how thrilling it is to be a network engineer today: technologies once considered niche or overlooked—such as In-Band Network Telemetry, CEC, adaptive routing, and data center source routing—are now being deployed globally at breakneck speed to power the AI revolution.

He concludes by displaying a QR code linking to a technical blog post co-authored with an Google Fellow and invites attendees to connect via email for future collaboration.

- **Target Expansion:** Cross-campus / multi-metro long-haul fabrics (100s to 1000s of km). - **Core Challenge:** Maintaining RDMA performance over millisecond-level RTTs. - **Industry Call:** Multi-vendor standardization around UEC, CEC, and adaptive routing protocols.

## Notable quotes > [00:41](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=41s) basically everything to break. — Jeongkeun "JK" Lee

> [08:36](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=516s) Skinny Links & Multi-Plane Isolation — Jeongkeun "JK" Lee

> [13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) MIN — Jeongkeun "JK" Lee

> [13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) MAX — Jeongkeun "JK" Lee

> [15:47](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=947s) Standard 30s View: [--- 10% Avg Load ---] -> (Looks healthy, hides problems) Sub-millisecond View: [||| 100% SPIKE |||] -> (Redlining switches continuously) — Jeongkeun "JK" Lee

## Mentioned Jeongkeun "JK" Lee, Google, HP Labs, Barefoot Networks, P4, Tofino, @Scale, Jupiter, Orion, Apollo, QUIC, TCP BBR, Andromeda, gRPC, PSP, Falcon, Virgo, A5x, NVIDIA Vera Rubin NVL72, TPU, TPU v6e, TPU v8, SRv6, CEC, In-Band Network Telemetry, INT, Ultra Ethernet Consortium, UEC, SIGCOMM 2026, RDMA

[Source](https://www.youtube.com/watch?v=00Gcgk9FJf4)

中文简报

谷歌部署 Virgo 网络架构扩展 AI 超算规模谷歌设计了采用多平面细链路（skinny links）与固定开销遥测技术的 Virgo 横向扩展网络架构，可在跨园区数据中心连接多达 100 万颗 AI 芯片，且不会中断训练任务。谷歌工程师 JK Lee 详细介绍了专为满足现代 AI 工作负载严苛跨楼宇需求而设计的横向扩展数据中心网络架构 Virgo。他概述了塑造 Virgo 两层扁平架构和多平面“细链路”（skinny link）拓扑的四大核心设计原则：一体化超级计算机设计、原生弹性、高分辨率遥测以及部署速度。演讲重点阐述了统一带内拥塞信令（CEC）和亚毫秒级遥测如何暴露隐蔽的微突发流量，从而保持高有效吞吐量（goodput）。展望未来，Lee 邀请业界通过超以太网联盟（Ultra Ethernet Consortium）展开合作，推进自适应路由的标准化，并实现跨多城域长途园区的 RDMA 规模扩展。 Google 提出了面向 AI 时代数据中心网络设计的四项核心原则：将整个数据中心或园区视为一体化超级计算机进行软硬件协同设计、内建原生故障弹性、引入高分辨率遥测，以及加速实体部署与交付效率。 (0:41) Google 将 AI 数据中心网络划分为三层扁平计算域：负责 Pod 内部紧耦合通信的 Scale-in 域（如 ICI）、专用于 Pod 间横向扩展的 Virgo Scale-out 矩阵，以及承载分布式存储与广域网接入的 Jupiter 前端矩阵。 (6:13) Virgo 采用高基数交换机构建扁平的两层无阻塞拓扑，并通过细链路（skinny links）实现多平面隔离；其 8 平面架构在遭遇整平面故障时仍可维持 87.5% 的芯片可用带宽，单逻辑集群最高可横向扩展至 100 万个 TPU/GPU 加速器芯片。 (8:36) 针对 AI 突发同步导致的不可见微突发流（micro-bursts），Virgo 引入了整合带内拥塞信令（CEC），利用 MIN/MAX 函数跨跳聚合指标以保持恒定报头开销，目前该方案正推进由超以太网联盟（UEC）进行标准化。 (13:05)

Reading summary

Google engineered Virgo as a specialized scale-out network fabric designed to prevent underlying infrastructure failures from disrupting large-scale AI supercomputing workloads. According to Google engineer Jeongkeun Lee, traditional multi-purpose data center fabrics break under modern AI scaling demands, necessitating dedicated architectures co-designed in lockstep with machine learning accelerators. Specialized Three-Layer Architecture for AI Workloads (6:13) Lee outlines a transition away from generic, backward-compatible networking toward fabrics tailored specifically to AI supercomputers. Under this design, Google decouples data center networking into three specialized tiers: a low-latency scale-in domain using optical circuit switching and Inter-Chip Interconnects (ICI) for intra-pod communication; the Virgo scale-out fabric for inter-pod accelerator networking; and the established Jupiter front-end fabric for distributed storage, general compute, and global WAN connectivity. This separation allows the accelerator networks to co-evolve directly with TPU and GPU hardware cycles without disrupting front-end infrastructure. By prioritizing workload-specific optimization over general-purpose flexibility, the architecture minimizes switching layers, lowers transit latency, and streamlines physical data center deployment. High-Radix Two-Layer Topology and Multi-Plane Scaling (8:36) To support clusters scaling up to 1,000,000 accelerator chips—specifically targeting TPU v6e and v8 classes—Virgo utilizes high-radix switches to collapse conventional multi-tier network trees into a flat, non-blocking two-layer topology. Bandwidth from each network interface card is split into "skinny links" operating at lane speeds down to 30 Gbps SerDes, dividing the network into multiple isolated parallel planes. In an eight-plane implementation using 1.6 Tbps NICs, the total failure of any single plane leaves 87.5% of fabric bandwidth available to every chip, preventing link or switch outages from halting training jobs. For traffic distribution across these paths, Virgo integrates hardware-level adaptive routing for fast downstream failure reactions alongside host-driven SRv6 source routing for packet-level balancing. Telemetry and Consolidated In-Band Congestion Signaling (13:05) AI collective communication produces severe, synchronized burstiness that traditional network monitoring cannot detect. A production telemetry comparison presented by Lee shows that 30-second metric averages displaying roughly 10% port utilization conceal recurring sub-millisecond micro-bursts that hit 100% link saturation, leading to invisible congestion and job delays. To manage these dynamics, Virgo pairs sub-millisecond streaming telemetry with Consolidated In-Band Congestion Signaling (CEC), an evolution of In-Band Network Telemetry. Instead of appending hop-by-hop metadata that causes packet header inflation, CEC applies mathematical functions such as minimum and maximum aggregations across switches. This ensures a fixed header overhead across any path depth while still identifying exact network bottlenecks. Google has submitted CEC to the Ultra Ethernet Consortium for open standardization, supported by a SIGCOMM 2026 paper. Google's planned expansion for Virgo involves connecting data center campuses across multi-metro distances of hundreds to thousands of kilometers. Lee notes that sustaining high-performance RDMA over multi-millisecond round-trip times remains an open challenge, requiring sub-RTT congestion signaling and broader industry collaboration around open telemetry and adaptive routing standards.

精炼阅读摘要

Google网络工程主管Jeongkeun "JK" Lee介绍了专为超大规模AI算力集群设计的横向扩展数据中心网络Virgo。其核心论点在于：传统通用数据中心网络无法承受基础大模型对带宽与延迟的极端需求，必须将整座建筑甚至园区视为单一AI超算进行端到端协同设计，通过网络层级解耦、双层扁平多平面拓扑以及亚毫秒级带内聚合遥测，使底层硬件故障对上层AI训练完全透明。 AI网络演进背景与四项核心工程原则 (0:41) Lee回顾了Google网络演进的四个关键阶段：支撑搜索与地图等全球服务的互联网时代（催生了Jupiter织网、Orion SDN与Apollo光线路交换）、推动QUIC和TCP BBR的流媒体时代、以Andromeda虚拟化和硬件卸载（如PSP加密与Falcon可靠传输协议）为标志的云时代，以及当下模型参数爆发增长的AI时代。随着AI超算跨越园区建筑边界，底层网络面临规模断裂风险。为此，Google确立了四项核心工程原则：第一，一体化超算设计，将网络作为与GPU/TPU硬件世代深度绑定的专用织网，放弃对传统向后兼容与通用扩展的严苛限制；第二，物理拓扑与控制环路内置高弹性，杜绝单点链路故障导致训练崩溃或重启；第三，全栈部署高精度遥测，为传输协议与拥塞控制提供亚毫秒级支撑；第四，优先保障部署效率，通过压缩交换层级减少用料与物理延迟，并依托软件建模提前规避园区施工冲突。三层解耦域与Virgo横向扩展织网 (6:13) 针对不同计算任务的异构通信模式，Google将数据中心网络重构为扁平化的三层计算网络域。第一层是基于光交换芯片间互连（ICI）的纵向扩展域（Scale-in Domain），专门为TPU/GPU Pod内部提供极高带宽与极低延迟的紧耦合加速器通信。第二层是横向扩展加速器织网（Scale-out Fabric）Virgo，专门负责跨Pod加速器之间的大规模横向互联。第三层则是前端通用织网（Jupiter），负责承载分布式存储、通用计算及跨站点WAN连接，确保数据加载不会成为训练与推理的瓶颈。这种前后端分离的解耦架构带来了三项关键优势：各层网络能够完全独立升级演进，避免全局干扰；在最大化AI训练所需无阻塞对分带宽的同时有效控制前端TCO；使网络硬件迭代能够与新一代机器学习加速器的生命周期保持严格对齐。双层多平面拓扑与细链路容错设计 (8:36) 在物理拓扑方面，Virgo利用高基数（High-Radix）交换芯片将传统多级胖树压缩为两层扁平的无阻塞拓扑，大幅降低了跨节点转发跳数与传输延迟。为了在有限基数下实现超大规模扩展，Virgo采用了"细链路"（Skinny Links）与多平面隔离机制，将物理SerDes速率配置低至30 Gbps。例如，单张1.6 Tbps带宽的网卡可划分为8条细链路，分别接入8个物理独立的网络平面。多平面设计提供了平滑的降级能力：当一个平面发生整体故障时，剩余7个平面仍能正常转发，所有芯片仍保有87.5%的可用带宽，彻底规避了传统架构中单点故障导致50%甚至100%带宽丢失的瘫痪风险。Lee指出，该设计支持单个逻辑集群在两层架构内横向扩展至100万颗TPU/GPU芯片（适配TPU v6e及v8等代际）。在路由调度上，Virgo结合了交换芯片硬件级自适应路由（用于毫秒级拥塞绕障与故障通告）与基于SRv6的主机源路由，满足裸金属租户精细化报文级分流需求。亚毫秒级遥测与带内拥塞聚合信号CEC (13:05) AI模型训练特有的集合通信模式具有极强的同步突发性。Lee通过现场监控对比演示表明，传统的30秒甚至100毫秒采样粒度会将瞬时拥塞平滑为约10%平均利用率的健康假象；而开启亚毫秒级流式遥测后，仪表盘暴露出了每隔数秒即冲击100%线速的微突发拥塞。这种隐性拥塞正是导致集合通信停滞与训练抖动的根源。为实时感知拥塞且不牺牲有效载荷，Google开发了带内拥塞聚合信号（CEC）技术。针对传统带内遥测（INT）在多跳路径中逐跳添加元数据导致报头膨胀的缺陷，CEC在交换芯片转发时使用MIN或MAX等数学函数聚合路径瓶颈数据，确保报头开销在任意跳数下始终恒定。目前CEC已在Virgo交换芯片与网卡中全量落地，被超以太网联盟（UEC）纳入标准化推进体系，其系统设计与生产部署经验将在SIGCOMM 2026会议正式发表。 Lee总结指出，Virgo未来的核心研发方向是跨园区及跨城际织网（Multi-Metro Fabrics）扩展，即在相距数百至数千公里的数据中心之间运行RDMA，这面临数毫秒往返时延（RTT）下的传输拥塞控制与亚毫秒快速自愈挑战。同时，Google呼吁行业针对遥测规范、CEC协议以及多厂商环境下的硬件自适应路由通告机制展开深度标准化合作。

Detailed notes — zh

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布，记录了 Google 网络专家 JK Lee 在 Networking @Scale 大会上的演讲。视频简介指出，Google 在沿用久经考验的 Jupiter 网络处理南北向流量的同时，正式推出了专为大规模 AI 横向扩展（Scale-out）东西向通信打造的全新 Fabric 架构 Virgo；该网络专为确定性延迟与极致韧性而设计，将为运行 NVIDIA Vera Rubin NVL72 平台的 Google Cloud A5x 裸金属实例及下一代 TPU 提供核心互联底座。 ## TL;DR 演讲系统拆解了专为万亿参数 AI 超级计算机打造的 scale-out 专用网络架构 Virgo。为了应对大模型训练对吞吐和容灾的苛刻要求，Google 打破传统网络设计，将体系解耦为纵向扩展（Scale-up）、横向扩展（Virgo）与前端存储/计算（Jupiter）三层架构，并通过“细链路”（Skinny Links）构建扁平二层、多平面隔离的无阻塞拓扑，支持高达百万芯片互联与单平面故障 87.5% 带宽保全。此外，系统原生支持自适应路由与 SRv6 源路由，深度集成了被超以太网联盟（UEC）采纳的恒定开销带内拥塞信标技术（CSIG），并通过实机 Demo 揭示了常规采样严重抹杀微突发的行业痛点，最后展望了跨城域 RDMA 扩展与多厂商生态标准化。 ## Atmosphere & style 整场演讲洋溢着硬核工程美学与深厚的技术怀旧情怀。JK Lee 身着 2018 年大会旧 T 恤开场，从早年学术界与 Barefoot Networks 探索 INT 的岁月侃侃而谈，语调从容自如且富有感染力。当他在实机仪表盘演示中一键切换监控精度、瞬间揭开“平稳假象”下惊心动魄的微突发拥塞时，现场氛围达到高潮；结尾处他由衷赞叹许多曾被视为超前的硬核网络创新终于在 AI 爆发期迎来用武之地，令观众深刻感受到身处“计算机网络黄金时代”的技术自豪感与激荡感。 ## Bulletin - [00:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=5s) Google 正式发布专为 AI 横向扩展（Scale-out）打造的全新数据中心网络架构 Virgo，打破传统通用架构，深度绑定下一代 TPU 及 NVIDIA Vera Rubin NVL72 平台。 - [10:13](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=613s) Virgo 引入基于 SerDes 通道的“细链路（Skinny Links）”与多平面隔离技术，当单个网络平面发生故障时，节点仍能保有高达 87.5% 的可用通信带宽。 - [13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) Google 提出了演进版带内遥测技术 CSIG（Congestion Signaling），无论跨越多少跳网络均保持恒定报头开销，该标准已被超以太网联盟（UEC）接纳。 - [15:47](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=947s) 现场演示表明，传统的 30 秒平均监控视图会严重掩盖 AI 流量的微突发；在亚毫秒级高精度遥测下，端口实际呈现出周期性 100% 满载爆表的隐形拥塞。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=5s) Google 历经互联网、流媒体和云三大网络时代的技术积累后，正式推出面向 AI 超级计算机的横向扩展专用网络 Fabric——Virgo。 - [02:23](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=143s) AI 超级计算机面临前所未有的指数级互联带宽需求与严苛延迟约束，底层设计原则转为将频繁的硬件失效对上层训练负载彻底透明化。 - [03:41](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=221s) Virgo 确立了四大核心设计原则：与加速器同频共振的专精设计、硬件失效不中断训练的内生韧性、毫秒级遥测闭环以及极简扁平层级的极致交付速度。 - [06:09](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=369s) Google 构筑了三层解耦的“统一计算域”：负责单机/单 Pod 芯片直连的 Scale-up 域、负责跨 Pod 通信的 Scale-out 域（Virgo）以及负责存储与广域分发的前端网络层（Jupiter）。 - [06:09](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=369s) 前端存储网络与后端加速网络物理解耦，兼顾了后端全线速无阻塞的高吞吐需求与前端网络的总体拥有成本（TCO）及弹性演进。 - [08:36](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=516s) Virgo 采用高基数交换芯片构建出极简的“二层无阻塞扁平拓扑”，将端到端转发延迟和抖动压低至物理极限，并跨越园区支持高达 100 万颗芯片协同训练。 - [08:36](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=516s) 多平面拓扑在每个物理平面部署完全独立的控制域，彻底遏制单平面的软硬件故障，避免故障向全局级联扩散。 - [10:13](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=613s) “细链路”技术将端口粒度精细化至 SerDes 通道级别，在无需引入第三层交换网络的情况下，使无阻塞集群组网规模呈线性倍数放大。 - [11:53](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=713s) Virgo 原生支持交换机驱动的微秒级自适应路由与面向裸金属多租户的 SRv6 源路由两类数据包级负载均衡机制，有效规避远端链路故障。 - [13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) 带内拥塞信标技术 CSIG 克服了传统 INT 报头随跳数线性膨胀的缺陷，采用跨跳聚合维持固定报头尺寸，并完整保留拥塞定位器字段。 - [15:47](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=947s) 常规的 30 秒或 100 毫秒监控粒度会抹杀周期性冲顶的微突发；唯有依赖亚毫秒级高精度遥测，才能暴露 AI 工作负载触发的 100% 满载隐形拥塞。 - [17:39](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=1059s) Virgo 未来的核心演进方向包括应对相距数千公里、毫秒级时延广域长途链路的跨多城域（Multi-Metro）RDMA 扩展，以及推进多厂商生态标准化。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=5s) 从 HP Labs 到 AI 时代：数据中心网络创新的演进历程来自 Google 的 JK Lee（Jeongkeun Lee）带着轻松而怀旧的基调开启了演讲。他回顾了自己早期在惠普实验室（HP Labs）从事科研的经历，坦言非常怀念当年学术共同体开放合作的氛围；随后他加入因 P4 编程语言和 Tofino 芯片闻名的 Barefoot Networks，并在 2018 年首次受邀参加 Networking @Scale 大会。他特意指了指自己身上穿的活动 T 恤，笑着说道：“顺便提一下，这件 T 恤是 2018 年的”，并表达了对该会议始终能够及时且公开地分享行业独特洞见的赞赏。如今步入 2026 年，身处轰轰烈烈的 AI 时代浪潮之巅，他感到非常荣幸能代表 Google 团队正式介绍专为 AI 打造的全新横向扩展（Scale-out）数据中心网络——Virgo。在切入 Virgo 之前，JK Lee 带领大家快速回顾了进入 AI 时代前十余年间 Google 经历的三次重大网络技术转折： - **互联网时代（Internet Era）**：为了支撑 Google 搜索、地图和 Gmail 等全球级服务，Google 从底层彻底重构了数据中心网络，推出了著名的 Jupiter 交换架构，并配合 Orion SDN 控制平面开创了软硬件解耦的先河；同时率先研发了 Apollo 光路交换（OCS）技术，实现了灵活扩容。 - **流媒体时代（Streaming Era）**：面对向海量终端低抖动、高质量分发视频的挑战，Google 主导开发了 QUIC 传输协议以及 TCP BBR 拥塞控制算法。 - **云时代（Cloud Era）**：面对极致的多租户隔离、高可用和安全性要求，团队打造了虚拟化网络系统 Andromeda 与广为人知的微服务框架 gRPC，并逐步将关键性能下沉至自研硬件加速，例如用于线速加密的 PSP 和实现可靠低时延传输的 Falcon 协议。 ### [02:23](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=143s) AI 时代的网络瓶颈与本次分享的核心议程在梳理完前三次技术革命后，JK Lee 转向了眼下 AI 时代前所未有的严峻挑战。从大模型参数规模的走势图可以清晰看到，基础模型的参数量正呈指数级爆炸，这使得 AI 分布式训练呈现出极其重度的网络吞吐敏感特性。与此同时，AI 集群所需的互联带宽同样呈现指数攀升，并且伴随着极为严苛的端到端延迟上限要求。如今的 AI 超级计算机早已不再局限于单栋机房，而是跨越了多个数据中心建筑乃至整个园区。在如此庞大的物理规模下，“基本上任何东西都会发生故障”，因此系统设计的首要目标，就是让这些底层硬件的频繁失效对上层的 AI 训练负载彻底透明、不可见。为了系统性拆解 AI 时代的网络挑战，JK Lee 在演讲开篇划定了四个分享阶段： 1. **设计原则**：驱动 Virgo 研发与落地的底层设计哲学。 2. **架构剖析**：Virgo 的网络拓扑细节及其在整个 AI 超算体系中的定位。 3. **遥测与演示**：高精度遥测技术如何保障系统性能与可用性，并辅以一段现场实机 Demo。 4. **未来展望**：技术演进的后续步骤。 ### [03:41](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=221s) 专为 AI 超级计算机打造：Virgo 的四大设计原则针对 AI 工作负载独特的物理与通信规律，Google 为 Virgo 确立了四项指导原则。首先，Virgo 必须被视作整栋建筑乃至整个园区规模超级计算机的有机组成部分，专门针对一到两种核心 AI 场景进行极限优化，而非作为兼容万物的通用基础设施。这也意味着，包括交换机芯片、光模块与光纤布线在内的所有网络硬件，都必须与每一代新型机器学习加速器（如新一代 TPU 与 GPU）“保持同步节拍（Lock-step）联合设计与定制”。过去 Jupiter 和 Apollo 所强调的前向/后向兼容性或平滑渐进扩容，在此不再是核心诉求，网络的生命周期已经与加速器硬件深度绑定。第二项原则是内生韧性（Resiliency）。在数万卡规模的训练集群中，网络不能因为单条光纤损坏或单个交换机宕机而导致整个大型训练任务崩溃中断。第三项原则是超高精度与高质量遥测。当高弹性的物理拓扑搭建完毕后，无论是在数据平面执行毫秒级拥塞控制与动态负载均衡，还是在管理平面进行瓶颈调优和故障根因排查，都需要高度及时、细粒度的遥测数据支撑，以此压榨出底层硬件的极限性能与有效算力输出（Goodput）。第四项原则是追求极致的部署速度。在日新月异的 AI 竞赛中，耗资数十亿美元的 AI 超级计算机绝不能因为网络交付滞后而闲置。为了加快落地，网络必须采用极简的扁平交换层级，这既能压低通信延迟，又能最大限度减少需要铺设和熔接的光纤线缆总量。此外，机架布局和布线工程要与土建施工阶段无缝衔接，并通过软件系统对物理世界的空间尺寸、弯曲半径与布线约束进行全真建模与预先推演，从源头上彻底消除初次现场施工时的装配冲突。 ```mermaid flowchart TD accTitle: Virgo Network Core Design Principles accDescr: 展现驱动 Google Virgo 数据中心网络研发与部署的四大核心设计原则。 subgraph Principles["Virgo 四大设计原则"] P1["专精超算架构 (与加速器硬件同频共振设计)"] P2["内生网络韧性 (单链路/单设备故障不中断训练)"] P3["高精度遥测闭环 (微秒级监控支撑流控与负载均衡)"] P4["极致交付速度 (极简网络层级与机房数字孪生预演)"] end ``` ### [06:09](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=369s) 统一计算域：Scale-up、Scale-out 与前端网络的三层专业化解耦为了落实上述原则，Google 毅然打破了传统的通用数据中心架构，转而构建了一个由三层专业化网络协同运作的“统一计算域”（Unified Compute Domain）。这种网络解耦清晰地将加速器内、跨节点横向扩展与前端数据供给划分为三个独立维度： ```mermaid flowchart TD accTitle: Three Specialized Layers in AI Supercomputer accDescr: 展示 AI 超级计算机中由 Scale-up、Scale-out (Virgo) 和前端网络 (Jupiter) 构成的三层解耦网络架构。 subgraph Top["前端网络层 (Front-end)"] JUPITER["Jupiter 经典网络 (连接分布式存储、广域网及通用计算节点)"] end subgraph Middle["横向扩展网络层 (Scale-out)"] VIRGO["Virgo 专用加速网络 (扁平无阻塞架构，连接 Pod 级横向通信)"] end subgraph Bottom["垂直扩展网络层 (Scale-up)"] TPU_ICI["TPU 内部光路互联 (ICI)"] GPU_NVL["GPU 内部高速互联 (NVLink)"] end JUPITER <--> VIRGO VIRGO <--> TPU_ICI VIRGO <--> GPU_NVL ``` - **Scale-up 纵向扩展域**：处于最底层，专为单 Pod 内芯片间的强耦合通信提供极致带宽和微秒级延迟。例如 TPU Pod 内部大量依托基于光交换芯片的芯片间互联（ICI, Inter-Chip Interconnect），而在 GPU 集群中则采用 NVLink 拓扑。 - **Scale-out 横向扩展域（Virgo）**：即本次重磅公布的核心层，专为跨 Pod 的加速器间大规模水平扩展而生，以实现巨幅双向等分带宽为第一设计指标。 - **前端网络层（Jupiter）**：作为 Google 久经考验的成熟网络，负责连接分布式海量存储与通用 CPU 计算节点，同时为跨数据中心的多站点超大模型训练提供中继。Jupiter 在同园区内由多个 GPU 和 TPU 集群安全共享，确保数据加载不会成为训练与推理服务的吞吐瓶颈。这种将前端存储与后端算力网络物理分离的架构带来了三大战略优势： 1. **各域独立演进**：前端存储网络与后端加速网络各自按需迭代升级，任何一层的架构调整都不会引起全局业务震荡。 2. **兼顾吞吐与 TCO**：在横向扩展层不计成本地部署全线速、无阻塞的对分带宽以保障大模型训练，而在前端网络侧则可继续平衡总体拥有成本（TCO）与网络弹性。 3. **加速器代际对齐**：Scale-out 专用网络能够与新一代 GPU（如 NVIDIA Vera Rubin NVL72 平台）或新一代 TPU 架构保持严格同频换代。 ### [08:36](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=516s) Virgo 深度剖析：扁平二层架构、多平面隔离与百万芯片扩展 JK Lee 随后通过系统大图深入拆解了 Virgo 网络的内部结构，并按照核心设计点由左向右进行了详细盘点。首先，Virgo 充分利用了具备超大端口密度的高基数（High-radix）交换芯片，成功将传统复杂的多层 Fat-Tree 拓扑压平成极为扁平的“二层无阻塞拓扑”（Two-layer non-blocking topology）。这一精简极大地减少了数据包在集群内穿透交换芯片的跳数（Hops），将网络端到端转发延迟和抖动压低到极致。其次，Virgo 在将加速器机柜连入网络架构时采用了创新的多平面（Multi-plane）拓扑，并在每个网络平面内部署了完全隔离独立的控制域。这种物理与逻辑层面的彻底解耦，带来了出色的故障遏制能力——任何单平面的软硬件失效都会被牢牢圈禁在当前平面内，而不会向全网级联扩散。第三，Virgo 的连接半径打破了单栋建筑的物理束缚，其光纤织网直接延展到了园区级（Campus-level）的多个数据中心楼宇。这种跨站点互联能力，使得 Virgo 能够在单个逻辑集群中弹性扩展并支持高达 100 万颗加速芯片（如 TPU 节点）协同训练，为超万亿参数模型的大规模分布式训练扫清了物理边界阻碍。在此过程中，中枢位置的 Jupiter 网络则平稳承担外围支撑，保障海量 Checkpoint 存储读取与外部 WAN 通信的稳定顺畅。 ### [10:13](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=613s) 细粒度“细链路”（Skinny Links）与多平面高韧性设计多平面设计在目前的大规模网络产业界属于相对前沿的构型，JK Lee 对其物理实现细节进行了透彻阐释。为了充分释放高基数交换机的互联规模，Virgo 引入了名为“细链路”（Skinny Links）的技术，将单条逻辑链路的线速直接精细化降维至 SerDes 物理通道级别（例如 200 Gbps SerDes 单通道速率）。这种做法使得单颗交换机芯片能拆分出最大化数量的独立物理端口，从而在同一层级内换取成倍提升的扇出连接能力与组网规模。在这种架构下，集群划分的网络平面数量 $N$，直接对应单个加速器芯片所能引出的细链路数量。以单颗芯片配置 1.6T 吞吐能力的网卡（NIC）为例： - 在采用 200G SerDes 速率的前提下，单颗加速器芯片可拆分出 8 条独立的细链路； - 这 8 条细链路分别上连至 8 个完全平行的物理网络平面（$N=8$）； - 当其中任意一个网络平面或某条链路发生严重硬件故障而宕机时，剩余的 $N-1$ 个平面仍然完好运行，单芯片依然保有高达 87.5% 的可用通信带宽。这一容灾表现相比于传统架构形成了代际质变——在过去单平面或二路冗余方案中，单一交换机或骨干光纤发生故障，往往会直接导致单节点承受 50% 甚至 100% 的带宽骤降，进而造成严重的长尾阻塞甚至任务死锁。更巧妙的是，借助“细链路”衍生出的多平面架构，无需在上方叠加第三层交换网络，就能使整个无阻塞网络的集群规模线性放大 $N$ 倍。此外，后端的加速器导轨（Rail）与网络平面做到了完全对应匹配，无论是加速器间的 Rail 内部通信还是跨 Rail 通信，均能在各自平面内得到全线速的端到端吞吐支持。 ### [11:53](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=713s) 交换层级精简与数据包级多路径负载均衡在数据中心内部，通过以全对分带宽（Full Bisection Bandwidth）构建扁平化网络平面，大幅减少了所需的交换架构层级（Switching Layers）。这种层级精简带来了显著的工程优势：部署和管理所需的机柜与光纤数量大幅减少，从而极大加快了数据中心的整体交付与部署速度。面对业界讨论极高的多路径路由（Multipath Routing）效率课题，Virgo 架构原生支持两种数据包级别的负载均衡机制： 1. **交换机驱动的自适应路由（Switch-driven Adaptive Routing）**：能够实现微秒级的极速响应，不仅可以瞬间应对交换机本地端口故障或拥塞，还能迅速侦测并避开远端下游（Remote Downstream）的链路故障。 2. **基于 SRv6 的源路由（SRv6-based Source Routing）**：专为运行自定义网络协议栈的裸金属（Bare-metal）租户设计，允许主机端直接感知网络拓扑，在数据包粒度上将流量均匀分发到整个 Fabric 网络中所有可用的路径上。 ```mermaid flowchart TD accTitle: Virgo 数据包级多路径负载均衡架构 accDescr: 展示自适应路由与 SRv6 源路由两类数据包级负载均衡方案 subgraph VirgoFabric["Virgo Fabric 负载均衡机制"] direction TB subgraph AR["方案一：自适应路由 (Adaptive Routing)"] A1["交换机硬件驱动"] --> A2["快速本地链路状态感知"] A2 --> A3["远端下游故障即时规避"] end subgraph SR["方案二：SRv6 源路由 (Source Routing)"] B1["裸金属主机网络栈"] --> B2["SRv6 报头路径编排"] B2 --> B3["全网络可用路径包级均衡"] end end ``` - 精简交换机层级后，机架占用与光缆布线规模显著下降，降低了物理维护复杂度。 - 架构全面兼顾了超大带宽、确定性低时延以及面向高故障率底座的韧性设计。 --- ### [13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) AI 工作负载的可观测性挑战与带内拥塞信标（CSIG） AI 训练任务具有典型的大体量同步（Bulk Synchronization）与极端突发特性，极易在局部触发微突发拥塞。在超大规模集群中，网络拥塞、硬件丢包和软件 Bug 必然发生，若无法及时捕捉，微小的扰动就会演化为阻断整个 AI 训练流水线的高昂停机事故。因此，Virgo 从一开始就将遥测（Telemetry）提升为系统的“一等公民”，并在架构中同步落地了两大全新方案。第一种方案是来自每个交换机端口的高分辨率流式拥塞指标监控；第二种方案则是直接将高精度拥塞遥测融入数据包内部的带内拥塞信标技术——**CSIG（Congestion Signaling）**。这项技术是传统带内网络遥测（In-band Network Telemetry, INT）的高级演进版本。传统的 INT 会在每个交换机跳点向报头堆叠元数据，导致数据包报头随着跳数增加而持续膨胀，造成严重的报头开销；而 CSIG 采用创新的跨跳聚合机制，利用 Min 或 Max 聚合函数在路径各跳之间动态汇总拥塞指标，保证无论数据包经过多少跳交换机，报头开销始终恒定。同时，CSIG 依然携带有精确定位器（Locator），能准确指明当前瓶颈产生所在的交换机节点。对于这项技术的演进，演讲者由衷感慨：“带内网络遥测（INT）实际上是我当年在 Barefoot Networks 的第一个项目，如今能继续参与 CSIG 的设计、知识产权推动以及后续标准化工作，让我倍感兴奋。”目前，CSIG 已获得最新交换芯片与智能网卡（NIC）的端到端硬件支持，不仅被超以太网联盟（Ultra Ethernet Consortium, UEC）接纳为核心标准，Google 团队也在 SIGCOMM 顶级学术会议上发表了论文，全面公开其架构设计与大规模生产部署经验。 - **报头开销对比**：传统 INT 随跳数线性膨胀，CSIG 无论跨越多少跳网络均保持固定尺寸报头。 - **拥塞定位**：在恒定报头中保留 Locator 字段，可直接追踪产生拥塞峰值的物理瓶颈节点。 - **标准化进展**：已成为 UEC 官方标准组件，并在 SIGCOMM 发表生产落地研究成果。 --- ### [15:47](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=947s) 遥测实时演示：揭开微突发与隐形拥塞的真相通过实际的遥测监控仪表盘对比演示，高分辨率遥测对保障 AI 系统的必要性被淋漓尽致地展现出来。仪表盘最初呈现的是业界通行的 30 秒采样分辨率标准监控视图，左侧记录工作负载带宽，右侧展示交换机端口利用率。在 30 秒平均值视角下，整网运行曲线异常平稳，看似处于极度健康的低负载状态，距离 100% 满载尚有巨大的安全余量。然而，演讲者一针见血地指出：“30 秒的平均值极具欺骗性，它们抹杀了导致系统不稳定的微突发（Micro-bursts）。”当操作员在系统界面一键切换到亚毫秒级（Sub-millisecond）高精度遥测视图时，原本风平浪静的健康基线瞬间荡然无存——右侧仪表盘爆发出密集冲顶的绿色立柱，展示着一次次高达 100% 饱和度的严重拥塞事件；而此时代表传统均值的蓝线却依然温和地显示着“10% 平均负载”。真实情况是，交换机硬件每隔几秒钟就被严重的突发流量推入“转速爆表（Red-lining）”的过载危险边缘。这种在 AI 同步突发阶段产生的“隐形拥塞”，在常规监控甚至应用层 100 毫秒粒度的监控视图中都完全无法被捕获（应用层视图显示的黄色立柱依然漏过了关键毛刺）。Google 目前正全面依托这套微秒级的遥测洞察能力，深挖链路拥塞根因，以反哺和加固现网基础设施与下一代网络架构设计。 - **宏观假象**：30 秒监控视图中显示端口平均负载仅约 10%，掩盖了周期性线速冲顶的真实情况。 - **微观真相**：亚毫秒级监控准确捕获到了每隔数秒即出现一次的 100% 端口饱和突发。 - **应用监控局限**：即便是应用侧 100 毫秒级别的细粒度监控，依然无法反映底层物理交换机的微突发拥塞。 --- ### [17:39](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=1059s) 跨城域扩展、多厂商互通与网络黄金时代回顾全篇，Virgo 作为专为大规模 AI 网络量身打造的横向扩展 Fabric，在可扩展性、超大带宽、超低时延与故障韧性方面树立了标杆，并由贯穿管理控制平面与数据平面的全栈高精度遥测深度赋能。放眼未来演进路径，研发团队正重点攻关两大核心方向： 1. **跨多城域（Multi-Metro）扩展**：将 Virgo Fabric 的连接边界从单一数据中心园区扩展至相距数百甚至数千公里的远距离异地园区。在跨城域毫秒级传播时延的广域长途连接（Long Haul）上实现无损、高性能的 RDMA 是一项巨大的工程挑战，Google 正与行业伙伴密切合作，推动 CSIG 演进以实现亚 RTT（Sub-RTT）级别的拥塞与故障信令通知。 2. **多厂商 Fabric 统一生态**：呼吁全行业加快推进跨厂商互操作性与协议标准化建设，无论是在底层遥测采集协议，还是在自适应路由中的快速路由变更通知机制上，都亟须建立统一的开放生态。在演讲尾声，讲者由衷感慨：“从宏观视角来看，当前正是从事计算机网络研究最激动人心的时代！”诸如带内网络遥测（INT）、CSIG、自适应路由、数据中心源路由等由网络技术社区在过去数十年间探索研发的硬核技术，有些在早期甚至一度被视作超前设计或被冷落，但如今爆发式的 AI 基础设施需求，为这些底层通信创新提供了前所未有的完美应用舞台，正在 Google 及行业各大尖端 AI 集群中以极快的节奏被全面采用落地。演讲在对 Google Fellow 团队的技术致谢中圆满结束。 - **远距挑战**：解决数百至数千公里广域链路毫秒级传播延迟下的 RDMA 吞吐与低时延保持。 - **协议演进**：研发支持亚 RTT 信令广播的增强型 CSIG，应对广域长距离拥塞与故障反馈。 - **行业呼吁**：加速推进自适应路由快变通告、多厂商硬件即插即用与标准化落地。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=5s) 顺便提一下，这件 T 恤是 2018 年的 — Jeongkeun "JK" Lee > [02:23](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=143s) 基本上任何东西都会发生故障 — Jeongkeun "JK" Lee > [03:41](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=221s) 保持同步节拍（Lock-step）联合设计与定制 — Jeongkeun "JK" Lee > [13:05](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=785s) 带内网络遥测（INT）实际上是我当年在 Barefoot Networks 的第一个项目，如今能继续参与 CSIG 的设计、知识产权推动以及后续标准化工作，让我倍感兴奋。 — Jeongkeun "JK" Lee > [15:47](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=947s) 30 秒的平均值极具欺骗性，它们抹杀了导致系统不稳定的微突发（Micro-bursts）。 — Jeongkeun "JK" Lee > [15:47](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=947s) 转速爆表（Red-lining） — Jeongkeun "JK" Lee > [17:39](https://www.youtube.com/watch?v=00Gcgk9FJf4&t=1059s) 从宏观视角来看，当前正是从事计算机网络研究最激动人心的时代！ — Jeongkeun "JK" Lee ## Mentioned Jeongkeun "JK" Lee, Google, HP Labs, Barefoot Networks, Networking @Scale, Virgo, Jupiter, Orion SDN, Apollo OCS, QUIC, TCP BBR, Andromeda, gRPC, PSP, Falcon, TPU, NVIDIA, Vera Rubin NVL72, Google Cloud A5x, ICI, NVLink, Skinny Links, SRv6, CSIG, INT, Ultra Ethernet Consortium (UEC), SIGCOMM, RDMA ## Video description To address the unique requirements of AI networking, we leverage our proven Jupiter network for north-south traffic and are introducing Virgo, a new fabric for east-west scale-out communication. Virgo is a dedicated accelerator-to-accelerator fabric optimized for massive horizontal scale. It is engineered for deterministic latency and maximum resilience to provide high goodput for the ML workload. Virgo will enable the new Google Cloud's A5x bare-metal instances, which run the NVIDIA Vera Rubin NVL72 platform. It is also a key building block for next-gen TPUs. This talk will discuss Virgo's design principles for resilience, observability, deployability, and multi-building scale. Learn more about the @Scale conference here: https://atscaleconference.com/ [Source](https://www.youtube.com/watch?v=00Gcgk9FJf4)

Source quotes

[0:05] Jeongkeun "JK" Lee: 顺便提一下，这件 T 恤是 2018 年的 [2:23] Jeongkeun "JK" Lee: 基本上任何东西都会发生故障 [3:41] Jeongkeun "JK" Lee: 保持同步节拍（Lock-step）联合设计与定制 [13:05] Jeongkeun "JK" Lee: 带内网络遥测（INT）实际上是我当年在 Barefoot Networks 的第一个项目，如今能继续参与 CSIG 的设计、知识产权推动以及后续标准化工作，让我倍感兴奋。 [15:47] Jeongkeun "JK" Lee: 30 秒的平均值极具欺骗性，它们抹杀了导致系统不稳定的微突发（Micro-bursts）。 [15:47] Jeongkeun "JK" Lee: 转速爆表（Red-lining） [17:39] Jeongkeun "JK" Lee: 从宏观视角来看，当前正是从事计算机网络研究最激动人心的时代！

Source quotes

[0:41] Jeongkeun "JK" Lee: basically everything to break. [8:36] Jeongkeun "JK" Lee: Skinny Links & Multi-Plane Isolation [13:05] Jeongkeun "JK" Lee: MIN [13:05] Jeongkeun "JK" Lee: MAX [15:47] Jeongkeun "JK" Lee: Standard 30s View: [--- 10% Avg Load ---] -> (Looks healthy, hides problems) Sub-millisecond View: [||| 100% SPIKE |||] -> (Redlining switches continuously)

Original transcript

[0:05] Hi everyone, thanks for having me here. [0:08] I am JK Lee representing my team at [0:11] Google. Before Google, I started my [0:14] career as a researcher at HP Labs where [0:17] I really enjoyed the collaborations in [0:19] the research community. Then I joined [0:21] barefoot networks known for P4 and [0:23] Tofino. And that's when I was invited to [0:27] my first networking and scale event back [0:30] in 20. [0:32] This t-shirt is from 2018 by the way and [0:35] I really appreciate the unique industry [0:37] perspectives openly share in that events [0:40] in a timely manner. Since that I've been [0:43] following and attending this event [0:44] series and that makes me feel very [0:47] honored to be here and introduce Virgo [0:50] brand new scale out data center network [0:52] for AI developed by many great teams at [0:55] Google. [0:57] Now we are living in 2026 and in the [1:00] midst of the AI era shown at the top of [1:03] this diagram and that's why we are here. [1:05] But let's briefly reflect on the last [1:07] decade of data center network [1:09] innovations before AI era. At Google, we [1:13] have crossed several fundamental [1:15] inflection points. Starting from the the [1:18] internet era at the bottom where to [1:21] provide global services like search maps [1:24] and Gmail, we had to rethink the data [1:28] center network from the ground up and [1:30] started Jupiter fabric and also pair [1:33] with Orion well known for its SDN [1:35] control plane. [1:37] We also pioneered Apollo the optical [1:40] switch circuit switching for flexible [1:44] expansion and intra. [1:46] Next came the streaming era where we [1:48] deliver high quality video at low jitter [1:51] to many end devices. [1:54] For that we have developed new [1:55] conjunction protocols like quick and TCP [1:58] VBVR. [2:00] And the third inflection point was the [2:02] cloud era which demanded extreme [2:05] multi-tenency reliability and security. [2:08] For that we develop Android meta and [2:10] gRPC well known for microservices [2:13] and top at the bottom we also move the [2:15] performance into the hardware for [2:17] example PSP for encryption and farcon [2:20] for reliable transport. [2:23] Next let's take a look at what's driving [2:26] the AI error. The grab on the left shows [2:29] that foundational models are growing [2:31] exponentially in terms of the number of [2:33] parameters. It makes AI training a very [2:36] network intensive. It also drives [2:39] exponential increase in the required [2:41] network bandage as shown on the right [2:43] hand side and it comes with very strict [2:45] delay bounds and the network for AI [2:48] supercomputers [2:49] spans across multiple data center [2:51] buildings, campuses and at this scale we [2:54] know basically everything breaks. So we [2:57] should make these failures invisible to [2:59] AI workloads. [3:03] To address these new challenges of AI [3:06] error in this talk, I'd like to share [3:08] four things with you. First, I want to [3:11] share the principles that drive our [3:13] design and execution of our new scale [3:16] out data center network which is Virgo. [3:19] Next, we will detail the Virgo [3:21] architecture and show how its position [3:23] with the rest of AI supercomputer [3:25] networks. [3:26] And we learned that telemetry is a key [3:28] for performance and reliability. We'll [3:31] introduce the new telemetry solutions [3:33] with a short demo video. Hope you enjoy [3:35] that. And we'll conclude with the next [3:37] steps. [3:41] All right, let's start from the design [3:42] principles that governed our design and [3:46] execution of verbal. [3:48] First, it should be designed as a part [3:51] of the entire building or campus as a [3:54] one big supercomputer optimized for the [3:57] one or two AI use cases instead of being [4:00] a generic infrastructure that serves [4:02] many different use cases. And this means [4:05] that the networking hardware including [4:07] switches and optics fibers, they need to [4:10] be designed and spec in a lock step with [4:12] each new generation of ML accelerators. [4:16] Consequently, the scaleout network life [4:19] cycle itself is tightly coupled with the [4:21] GPU and TPU AI cluster and things like [4:25] backward or forward compatibility and [4:27] flexible expansion where supported by [4:30] Jupiter and Apollo are not strong [4:32] requirements anymore. [4:34] And secondly, the resiliency should be [4:37] baked in the network design so that the [4:40] training job should not crash for a [4:42] single link or switch failure. [4:45] And once the physical infra is there [4:47] built for the resiliency we need high [4:50] resolution high quality telemetry [4:53] to enable transport and management [4:55] protocols to to drive the best [4:58] performance out of this physical [4:59] infrastructure and also availability out [5:02] of it. It can be done by congestion [5:06] control and load balancing at data plane [5:08] or the tuning and debugging at the [5:10] management plane all heavily depend on [5:13] the quality and timeliness of the [5:15] telemetry data. [5:18] Finally, the world is moving so fast in [5:21] a error. So design must enable speed [5:24] especially the deployment speed. [5:27] The networking should not be in the way [5:30] of delivering this multi-billion AI [5:33] supercomputer to users and this [5:35] motivates us to have a minimal switching [5:38] layers and it helps not only the latency [5:40] performance but also minimize the number [5:42] amount of networking materials that we [5:44] have to deploy [5:47] and then we found the rack layout and [5:49] fiber design should support fast [5:51] deployment workflow and better align [5:53] with the building construction phases. [5:56] And finally, modeling such a physical [5:58] word dimensions and constraints in [6:00] software and being able to draw on them [6:02] in advance helps a lot in terms of [6:05] avoiding the first time deployment [6:07] issues. [6:09] With this set of requirements in mind, [6:13] let's take a look at how we solve the [6:16] extreme requirements [6:18] and in a more fundamental way. [6:21] So we made a fundamental shift away from [6:24] general purpose network design towards a [6:26] specialized flat low latency network [6:29] architecture. So our new data center [6:31] architecture features three distinct and [6:34] specialized layers that operate as a one [6:37] unified compute domain. First, as you [6:39] look at the bottom orange boxes, the [6:42] scale of network domain, which is high [6:45] bandwidth, low latency fabric designed [6:47] for tightly coupled communication [6:49] between accelerators within a single [6:51] pod. For example, the optical switching [6:54] based ICI interchip interconnect is [6:57] heavily used at the TPU pot. Secondly, [7:00] our scale out accelerator fabric in [7:02] yellow boxes are dedicated fabric [7:05] optimized for massive horizontal scale [7:08] across the pots. And we will driving [7:11] deeper into the specifics of this new [7:13] layer in just a moment. And third, we [7:16] have Jupiter in between both GPU and TPU [7:20] clusters and which is our proven [7:22] front-end network fabric that provides [7:25] fast and reliable access to the [7:27] distributed storage and general purpose [7:29] computer resources. It ensures that data [7:32] access does not become a bottleneck for [7:35] training and then AI serving workloads [7:39] and is also used to scale out multiple [7:42] sites for very large training runs. And [7:45] as you can see here, Jupiter can be [7:47] easily shared by multiple AI cluster in [7:49] the same building. [7:53] Then this architecture decoupling [7:55] between front end and scale our back end [7:57] provides three strategic advantages. [8:00] First, we can evolve and upgrade each [8:02] network domain independently preventing [8:05] any systemwide disruptions. [8:08] Secondly, by using a dedicated [8:10] non-blocking network and scale out, we [8:12] deliver massive bisectional bandwidth [8:14] for critical training task while we can [8:17] still optimize for TCO and expansibility [8:20] at front end networks. [8:23] And third, the scale network is built in [8:26] lock step with each new generation of ML [8:28] accelerators, helping ensure the fabric [8:32] is matched to the AI hardware that it [8:34] supports. [8:36] In this new slide, let's zoom more into [8:39] the Virgo. There's a lot going on. [8:41] Please take a peek. I will walk you [8:43] through from left to right with the [8:46] number based cues. On the left hand [8:48] side, we have Virgo network which [8:50] leverages higher radics switches. By [8:53] doing so, we are able to collapse many [8:56] networking layers into a flat two-layer [8:59] non-blocking topology, which is critical [9:02] because it significantly reduces latency [9:05] by minimizing number of switching hops [9:07] that a packet has to traverse. And [9:09] secondly, number two, Virgo employs a [9:12] multiplaner design to connect our [9:15] accelerator racks to the fabric in a [9:17] scalable manner. And we also deploy [9:19] independent control domains per plane. [9:22] By doing so, we ensure that the network [9:24] isn't just fast, but also reliable [9:27] against many different failures [9:29] containing the forts within a single [9:31] plane. To add to it, number three, VGO [9:34] is expensible across multiple data [9:37] center sites, effectively removing the [9:39] physical boundaries of a single building [9:41] and it allows us to scale flexibly at a [9:45] campus level. So it can provide the [9:48] scaling for up to a million TPU 80 chips [9:51] in a single logical cluster. [9:55] And again in the in the middle of [9:56] diagram we still have Jupiter the Google [10:00] state-of-the-art front-end network. So [10:02] while Verbal network handles the [10:04] accelerator traffic, Jupiter ensures [10:07] that access to storage and the global [10:09] WAN is fast and reliable. [10:13] And let's get into more details about [10:15] the multiplane design of Virgo which is [10:18] relatively new in the industry. To [10:21] enable the high and maximum radics [10:24] switching we use something called skinny [10:27] links where each logical link speed is [10:30] configured as low as 30 lane speed. So [10:33] this gives us a maximum switching radics [10:35] has maximum scaling in the same layer [10:37] and given the number of planes denoted [10:40] as n in this slide is a function of [10:43] number of skinny links available per [10:45] accelerator. For example, a GPU may have [10:49] a 1.6T nick bandwidth with the use of [10:52] 200 gig 30 lane speed. It gives us eight [10:55] skin links available per chip and this [10:58] improves the resiliency because when one [11:01] plane is down we still have n minus one [11:03] planes available. So for eight planes it [11:06] allows 87.5% [11:09] bandwidth still available for each check [11:12] which is a big improvement over the [11:14] prior designs where single link or [11:16] single switch failure leads to uptain [11:20] 100% or 50% bandwidth reduction per [11:23] chip. [11:25] And here the multiplane design with [11:27] skinny links allows non-blocking domain [11:29] to scare by n times which is number [11:32] planes without needing another third [11:34] layer switching on top. [11:36] And another interesting point here is [11:38] that the accelerator rares or GPU rares [11:41] or TPU rares as we know so far and the [11:44] network planes are fully matched each [11:47] other. So not only intraare [11:49] communications but also crossware [11:51] communications are supported in each [11:53] plane at full bisection bandwidth and [11:57] this reduction of switching layers [11:59] improves deployment speed because less [12:02] number of racks and fibers we need to [12:03] deploy and manage. [12:07] And finally for efficient multipath [12:10] routing which has been pretty hot topic [12:13] in the industry we support two packet [12:15] level load balancing schemes. [12:18] First switchdriven adaptive routing. It [12:21] reacts to reacts really fast to not only [12:24] the switch local failures or conditions [12:27] but also we enable speedy reaction to [12:30] remote downstream failures. [12:33] And we also have um SRV6 based source [12:37] routing supported by Virgo switches for [12:40] the biomet customers who can bring in [12:42] their host stack which can distribute [12:44] the load across the all available paths [12:47] in the fabric at packet level [12:49] granularity. [12:51] Thanks for walking me through this very [12:54] busy slide. And to recap, Virgo is [12:56] designed to provide a vast bandage [12:58] latency performance at scale with [13:01] resiliency to failures. [13:05] But we know that congestions and [13:08] failures and bugs can still happen and [13:11] they do happen and particularly AI [13:14] workloads are known for its [13:16] characterization of Burke [13:18] synchronization [13:20] and this extreme burstiness as shown in [13:22] this dashboard on the left hand side and [13:26] this demands high resolution [13:27] observability for us to prevent costly [13:30] interruptions to AI workloads. because [13:32] of any congestion or failure events. So [13:35] we took telemetry as a first class [13:37] citizen in Virgo design and and and [13:40] execution and we support two types of [13:42] new solutions. [13:45] First we have a streaming of high [13:47] resolution condition metrics such as the [13:50] port switch port bandwidth utilization [13:54] shown here from every switch port. [13:58] We will see a short demo in the next [14:00] slide. But before that, I'm also happy [14:03] to share that the same high resolution [14:06] congestion metrics are also available at [14:09] each data packet in in for data data [14:12] plane consumption. This technology is [14:15] called CEC. It's a new inband congestion [14:18] signaling mechanism. [14:20] This sound may this may sound familiar [14:22] to intend network telemetry if you have [14:26] heard about it. [14:28] And you're right. CIC is a advanced [14:31] version of INT. So instead of stacking [14:34] the per switch hub information in a [14:37] header, increasing the header size [14:39] bigger than the original size, CCI uses [14:43] a new mechanism aggregating the matrix [14:46] across different switch hops either [14:48] using min or max aggregation functions. [14:51] So it keeps the [14:53] header overhead constant regardless of [14:56] how many hops the packet has go through. [14:58] It also provides the locator information [15:00] telling the exact bottleneck point where [15:04] the congestion metric is collected [15:07] and CC is supported end to end in verbal [15:10] by the latest switch and nick hardware [15:12] and personally it's very exciting for me [15:14] because actually in was my inband [15:17] network calmetry was my first project at [15:19] the airport networks and I could help on [15:22] C6 design and IPO smart and later [15:24] standardation efforts speaking of that [15:28] CIC is going to be a new UEC ultra [15:31] ethernet consortium standard and there [15:34] is another SECOM 2006 26 paper coming [15:38] out to detail the CC design and our [15:41] deployment history and experiences [15:44] written by my colleagues at Google. [15:47] Next we will do a short demo video of [15:50] the high resolution streaming telemetry. [15:52] You see here the dashboard for the [15:54] telemetry. Left hand side we have a [15:57] workload bandwidth sampled at the 30 [16:00] secondond resolution. Also right hand [16:02] side switch port utilization measured at [16:05] the same resolution. [16:07] So our dashboard start with the standard [16:10] monitoring view at 30 second resolution. [16:13] The network looks all healthy showing [16:15] massive headroom before 100%. [16:18] But the 30 seconds averages are [16:21] misleading. They erase the micro bursts [16:24] that derive the system instability. [16:27] Now let's look at what is actually [16:29] happening on the wire by toggling to the [16:32] sub millisecond telemetry view. [16:36] The health baseline is gone. Those green [16:38] pillars on the right hand side 100% [16:42] saturation event while the blue line the [16:45] average still claims a 10% average load. [16:48] We are actually red lining the switch [16:50] harder every few seconds. [16:52] This is the invisible congestion during [16:55] AI workload synchronized burst. The [16:57] standard monitoring cannot detect even [17:00] the 100 millisecond view from the [17:03] application itself. The yellow pillars [17:06] on the left it fails to detect such [17:08] burst. So we are currently using this [17:11] high resolution insights to harden [17:13] Google infrastructure and next [17:15] generation designs. To recap, Virgo is a [17:19] new scaleout fabric designed for AI [17:23] networking delivering the performance in [17:26] the latency and bandwidth performance at [17:28] scale and reliability. [17:30] It is powered by new high resolution [17:32] telemetry available both in the [17:35] management control plane also in the [17:37] data planes. [17:39] Looking forward, our first next step [17:42] includes [17:44] scaling the Virgo fabric across multiple [17:46] metros, not only campuses. It means that [17:50] we need to connect data center campuses [17:54] hundreds or thousands of kilometers [17:56] apart. [17:58] And as you can imagine, delivering high [18:00] performance RDM performance over long [18:02] haul connection is very challenging due [18:04] to the millisecond level latency between [18:06] the sites. So we are working closely [18:09] with industry partners to expand CC for [18:13] a sub RT signaling of condition and [18:15] failure events. [18:17] And the next part is a multi vendor [18:19] fabric that many of us might be [18:21] interested in. And this is a call for [18:24] many of us to introduce better [18:27] interoperability and protocol [18:29] unification whether this is telemetry [18:31] protocol or our fast route change [18:34] notification part of adaptive routing. [18:37] But taking a step back, it's really [18:40] exciting time to do computer networking. [18:42] The cool innovations that we networking [18:46] community have developed for the last [18:48] decades like inbed network telemetry, [18:51] CC, adaptive routing, source routing in [18:53] data center are being widely adapted at [18:57] Google and many other AI clusters at [18:59] very fast pace. Some of them were [19:02] overlooked initially, but now AI [19:04] networking gives us a perfect use cases [19:06] for that. [19:09] With that, thank you for having me again [19:11] here. I'd like to close with this QR [19:14] code leading to a blog post from [19:16] Argentine Google fellow who also help me [19:20] on this slide deck. And feel free to [19:22] reach out to me in the email address [19:24] here for any questions and collaboration [19:27] opportunities. Thank you.