Google Overhauls TPU Silicon and Pod Fabric to Speed AI Agents

Architecting Infrastructure for the AI Native Future: Scaling Autonomous Agents on Google TPUs — @Scale

@Scale

Inference latency has replaced algorithmic design as the binding constraint for autonomous agents, forcing hardware architectures to shift from raw training throughput to custom on-die memory and low-diameter network fabrics.

AI-assisted summary of Architecting Infrastructure for the AI Native Future: Scaling Autonomous Agents on Google TPUs — original source. Check the original for full context.

Source published:

Key ideas

  • Google outlined its latest TPU hardware innovations designed specifically for autonomous AI agents, highlighting the throughput-oriented TPU 8T and latency-optimized TPU 8i architectures. (2:27)
  • TPU 8T scales to 9,600 chips per pod via Optical Circuit Switches (OCS), reaching 121 exaflops of FP4 compute while cutting networking power by 40% and costs by 30% compared to electronic switches. (8:41)
  • The TPU 8i introduces the hierarchical 'Boardfly' interconnect topology across 1,152 chips, reducing network diameter by 53% and cutting distributed inference latency by over 50%. (14:25)
  • Google's VirgoNet fabric links more than 134,000 TPU chips across a single data center building and scales beyond 1 million TPUs when unified across global WAN networks. (10:45)

Detailed notes

## Background Presented by Sabastian Mugazambi from Google and published on YouTube by the @Scale channel on June 19, 2026, this conference presentation examines how physical infrastructure has become the primary bottleneck for AI innovation. The accompanying description highlights how Google TPU generations are architected to support the massive throughput and low-latency inference demanded by recommendation engines, Mixture-of-Experts models, and emerging autonomous agents.

## TL;DR As autonomous AI agents shift workloads toward continuous execution, expansive context windows, and real-time reasoning loops, hardware infrastructure must fundamentally evolve to sustain agentic cognition. Google addresses these demands through specialized silicon—pairing throughput-focused TPU 8T clusters for pre-training and decodes with latency-optimized TPU 8i chips featuring massive on-chip SRAM and direct vector streaming. Scale-up and scale-out are achieved using Optical Circuit Switching (OCS), the datacenter-wide VirgoNet fabric, and the three-tiered Boardfly interconnect topology that cuts network diameter by over 50%. Ultimately, these low-latency architectural co-designs lay the essential groundwork for training frontier interactive world models and real-time physical AI.

## Atmosphere & style The presentation carries the polished, authoritative, and direct tone of a tier-one systems architecture keynote. Mugazambi delivers the technical breakdown with measured precision, methodically unpacking deep silicon microarchitecture, optical fabrics, and distributed routing without getting bogged down in marketing jargon. It feels like an insider look into the physical realities of hyperscale AI, creating an engaging tension between low-level hardware constraints and the high-concept future of real-time autonomous reasoning.

## Bulletin - [02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) Google outlined its latest TPU hardware innovations designed specifically for autonomous AI agents, highlighting the throughput-oriented TPU 8T and latency-optimized TPU 8i architectures. - [08:41](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=521s) TPU 8T scales to 9,600 chips per pod via Optical Circuit Switches (OCS), reaching 121 exaflops of FP4 compute while cutting networking power by 40% and costs by 30% compared to electronic switches. - [14:25](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=865s) The TPU 8i introduces the hierarchical 'Boardfly' interconnect topology across 1,152 chips, reducing network diameter by 53% and cutting distributed inference latency by over 50%. - [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) Google's VirgoNet fabric links more than 134,000 TPU chips across a single data center building and scales beyond 1 million TPUs when unified across global WAN networks.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) The primary bottleneck for AI progress has shifted from pure algorithmic design to physical compute and networking infrastructure capable of handling autonomous agents at scale. - [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) Autonomous agents require continuous execution across thousands of instances, long-horizon context windows, and real-time reasoning loops, pushing pre-training MoEs up to 10 trillion parameters. - [02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) Unlike general-purpose serial CPUs or parallel GPUs, Google TPUs are custom-engineered specialist ASICs optimized explicitly for matrix multiplication and high arithmetic intensity. - [05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) Moving block scale multiplication inside the MXUs removes VPU overhead during quantized matrix multiplications, improving Model FLOPs Utilization (MFU) across FP4, FP8, and BF16 formats. - [05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) Dedicated Sparse Cores on TPU 8T feature an LLM decoder engine that allows overlapping prompt prefill on Tensor Cores with autoregressive decode phases. - [05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) TPU RDMA and TPU Direct Storage bypass host CPU memory entirely, enabling ultra-long KV caches to stream directly between managed storage and HBM. - [08:41](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=521s) Racks built as 64-chip 3D torus cubes are dynamically interconnected using Optical Circuit Switching (OCS), avoiding electronic packet switches and physical recabling. - [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) Google's VirgoNet fabric offers non-blocking connectivity for over 134,000 TPUs within a single facility and achieves 40% lower training latency. - [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) Inference latency functions as a cognitive constraint for agent utility, because multi-step internal reasoning loops compound small delays into massive systemic latencies. - [12:18](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=738s) The latency-optimized TPU 8i places high-capacity SRAM on-die adjacent to compute units, enabling KV caches to reside beside MXUs and bypassing off-chip memory fetch penalties. - [12:18](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=738s) A specialized collective acceleration engine on TPU 8i delivers a 5× latency reduction for collective communication primitives. - [14:25](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=865s) The three-tier Boardfly topology connects 4 chips per board, 32 chips per rack group, and 1,152 chips per pod via OCS, halving end-to-end communication latency. - [15:32](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=932s) Emerging interactive world models and physical robotics demand millisecond reaction times, allowing autonomous agents to simulate thousands of lifetimes in virtual environments before physical deployment.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) The Demands of Autonomous AI Agents on Infrastructure

The rapid emergence of AI agents is stretching infrastructure far beyond its traditional boundaries. Unlike one-off queries, autonomous agents require continuous execution across thousands of concurrent instances. This demands massive throughput from AI accelerators as a fundamental prerequisite for continuous, real-time autonomy. Furthermore, these agents operate across long-horizon tasks, requiring expansive context windows—the core working memory that grounds agents in deep situational awareness and enables sustained multi-step task execution.

```mermaid flowchart TD accTitle: AI Agent Demands on Hardware Infrastructure accDescr: Three foundational requirements driven by autonomous agents: high throughput, vast context windows, and low-latency inference.

subgraph AgentNeeds["Core Demands of Autonomous Agents"] direction LR TP["Continuous Execution<br/>(Massive Accelerator Throughput)"] CW["Long-Horizon Tasks<br/>(Vast Context Windows & Memory)"] RT["Real-Time Reasoning<br/>(Low-Latency Memory Access & High-Speed Inference)"] end

subgraph Lifecycle["Impact Across Model Lifecycle"] direction LR Pre["Pre-Training MoEs<br/>(Up to 10T Parameters)"] RL["Reinforcement Learning<br/>(Multi-Step Reasoning Loops)"] Serve["Serving Hundreds of Thousands<br/>(Ultra-Low Latency Orchestration)"] end

AgentNeeds --> Lifecycle ```

These operational constraints span the entire development lifecycle of the agentic era: - **Pre-training:** Driving massive Mixture-of-Experts (MoE) architectures that now scale up to 10 trillion parameters, requiring specialized, high-throughput hardware. - **Post-training & Reasoning:** Employing reinforcement learning techniques such as sampling and iterative test-time reasoning loops before deployment to boost accuracy and efficiency. - **Serving at Scale:** Orchestrating hundreds of thousands of agents that are continuously prompting and awaiting immediate responses.

Because of this shift across the entire pipeline, the fundamental bottleneck for AI innovation is no longer purely algorithmic design; it has expanded to encompass the underlying physical infrastructure.

---

[02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) Compute Paradigms and a Decade of Google TPU Evolution

When evaluating AI and machine learning compute engines, processors fall into three distinct philosophical camps: - **CPUs:** "Generalist and serial thinkers." While highly versatile and flexible, they are bottlenecked by workloads requiring massive throughput. - **GPUs:** "Generalist, but they're parallel thinkers." Highly capable across graphics and parallel ML workloads, leveraging parallelism to avoid throughput choke points. - **TPUs:** "Specialists." Custom-engineered specifically for matrix multiplications—the fundamental mathematical operation behind modern neural networks.

Google has developed custom Tensor Processing Units for more than a decade, beginning with TPU v1 in 2015, specifically designed to break through the arithmetic intensity and interconnect walls that throttle agentic systems. Over that decade, Google introduced distributed shared memory across Inter-Chip Interconnects (ICI), direct liquid cooling, and 3D torus topologies to knit chips together into coherent supercomputers. The presentation highlights the newly announced TPU v6e and TPU v6p (referred to as TPU 8T and TPU 8i/TPU v8 generations).

Reimagining accelerator systems requires end-to-end co-design: - Custom silicon features tailored to matrix math. - Proprietary low-latency Inter-Chip Interconnect (ICI) topologies. - Data-center-wide scale-out fabric. - Physical infrastructure innovations, including vertical power delivery directly to the die and advanced liquid cooling.

---

[05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) Inside TPU v8: MXUs, Sparse Cores, and Direct Storage

Focusing on the throughput-optimized TPU silicon (TPU 8T), each Tensor Core incorporates Matrix Multiply Units (MXUs), Vector Processing Units (VPUs), and Activation Units (AXUs). The architecture supports low-precision arithmetic spanning FP4, FP8, and BF16, as well as mixed-precision workflows.

A notable microarchitectural advancement is moving block scale multiplication directly inside the MXUs rather than handling it on the VPUs. This provides zero VPU overhead for quantized matrix multiplications, delivering higher Model Flops Utilization (MFU) alongside high precision. The core relies on systolic arrays, which pipeline inputs directly through computational grids rather than fetching back and forth between memory and compute units on every cycle, ensuring the MXUs are continuously fed without stalling.

```mermaid flowchart LR accTitle: Direct TPU Memory Transfer Architectures accDescr: TPU RDMA and Direct Storage bypass host memory for ultra-low latency transfers.

subgraph RDMA["TPU RDMA"] T1["TPU A HBM"] <-->|"Direct via NIC (Zero Host Hop)"| T2["TPU B HBM"] end

subgraph Storage["TPU Direct Storage"] MS["Managed Storage<br/>(Long-Context KV Cache)"] -->|"Direct DMA to HBM"| T3["TPU HBM"] end ```

Beyond the dense Tensor Core, TPU silicon includes specialized **Sparse Cores**: - **Embeddings Acceleration:** Historically used in recommendation models to overlap dense compute layers (on Tensor Cores) with embeddings operations offloaded to the Sparse Cores. - **LLM Decoder Engine:** TPU 8T adds a dedicated LLM decoder engine to the Sparse Cores, granting them sufficient arithmetic intensity to overlap prefill execution on Tensor Cores with auto-regressive decode execution on the Sparse Cores.

For memory and data ingestion, Google eliminates host CPU memory hops entirely: - **TPU RDMA:** Directly streams data between the High Bandwidth Memory (HBM) of two separate TPUs via Network Interface Cards (NICs). - **TPU Direct Storage:** Allows TPU HBM to stream data directly to and from managed external storage. This enables ultra-long context KV caches to live on managed storage and swap into HBM without passing through host DRAM.

---

[08:41](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=521s) Scale-Up via 3D Torus and Optical Circuit Switching (OCS)

To scale up accelerators into a unified coherent unit, TPUs use a throughput-optimized 3D torus ICI topology. Every TPU connects directly to adjacent neighbors using wrap-around links, forming a tightly coupled cube of 64 chips that physically populates a single data center rack.

```mermaid flowchart TD accTitle: Optical Circuit Switching TPU Scale-Up Architecture accDescr: Optical Circuit Switches dynamically route fiber connections between 64-chip TPU 3D torus cubes to create a 9600-chip supercomputer.

subgraph TorusCubes["Racks (3D Torus Cubes)"] C1["Cube X<br/>(64 TPU Chips)"] C2["Cube Y<br/>(64 TPU Chips)"] C3["Cube Z<br/>(64 TPU Chips)"] end

subgraph Switching["Software-Controlled Routing"] OCS["Optical Circuit Switches (OCS)<br/>Direct Dynamic Fiber Reconfiguration"] end

C1 <--> OCS OCS <--> C2 OCS <--> C3 ```

Single racks are then interconnected using Optical Circuit Switching (OCS) over optical fiber paths: - **Cluster Scale:** Up to 9,600 TPU chips integrated into a single supercomputer pod. - **Compute Performance:** 121 exaflops of FP4 peak performance. - **Aggregated Memory & Bandwidth:** Over 2 petabytes of shared HBM and over 19 terabits per second of bidirectional ICI bandwidth per node. - **Software Reconfigurability:** OCS allows an input fiber from one rack/cube to be dynamically switched and pointed to another cube in software without physical recabling or manual intervention. - **Efficiency Gains:** OCS eliminates electronic packet switches in the optical path, yielding 30% lower cost and 40% lower power draw while unlocking arbitrary topological reconfigurations optimized for dynamic model and data parallelism.

---

[10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) Scale-Out Networking: VirgoNet and Planetary Clusters

Scaling out beyond single ICI pods into hundreds of thousands of chips requires purpose-built data center fabrics. Google introduced the **Virgo Net** architecture for TPU 8T:

- **Datacenter-Scale Fabric:** Scales across an entire physical building, providing a non-blocking fabric linking over 134,000 TPU chips. - **Global WAN Expansion:** Interconnecting multiple VirgoNet fabrics via Google's distributed global Wide Area Network (WAN) unlocks distributed training superclusters exceeding 1 million TPU chips. - **Latency Optimization:** Delivers 40% lower latency during large-scale distributed training phases.

Turning to agent serving, the speaker emphasizes that low-latency inference is "the real hero of the agentic era." Latency is no longer merely a user experience metric—it has transitioned into an explicit "cognitive constraint" governing agent utility. Because autonomous agents execute internal reasoning iterations, search loops, and tool calls, high latency rapidly compounds into catastrophic systemic delays and elevated iteration costs.

[12:18](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=738s) Purpose-Built Silicon for Low-Latency Inference

An autonomous agent is fundamentally only as reliable as the consistency of its underlying model execution and the hardware fabric powering it. To deliver the responsiveness required for real-world agentic workflows, Google designed latency-optimized silicon, introduced here as the TPU 8i. The processor incorporates architectural enhancements explicitly targeted at slashing inference latency, beginning with an exceptionally large SRAM capacity placed directly on-die. By positioning a substantial SRAM adjacent to the primary compute blocks, the chip can retain frequently accessed execution state—most notably the transformer key-value (KV) cache—directly beside the compute units, dramatically curbing the high latency penalty of repeatedly fetching memory from off-chip storage.

```mermaid flowchart TD accTitle: TPU Data Streaming Architecture accDescr: Diagram contrasting typical GPU cache movement with TPU shared vector memory and MXU streaming.

subgraph GPU_Flow ["Conventional GPU Pipeline"] G_L2["L2 Cache"] --> G_L1["L1 Cache"] G_L1 --> G_TC["Tensor Cores (Small Tiles)"] end

subgraph TPU_Flow ["TPU Low-Latency Pipeline"] T_IN["Large Inputs"] --> T_SVM["Shared Vector Memory"] T_SVM -->|"Streamlined Stream"| T_MXU["Matrix Multiplication Units (MXUs)"] T_SRAM["Large On-Chip SRAM (KV Cache)"] -.->|"Near-Compute Access"| T_MXU end ```

Beyond on-chip memory sizing, chip-to-chip bandwidth has been significantly scaled up. Inter-chip interconnect (ICI) bandwidth and chip-to-chip communication speeds were doubled, and Google integrated a specialized collective acceleration engine that achieves an additional 5× latency reduction for collective communication primitives. This dataflow design directly challenges conventional architectures:

> "When you really think about low latency inference, you have to be efficient in how you moving data from the memory onto the computational units. In a typical GPU floor, you have data moving from the L2 cache to the L1 to the tensor cores where operations are happening on the small tiles. But on the other hand with TPUs, large inputs are copied to the shared vector memory. And these large inputs are further streamlined to the matrix multiplication units."

By continuously saturating the matrix multiplication units (MXUs) with wide streams of data rather than bottlenecking on fragmented tile transfers, the hardware maintains exceptionally high Model FLOPs Utilization (MFU), ensuring compute engines stay constantly busy.

- **High-Capacity On-Chip SRAM**: Houses large KV caches adjacent to compute engines to eliminate off-chip memory fetch overheads. - **Doubled Interconnect Bandwidth**: 2× speed and bandwidth increase across direct chip-to-chip interfaces. - **Dedicated Collective Acceleration Engine**: Hardware acceleration reducing collective communication latency by 5×. - **Direct Vector-to-MXU Streaming**: Shared vector memory architecture designed to bypass hierarchical cache bottlenecks and maximize sustained compute utilization.

---

[14:25](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=865s) The Boardfly Hierarchical Interconnect Topology

Scaling inference clusters past a handful of chips demands rethinking network topologies beyond classical multi-dimensional tori. To scale clusters efficiently, Google introduced the "Boardfly" topology for TPU 8i, a non-torus hierarchical interconnect architecture capable of linking up to 1,152 chips within a single tightly coupled pod. The hierarchy is organized in three modular stages from the board level up to the optical switching layer:

```mermaid flowchart TD accTitle: Boardfly Hierarchical Interconnect Topology accDescr: Three-level hierarchy showing TPU boards, rack groups, and optical circuit switches linking 1152 chips.

subgraph Level1 ["Level 1: Board"] B_CHIPS["4 Fully Connected TPU Chips"] end

subgraph Level2 ["Level 2: Group / Rack"] R_BOARDS["8 Fully Connected Boards (32 Chips)"] end

subgraph Level3 ["Level 3: Pod Fabric"] OCS["Optical Circuit Switches (OCS)"] POD["36 Connected Groups (1,152 TPU Chips)"] end

B_CHIPS --> R_BOARDS R_BOARDS --> OCS OCS --> POD ```

At the base of the topology, four TPU chips are fully connected to one another on an individual board. Eight of these boards are then fully interconnected to construct a group—representing a physical rack on the datacenter floor housing 32 chips. Finally, 36 of these rack groups are bound together through Google's Optical Circuit Switches (OCS) to assemble the full 1,152-chip pod.

By structuring the network hierarchically rather than routing across a large multi-hop torus, the Boardfly topology achieves a 53% reduction in network diameter between any two arbitrary chips across the entire 1,152-node fabric. This drastic reduction in physical network traversals translates directly into a greater than 50% end-to-end latency improvement across distributed inference and collective communication jobs.

---

[15:32](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=932s) The Frontier: World Models, Physical AI, and Real-Time Reasoning

Looking ahead, the frontier of machine learning workloads is shifting decisively toward extreme low-latency inference. While language-based autonomous agents are already driving rapid progress, frontier agents increasingly demand far more inference cycles to execute deep reasoning steps and internal deliberation per second. Simultaneously, this trajectory intersects with two emergent technical paradigms: interactive world models and physical AI.

World models dynamically synthesize interactive, frame-by-frame simulations reflecting real-world physical dynamics in real time. This capability completely changes how autonomous agents are developed:

> "What this means is that you could actually take AI agents and have them operate and practice operating in this simulated real-time environment before you actually have them perform tasks in the real world where they're now more trained and more accurate."

In parallel, physical AI—spanning humanoid robotics and autonomous vehicles—demands reaction times operating strictly in the millisecond regime, making sustained, low-latency execution non-negotiable. The eventual convergence of world models and physical robotics allows autonomous systems to simulate thousands of lifetimes of physical actions safely inside high-throughput virtual worlds before deploying into physical hardware. Mugazambi concludes by noting Google's excitement for this trajectory, emphasizing that continuous custom hardware innovation across the TPU lineage is engineered specifically to satisfy the computational and latency requirements of this AI-native future.

## Notable quotes > [02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) Generalist and serial thinkers. — Sabastian Mugazambi

> [02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) Generalist, but they're parallel thinkers. — Sabastian Mugazambi

> [02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) Specialists. — Sabastian Mugazambi

> [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) the real hero of the agentic era. — Sabastian Mugazambi

> [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) cognitive constraint — Sabastian Mugazambi

> [12:18](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=738s) When you really think about low latency inference, you have to be efficient in how you moving data from the memory onto the computational units. In a typical GPU floor, you have data moving from the L2 cache to the L1 to the tensor cores where operations are happening on the small tiles. But on the other hand with TPUs, large inputs are copied to the shared vector memory. And these large inputs are further streamlined to the matrix multiplication units. — Sabastian Mugazambi

> [15:32](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=932s) What this means is that you could actually take AI agents and have them operate and practice operating in this simulated real-time environment before you actually have them perform tasks in the real world where they're now more trained and more accurate. — Sabastian Mugazambi

## Mentioned Google, Sabastian Mugazambi, @Scale Conference, TPU (Tensor Processing Unit), TPU v1, TPU v6e, TPU v6p, TPU 8T, TPU 8i, CPU, GPU, Matrix Multiply Unit (MXU), Vector Processing Unit (VPU), Activation Unit (AXU), Sparse Core, High Bandwidth Memory (HBM), Inter-Chip Interconnect (ICI), Optical Circuit Switch (OCS), VirgoNet, Boardfly topology

## Video description Architecting Infrastructure for the AI Native Future: Scaling Autonomous Agents on Google TPUs by Sabastian Mugazambi from Google

As the industry pivots toward an "AI Native" paradigm, the bottleneck for innovation has shifted from algorithmic design to the underlying infrastructure's ability to handle unprecedented scale and complexity. This session explores how Google TPU (Tensor Processing Unit) infrastructure serves as the catalyst for this transformation, specifically within the domains of large-scale Recommender Systems, MoEs, LLMs and the emerging era of Autonomous Agents.

We will delve into the architectural innovations of the latest TPU generations, demonstrating how their purpose-built design facilitates the massive throughput required for real-time recommendation engines and the high-speed inference necessary for agentic orchestration.

Want to learn more about the @Scale Conference? Visit our website here: atscaleconference.com

[Source](https://www.youtube.com/watch?v=MUxbhxzIMpQ)

中文简报

Google重构TPU芯片与Pod架构以加速AI智能体 推理延迟已取代算法设计成为自主智能体的核心约束，迫使硬件架构从纯粹的训练吞吐量转向定制片上存储与低直径网络架构。 随着自主AI智能体的工作负载转向持续执行、超大上下文窗口和实时推理循环，硬件基础设施必须发生根本性演进以支撑智能体认知。Google通过专用芯片应对这些需求——将专注于预训练和解码、主打吞吐量的TPU 8T集群，与具备海量片上SRAM和直接向量流式传输且针对延迟优化的TPU 8i芯片相结合。系统通过光路交换（OCS）、覆盖整个数据中心的VirgoNet网络架构以及将网络直径缩减50%以上的三层Boardfly互连拓扑，实现了纵向扩展与横向扩展。归根结底，这些低延迟架构协同设计为训练前沿交互式世界模型和实时物理AI奠定了关键基础。 Google概述了专为自主AI智能体设计的最新TPU硬件创新，重点介绍了面向吞吐量的TPU 8T和针对延迟优化的TPU 8i架构。 (2:27) TPU 8T通过光路交换机（OCS）使每个Pod可扩展至9,600颗芯片，实现121 exaflops的FP4算力，同时相比电子交换机降低了40%的网络功耗和30%的成本。 (8:41) TPU 8i在1,152颗芯片间引入了分层“Boardfly”互连拓扑，将网络直径缩减53%，并将分布式推理延迟降低50%以上。 (14:25) Google的VirgoNet网络架构可在单个数据中心建筑内连接超过134,000颗TPU芯片，并在跨全球WAN网络整合时扩展至超过100万颗TPU。 (10:45)

Reading summary

Google argues that autonomous AI agents have shifted the fundamental bottleneck of AI innovation from model design to physical infrastructure, requiring specialized accelerator silicon and dynamic optical fabrics optimized separately for high-throughput scaling and real-time inference. Infrastructure Demands of the Agentic Lifecycle (0:00) Unlike standard query-response models, autonomous agents require continuous execution across thousands of concurrent instances and deep working memory supported by expansive context windows. According to Google, this agentic workload places acute pressure across the entire machine learning lifecycle: pre-training Mixture-of-Experts architectures scaling up to 10 trillion parameters, post-training reinforcement learning with multi-step test-time reasoning loops, and low-latency orchestration for hundreds of thousands of active agents. Consequently, compute constraints have expanded from algorithmic optimization into the physical design of data center silicon, memory pathways, and network fabrics. Throughput Scaling via TPU Silicon and Optical Switching (5:02) To maximize training and prefill throughput, Google’s TPU 8T architecture integrates Matrix Multiply Units (MXUs) that process block scale quantization internally, bypassing Vector Processing Unit overhead. The silicon incorporates specialized Sparse Cores equipped with a dedicated LLM decoder engine, allowing dense prefill operations on Tensor Cores to overlap with auto-regressive decode execution. Direct High Bandwidth Memory (HBM) data transfers eliminate host CPU memory hops via TPU RDMA for chip-to-chip traffic and TPU Direct Storage, which streams long-context KV caches directly to external managed storage. At the rack level, 64-chip 3D torus cubes are networked using dynamic Optical Circuit Switches (OCS) rather than electronic packet switches, reducing power by 40% and costs by 30%. This fabric scales up to 9,600-chip pods delivering 121 exaflops of peak FP4 compute and over 2 petabytes of shared HBM. Across data centers, Google deploys the non-blocking VirgoNet fabric to connect up to 134,000 chips per building, with Wide Area Network extensions enabling superclusters exceeding 1 million chips while cutting distributed training latency by 40%. Inference Optimization and the Boardfly Topology (12:18) Google frames inference latency as an explicit cognitive constraint for autonomous agents, because internal deliberation loops, search routines, and tool calls compound single-hop delays into severe operational latency. The latency-optimized TPU 8i addresses this by placing high-capacity SRAM directly on-die to hold key-value caches adjacent to compute units, curbing off-chip memory fetch penalties. Rather than moving small tiles through conventional L2/L1 cache hierarchies, TPU 8i streams wide inputs from shared vector memory directly into MXUs, complemented by doubled interconnect bandwidth and a hardware collective acceleration engine that cuts collective communication latency by 5×. To link inference clusters without the latency penalty of large multi-hop tori, Google introduced the Boardfly hierarchical interconnect. Boardfly links four chips per board, connects eight boards into a 32-chip rack group, and couples 36 rack groups through optical switches into a 1,152-chip pod. This structure reduces network diameter between any two nodes by 53%, yielding a greater than 50% end-to-end latency improvement for distributed serving and collective tasks. Frontier Horizons: World Models and Physical AI (15:32) Looking beyond pure language agents, Google projects that frontier systems will increasingly rely on interactive world models and physical AI. World models generate interactive, frame-by-frame physical simulations in real time, enabling agents to practice multi-step tasks in synthetic environments before operating in physical settings. Because physical robotics and autonomous vehicles demand reaction times strictly in the millisecond range, Google maintains that continuously co-designing specialized accelerator silicon and optical networking remains mandatory to support high-frequency reasoning loops. Google contends that the future of agentic autonomy relies on specialized physical infrastructure co-designed for matrix math, large on-die SRAM caches, and reconfigurable optical switching. However, maintaining real-time execution will depend on infrastructure keeping pace as models expand toward multi-trillion parameter scales and millisecond-level physical simulations.

Detailed notes — zh

## Background 本内容由 @Scale 频道于 2026 年 6 月 19 日在 YouTube 平台发布，记录了来自 Google 的 Sabastian Mugazambi 在 @Scale 大会上的技术分享。演讲围绕“为 AI 原生未来构建基础设施”展开，重点剖析了 Google TPU 硬件与集群网络如何突破算力与互连瓶颈，以支撑十万亿级 MoE 模型、大规模推荐系统以及自主智能体（Autonomous Agents）的高速扩展。 ## TL;DR 演讲系统拆解了自主智能体在长上下文处理、多轮推理思考循环和海量并发编排下对高吞吐量与极低延迟提出的严苛要求。Google 针对性地演进了专为吞吐优化的 TPU 8T 与专为推理优化的 TPU 8i：在微架构上引入脉动阵列、片上超大 SRAM、嵌入 LLM 解码引擎的 SparseCore 以及块缩放乘法下沉；在系统与网络层通过 TPU Direct Storage、基于光路交换机（OCS）的 3D Torus 超算集群、Dragonfly 层次互联以及连接百万卡级别的 VirgoNet 全面消除通信跳数与内存墙。最后，演讲展望了世界模型与具身智能（Physical AI）时代的到来，指出毫秒级超低延迟推理正在从普通的用户体验指标蜕变为定义智能体认知深度的核心物理约束。 ## Atmosphere & style 整场演讲呈现出极为扎实、严谨且高密度的前沿硬核工程技术峰会风格。讲者以条理清晰、抽丝剥茧的节奏展开叙述，不谈空泛的概念炒作，而是直切芯片微架构、数据流管线与光互连拓扑的物理本质。台下听众仿佛置身于下一代超算机房的工程设计图纸前，能真切感受到大规模分布式计算在应对万亿参数模型与复杂智能体时面临的物理极限挑战，以及顶级基础架构工程所带来的震撼力量。 ## Bulletin - [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) AI 基础设施的核心瓶颈已从纯算法设计转向全生命周期算力协同，自主智能体的持续执行与强化学习多轮推理循环对系统吞吐和实时延迟提出了极高要求。 - [08:41](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=521s) Google 推出面向吞吐量的 TPU 8T 超算系统，通过 3D Torus 拓扑与 OCS 全光交换机互连 9,600 颗芯片，可提供 121 Exaflops 的 FP4 算力与 2 PB 以上共享 HBM 内存池。 - [14:25](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=865s) 新一代推理芯片放弃环形网格拓扑，采用 1,152 芯片 Dragonfly 分层互连架构，使网络直径缩减 53%，系统端到端通信延迟改善超 50%。 - [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) 超低延迟推理已成为智能体的“认知约束”，并成为物理 AI 与世界模型在毫秒级闭环交互中实现虚拟到现实训练迁移的核心先决条件。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) 自主智能体需依赖持续执行、海量上下文窗口与长流程推进，要求底层算力在长跨度任务中兼顾高吞吐与极低访存延迟。 - [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) 智能体全生命周期横跨预训练（10 万亿级 MoE）、RL 阶段的多轮推理循环以及推理阶段的高并发编排，协同演进的专用硬件成为创新瓶颈。 - [02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) 相较于串行通才 CPU 和并行通才 GPU，TPU 剥离了非必要指令，完全专注于神经网络基石的矩阵乘法加速以突破算力与互连墙。 - [05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) TPU 8T 将块缩放乘法直接下沉到 MXU 内部执行，消除量化对 VPU 的额外开销，同时通过脉动阵列维持高密度流水式计算。 - [05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) TPU 8T 的 SparseCore 引入了 LLM Decoder Engine，支持将计算密集的 Prefill 留在 Tensor Core，并将带宽敏感的 Decode 阶段卸载并行。 - [07:34](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=454s) 借助 TPU RDMA 与 Direct Storage 技术，系统可绕过 CPU 与主机内存直接传输 HBM 数据，支持将巨型长上下文 KV Cache 从托管存储直接调度换入。 - [08:41](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=521s) 单机架 64 芯片 3D Torus 立方体结合软件定义的 OCS 光路交换，可在毫秒级重构拓扑，相比传统电交换方案降低 30% 成本和 40% 能耗。 - [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) 数据中心级网络 VirgoNet 能在单 Fabric 内无阻塞连接超过 134,000 颗 TPU，配合全球 WAN 扩展支持超百万芯片集群并将端到端网络延迟降低 40%。 - [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) 在智能体长考与内部思考循环中，延迟已不再仅是用户体验指标，而是实质性决定单位时间内推理发散能力的“认知约束”。 - [12:18](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=738s) 新一代推理优化芯片将 KV Cache 放入超大片上 SRAM，配合双倍 ICI 带宽和专用集合通信加速引擎，使集合通信延迟缩减达 5 倍。 - [12:18](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=738s) 与传统 GPU 逐级缓存不同，TPU 采用数据直拷至共享向量内存并流水线直达 MXU 的架构，显著提高了模型 FLOP 利用率（MFU）。 - [14:25](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=865s) 面向规模化推理的 1,152 芯片 Pod 采用板级、机架级与 Pod 级三层 Dragonfly 互连，大幅减少网络跳数并使端到端通信延迟下降超过 50%。 - [15:32](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=932s) 世界模型支持在毫秒级生成可交互物理沙盒，智能体可在虚拟环境中完成深度强化学习训练后再部署至真实场景，极度依赖毫秒级的超低延迟推理支持。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) AI Agent 崛起与全生命周期的基础设施瓶颈 AI Agent（智能体）的爆发式增长正在彻底刷新 AI 基础设施的边界。与传统的单次交互不同，自主智能体的本质特性决定了它们需要不间断地“持续执行”（continuous execution）。想象一下成千上万个智能体同时在云端运转，这对底层的 AI 加速器提出了极高的吞吐量（throughput）诉求，而高吞吐量正是保障复杂智能体实现全天候、实时自主运行的基石。此外，由于智能体处理的是长跨度任务（long horizon tasks），海量上下文窗口（vast context windows）成了关键的记忆基石，它们负责为智能体提供深度的情境感知能力，并驱动复杂长流程的稳步落地。而在执行过程中，实时推理与低延迟的内存访问更是必不可少，只有这样才能撑起连续的实时 Agent 编排。 这种严苛的硬件诉求贯穿了智能体开发的全生命周期： - **预训练阶段**：驱动 Agent 的 MoE（Mixture of Experts，混合专家）架构模型参数量正一路狂飙至 10 万亿（10 trillion）级别，迫切需要专为超大吞吐量预训练定制的专用硬件。 - **训练后与强化学习阶段**：模型在正式上线服务之前，以采样推理为代表的强化学习（RL）技术日益普及，模型必须经历多轮推理循环（reasoning loops）来自我迭代，以提升准确率与效率。 - **推理服务阶段**：上线后面对成百上千个不断自我提示（prompting）、等待反馈并陷入持续编排的智能体，延迟与能效控制变成了生死线。 正因如此，行业创新的瓶颈已经从早期的“单纯依靠算法设计优化”，彻底转移到了“算法架构与底层 AI 基础设施协同演进”。 ```mermaid flowchart LR accTitle: Agentic Development Lifecycle Bottlenecks accDescr: The shift of innovation bottlenecks across pre-training, reinforcement learning, and serving stages. subgraph PreTrain ["预训练阶段"] A["10 万亿级 MoE 模型"] --> B["高吞吐专用硬件需求"] end subgraph TrainPost ["训练后优化"] C["RL 强化学习采样"] --> D["多轮推理思考循环"] end subgraph Serving ["推理服务阶段"] E["海量并发自主 Agent"] --> F["极低延迟交互与成本控制"] end PreTrain --> TrainPost --> Serving ``` --- ### [02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) 通用计算与专用矩阵硬件：Google TPU 的架构基因 在处理 AI/ML 任务的专用硬件选型上，业界的各类处理器有着截然不同的底层设计哲学： - **CPU**：属于典型的“通才”与“串行思考者”（serial thinkers）。它通用灵活，但没有针对特定计算进行特化，遇到高吞吐数据流任务时极易遭遇算力瓶颈。 - **GPU**：同样具备通才属性，但属于“并行思考者”（parallel thinkers）。它极度擅长图形渲染及大规模并行 AI 任务，能通过海量并行计算冲破单流吞吐的制约。 - **TPU（Tensor Processing Unit）**：彻底的“专才”（specialist）。它从设计之初就完全剥离与 AI 无关的指令，全心专注于矩阵乘法（Matrix Multiplications）这一构成现代神经网络基石的底层数学运算。 Google 自 2015 年发布第一代 TPU v1 以来，自研 TPU 已走过十余年历程，其演进目标始终明确：突破制约智能体系统的算力强度墙（arithmetic intensity wall）与互连墙（interconnect wall）。从通过 ICI 实现分布式共享内存、全面普及液冷技术，到引入 3D Torus（3D 环形拓扑）超算架构，Google 构建起了一整套支持大规模设备协同工作的超算体系。随着全新 TPU 架构的亮相，针对吞吐量优化的 **TPU v8e/8T** 与针对推理优化的 **TPU v8p/8i** 带来了跨越式的硬件迭代。重构加速器架构不仅涵盖硅片本身的指令单元设计，更一路延伸至芯片间互连（ICI）、数据中心网络扩展、垂直供电（vertical power delivery）与系统级液冷工程。 --- ### [05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) TPU 8T 硅片微架构：脉动阵列、低精度算力与稀疏核心解码引擎 进入专为吞吐量打造的 TPU 8T 硅片内部，其核心算力引擎 Tensor Core 由 MXU（矩阵乘法单元）、AXU 以及 VPU（向量处理单元）紧密配合构成。为了在不牺牲模型精度的前提下逼近性能极限，TPU 8T 全面支持从 BF16、FP8 到 FP4 的低精度计算与混合精度优化。 - **块缩放乘法（Block Scale Multiplication）下沉**：在量化矩阵乘法中，原本运行在 VPU 上的块缩放乘法被直接搬移到了 MXU 内部执行。这一改进直接消除了量化计算对 VPU 的额外开销（zero VPU overhead），让硬件利用率（MFU）与量化模型精度双双大幅提升。 - **脉动阵列（Systolic Arrays）流水线**：摒弃传统计算中反复在内存和寄存器之间“来回搬运”数据的模式，TPU 依靠脉动阵列建立起流水式数据流，让数据像血液一样持续穿透计算核心，确保 MXU 始终被高密度算力喂饱，绝不陷入“等待数据（stalled on memory）”的空转。 - **SparseCore（稀疏核心）与 LLM Decoder Engine**：过去 SparseCore 主要用于加速推荐系统等 Embeddings 密集型负载，通过将密集计算层留在 Tensor Core 上、将 Embeddings 层卸载到 SparseCore 上实现重叠并行。TPU 8T 则更进一步，在 SparseCore 中嵌入了全新的 **LLM Decoder Engine**。该引擎赋予了 SparseCore 恰到好处的计算强度，使得在运行大语言模型推理时，可以将计算密集的 Prefill（预填充）阶段交给 Tensor Core，而将内存带宽敏感的 Decode（逐字解码）阶段无缝重叠卸载到 SparseCore 上，达成极致的计算并行度。 --- ### [07:34](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=454s) 跨越内存墙：TPU RDMA 与 TPU Direct Storage 的长上下文直通 在智能体协同作业的场景中，加速器芯片间的数据交互效率直接决定了系统的实际吞吐。为此，系统引入了两项突破性的数据旁路传输技术： - **TPU RDMA（远程直接内存访问）**：允许两枚 TPU 的高带宽内存（HBM）通过网卡（NIC）直接对传数据，彻底砍掉了传统架构中经过 Host 主机内存中转的额外跃点（hops），大幅压低芯片间通信延迟。 - **TPU Direct Storage（直接存储直连）**：将这一理念延展到外挂托管存储设备。TPU 能够绕过 CPU 和主机系统，直接将数据从托管存储中批量加载到 TPU HBM。这一功能对于长上下文智能体而言至关重要：在运行极长上下文任务时，不断膨胀的 KV Cache 可以保存在成本较低的集中式托管存储中，一旦需要，TPU 便通过 Direct Storage 直取数据送入 HBM，让巨型上下文的快速调度与记忆换入换出成为现实。 --- ### [08:41](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=521s) Scale-Up 超算集群：3D Torus 拓扑与全光交换（OCS） 单芯片算力的提升必须配合 Scale-Up（垂直扩展）拓扑才能撑起万亿级模型。TPU 8T 沿用并强化了 Google 标志性的 3D Torus ICI 域网络架构： - **单机架 64 芯片 3D Torus 立方体**：在 3D Torus 拓扑中，每颗 TPU 芯片都与空间相邻的最近邻节点建立物理互连，并在边缘引入绕回链路（wrap-around links），构成一个包含 64 颗 TPU 芯片的紧凑计算“立方体（cube）”。在数据中心机房中，这一个完整立方体通常直接映射为一个物理机架（rack）。 - **9,600 芯片超算集群**：通过光纤将这些单机架立方体级联，构建起包含 9,600 颗 TPU 芯片的庞大超级计算机。该集群能提供高达 **121 Exaflops** 的 FP4 计算性能，坐拥超过 **2 PB** 的统一共享 HBM 内存池，以及超过 **19 Tbps** 的双向芯片间（ICI）互联带宽。 - **OCS（Optical Circuit Switching，光路交换机）自动化重构**：最精妙的黑科技在于机架间的 OCS 光交换。来自某一个立方体输入端的光纤信号，不需要任何人工拔插线缆，纯靠软件控制就能在毫秒级内自动偏转、重定向到任意立方体的输出端。这彻底消除了电交换机引入的带宽瓶颈，相比传统方案降低了 **30% 的系统成本** 和 **40% 的能源功耗**，并赋予拓扑极强的动态调整能力，可针对不同的模型并行（Model Parallelism）与数据并行（Data Parallelism）策略一键重组物理链路。 ```mermaid flowchart TD accTitle: 3D Torus and OCS Optical Interconnect accDescr: Interconnection from individual 64-chip 3D Torus racks to OCS and the unified 9,600-chip TPU supercomputer. subgraph PodUnit ["基础物理单元"] R1["单机架 Cube 1<br>(64 TPU 3D Torus)"] R2["单机架 Cube 2<br>(64 TPU 3D Torus)"] RN["单机架 Cube N<br>(64 TPU 3D Torus)"] end subgraph OpticalLayer ["全光动态交换层"] OCS["OCS 软件定义光路交换机<br>(零人工拔插 / 动态光纤重定向)"] end subgraph ClusterSystem ["超算系统"] Supercomputer["9,600 芯片 TPU 统一系统<br>121 Exaflops FP4 算力 / 2 PB+ HBM / 19 Tbps+ 双向 ICI"] end R1 <-->|"光纤直连"| OCS R2 <-->|"光纤直连"| OCS RN <-->|"光纤直连"| OCS OCS <--> ClusterSystem ``` --- ### [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) Scale-Out 百万级扩展：VirgoNet 与面向推理的“认知约束” 除了机架间的高速 Scale-Up 互联，大规模智能体基础设施更需要面向数据中心级 Scale-Out（水平扩展）的宏大视野。TPU 8T 引入了全新的数据中心网络架构——**VirgoNet**： - **单 Fabric 统领整座数据中心**：VirgoNet 构建起无阻塞（non-blocking）的统一数据中心 Fabric，单套网络架构能无缝连接单栋建筑内超过 **134,000 颗** TPU 芯片。 - **全球广域网构建百万卡集群**：在此基础上，通过分布式全球 WAN 网络将各数据中心 Fabric 进行超远距离互联，系统能够聚合组装出包含超过 **1,000,000 颗 TPU** 的超大规模分布式训练集群，将超大规模分布式训练的端到端网络延迟直降 **40%**。 在拥有如此算力后，演讲者抛出核心观点：Agent 时代真正的无名英雄是“超低延迟推理”。在智能体工作流中，延迟指标的内涵已经发生了质的演变——它“不再仅仅是一个衡量用户体验的普通性能指标，而是蜕变成了一种实质性的‘认知约束’（cognitive constraint）”。因为自主智能体往往要嵌入复杂的内部思维循环（reasoning loops）并执行深度长考，延迟的高低直接划定了智能体在单位时间内能够发散推理与调配能力的物理边界。 ### [12:18](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=738s) 面向超低延迟推理的 TPU 芯片架构创新 智能体（AI Agent）在现实世界中执行任务的准确度与表现，归根结底取决于其底层模型执行的稳定性以及支撑它的算力基础设施。为了彻底解决这一瓶颈，Google 专门推出了面向超低延迟推理量身定制的专用芯片——TPU v5e / 早期代际之后的新一代推理优化芯片（演讲中提及的 TPU 8i / 推理系列）。为了实现极致的低延迟，该芯片在微架构层面进行了全方位的底层重构。 芯片首先引入了超大容量的片上 SRAM。大容量 SRAM 使得高频访问的数据能够紧贴计算单元存放，最典型的应用便是 KV Cache（键值缓存）。将规模庞大的 KV Cache 直接放置在紧挨计算核心的位置，极大消除了原本需要频繁跨总线、跨芯片从高带宽内存（HBM）中拉取数据带来的严重延迟开销。与此同时，芯片间互联带宽与通信速率实现了直接翻倍。芯片内部还嵌入了专用的集合通信加速引擎（Collective Acceleration Engine），使集合通信的延迟进一步缩减了 5 倍之多。 在数据流向设计上，TPU 展现出了与传统 GPU 截然不同的设计哲学。在常规的 GPU 工作流中，数据层层流经 L2 Cache、L1 Cache，最终进入 Tensor Core 在细小的矩阵瓦片（Tiles）上执行运算；而在 TPU 的流式架构中，大规模输入数据被直接复制到共享向量内存（Shared Vector Memory），并由此高度流水线化地持续喂入矩阵乘法单元（MXU, Matrix Multiplication Units）。这种设计保障了 MXU 能够全天候处于满负荷的高速计算状态，大幅提升了模型 FLOP 利用率（MFU），消除了计算核心等待数据的空闲周期。 - **超大片上 SRAM**：直接紧邻算力单元存储 KV Cache，根除访存延迟开销。 - **互联带宽倍增**：芯片间互联（ICI）带宽与通信速度提升至原先的 2 倍。 - **集合通信加速**：引入专用通信加速硬件，集合通信延迟降低达 5 倍。 - **流水线化数据流**：数据由共享向量内存平滑直通 MXU，最大化提升 MFU（Model FLOPs Utilization）。 ```mermaid flowchart TD accTitle: GPU 与 TPU 数据传输与计算流水线对比 accDescr: 对比传统 GPU 的多级缓存到 Tensor Core 流程与 TPU 共享向量内存流水线直达 MXU 的架构差异 subgraph GPU_Flow["GPU 传统数据流"] direction TB G1["L2 Cache"] --> G2["L1 Cache"] G2 --> G3["Tensor Cores (小 Tile 计算)"] end subgraph TPU_Flow["TPU 流式低延迟数据流"] direction TB T1["超大片上 SRAM<br>(存放 KV Cache)"] --> T2["共享向量内存<br>(Shared Vector Memory)"] T2 --> T3["流水线直供 MXU<br>(矩阵乘法单元满载计算)"] end ``` --- ### [14:25](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=865s) 突破传统拓扑：1,152 芯片规模的 Dragonfly 层次互联 当单芯片性能达到极致后，芯片间的互连拓扑结构成为了系统级延迟的核心掣肘。针对规模化集群，新一代推理芯片放弃了传统 TPU 标志性的环形网格拓扑（Torus topology），创新性地采用了分层的 Dragonfly（Board-fly）网络拓扑架构，将单一 Pod 内的 1,152 颗 TPU 芯片高度互联起来。 这种 Dragonfly 互联架构呈现出清晰的三级层次结构： 1. **板级互联（Board Level）**：每块 TPU 算力板上搭载 4 颗全互联的 TPU 芯片； 2. **机架级互联（Rack/Group Level）**：由 8 块算力板（共 32 颗芯片）在内部实现全互联，构成一个机架单元（Group）； 3. **集群级光交换（Pod/OCS Level）**：数据中心内将 36 个这样的机架单元通过光路交换机（OCS, Optical Circuit Switch）紧密连接，最终汇聚为拥有 1,152 颗芯片的庞大算力 Pod。 这种新型拓扑结构带来了显著的通信效率飞跃。相比于传统拓扑，该网络架构使 1,152 芯片 Pod 内任意两颗芯片之间的网络直径（Network Diameter）缩减了 53%。网络跳数的断崖式下降直接带来了系统端到端通信延迟超过 50% 的巨大改善，为大规模模型分布式推理的极致响应速度铺平了道路。 ```mermaid flowchart LR accTitle: 1152 芯片 Dragonfly 分层互联拓扑 accDescr: 展示从4颗芯片板卡到8块板机架再到36个机架通过OCS连接的1152芯片集群分层架构 subgraph Board["算力板 (Board)"] B1["4 颗芯片全互联"] end subgraph Rack["机架组 (Rack / Group)"] R1["8 块算力板全互联<br>(32 颗 TPU 芯片)"] end subgraph Pod["算力集群 (Pod)"] P1["36 个机架组<br>(1,152 颗 TPU 芯片)"] end Board --> Rack Rack -->|"通过 OCS (光路交换机) 互联"| Pod ``` --- ### [15:32](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=932s) 前沿 AI 的下半场：低延迟推理、世界模型与具身智能 展望未来，AI 模型的竞争前沿正加速从离线训练向“超低延迟实时推理”全面转移。大语言模型驱动的自主智能体（LLM Agents）正展现出颠覆性力量，但为了执行更复杂、更严谨的多步推理与深度思考，智能体每秒所需的推理计算周期（Inference Cycles）正呈指数级攀升。 与此同时，以实时环境模拟为核心的“世界模型”（World Models）正迅猛崛起。世界模型能够在毫秒级别实时生成高保真、可交互的物理现实模拟环境，并支持逐帧级别的交互反馈。这一突破为 AI 智能体的进化提供了颠覆性的训练范式：智能体无需一开始就在不可控的现实世界中试错，而是可以先被投放到世界模型生成的实时数字沙盒中进行成千上万次实操练习，在虚拟环境中打磨出极高的决策鲁棒性与精度后，再部署至真实场景。 这一趋势与“具身智能/物理 AI”（Physical AI）的爆发形成了强大的技术共振。机器人、自动驾驶等物理系统对控制闭环的时延要求极其苛刻，必须在毫秒（Millisecond）级的极窄时间窗口内做出反应，这让底层计算设施的超低延迟推理能力成为了生死攸关的先决条件。在世界模型与具身智能的交汇点上，机器人可以先在世界模型搭建的实时模拟空间中练习交互动作，再安全高效地步入现实世界执行物理任务。面对前沿 AI 时代的澎湃浪潮，Google 正在持续重构其底层 AI 基础设施与 TPU 矩阵，为下一代智能原生应用奠定坚实的算力底座。 ## Notable quotes > [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) 自主智能体的本质特性决定了它们需要不间断地“持续执行” — Sabastian Mugazambi > [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) 驱动 Agent 的 MoE（Mixture of Experts，混合专家）架构模型参数量正一路狂飙至 10 万亿级别 — Sabastian Mugazambi > [00:00](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=0s) 行业创新的瓶颈已经从早期的“单纯依靠算法设计优化”，彻底转移到了“算法架构与底层 AI 基础设施协同演进” — Sabastian Mugazambi > [02:27](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=147s) CPU：属于典型的“通才”与“串行思考者”……GPU：同样具备通才属性，但属于“并行思考者”……TPU：彻底的“专才” — Sabastian Mugazambi > [05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) 依靠脉动阵列建立起流水式数据流，让数据像血液一样持续穿透计算核心，确保 MXU 始终被高密度算力喂饱，绝不陷入“等待数据”的空转 — Sabastian Mugazambi > [05:02](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=302s) 将计算密集的 Prefill（预填充）阶段交给 Tensor Core，而将内存带宽敏感的 Decode（逐字解码）阶段无缝重叠卸载到 SparseCore 上 — Sabastian Mugazambi > [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) 通过分布式全球 WAN 网络将各数据中心 Fabric 进行超远距离互联，系统能够聚合组装出包含超过 1,000,000 颗 TPU 的超大规模分布式训练集群 — Sabastian Mugazambi > [10:45](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=645s) 不再仅仅是一个衡量用户体验的普通性能指标，而是蜕变成了一种实质性的‘认知约束’ — Sabastian Mugazambi > [14:25](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=865s) 该网络架构使 1,152 芯片 Pod 内任意两颗芯片之间的网络直径缩减了 53% — Sabastian Mugazambi > [15:32](https://www.youtube.com/watch?v=MUxbhxzIMpQ&t=932s) 智能体无需一开始就在不可控的现实世界中试错，而是可以先被投放到世界模型生成的实时数字沙盒中进行成千上万次实操练习 — Sabastian Mugazambi ## Mentioned Google, Sabastian Mugazambi, TPU (Tensor Processing Unit), TPU v1, TPU v5e, TPU v8e/8T, TPU v8p/8i, Tensor Core, MXU (Matrix Multiplication Unit), AXU, VPU (Vector Processing Unit), SparseCore, LLM Decoder Engine, TPU RDMA, TPU Direct Storage, 3D Torus, OCS (Optical Circuit Switching), VirgoNet, Dragonfly (Board-fly), MoE (Mixture of Experts), HBM, KV Cache, SRAM ## Video description Architecting Infrastructure for the AI Native Future: Scaling Autonomous Agents on Google TPUs by Sabastian Mugazambi from Google As the industry pivots toward an "AI Native" paradigm, the bottleneck for innovation has shifted from algorithmic design to the underlying infrastructure's ability to handle unprecedented scale and complexity. This session explores how Google TPU (Tensor Processing Unit) infrastructure serves as the catalyst for this transformation, specifically within the domains of large-scale Recommender Systems, MoEs, LLMs and the emerging era of Autonomous Agents. We will delve into the architectural innovations of the latest TPU generations, demonstrating how their purpose-built design facilitates the massive throughput required for real-time recommendation engines and the high-speed inference necessary for agentic orchestration. Want to learn more about the @Scale Conference? Visit our website here: atscaleconference.com [Source](https://www.youtube.com/watch?v=MUxbhxzIMpQ)

Source quotes

[0:00] Sabastian Mugazambi: 自主智能体的本质特性决定了它们需要不间断地“持续执行” [0:00] Sabastian Mugazambi: 驱动 Agent 的 MoE（Mixture of Experts，混合专家）架构模型参数量正一路狂飙至 10 万亿级别 [0:00] Sabastian Mugazambi: 行业创新的瓶颈已经从早期的“单纯依靠算法设计优化”，彻底转移到了“算法架构与底层 AI 基础设施协同演进” [2:27] Sabastian Mugazambi: CPU：属于典型的“通才”与“串行思考者”……GPU：同样具备通才属性，但属于“并行思考者”……TPU：彻底的“专才” [5:02] Sabastian Mugazambi: 依靠脉动阵列建立起流水式数据流，让数据像血液一样持续穿透计算核心，确保 MXU 始终被高密度算力喂饱，绝不陷入“等待数据”的空转 [5:02] Sabastian Mugazambi: 将计算密集的 Prefill（预填充）阶段交给 Tensor Core，而将内存带宽敏感的 Decode（逐字解码）阶段无缝重叠卸载到 SparseCore 上 [10:45] Sabastian Mugazambi: 通过分布式全球 WAN 网络将各数据中心 Fabric 进行超远距离互联，系统能够聚合组装出包含超过 1,000,000 颗 TPU 的超大规模分布式训练集群 [10:45] Sabastian Mugazambi: 不再仅仅是一个衡量用户体验的普通性能指标，而是蜕变成了一种实质性的‘认知约束’ [14:25] Sabastian Mugazambi: 该网络架构使 1,152 芯片 Pod 内任意两颗芯片之间的网络直径缩减了 53% [15:32] Sabastian Mugazambi: 智能体无需一开始就在不可控的现实世界中试错，而是可以先被投放到世界模型生成的实时数字沙盒中进行成千上万次实操练习

Source quotes

[2:27] Sabastian Mugazambi: Generalist and serial thinkers. [2:27] Sabastian Mugazambi: Generalist, but they're parallel thinkers. [2:27] Sabastian Mugazambi: Specialists. [10:45] Sabastian Mugazambi: the real hero of the agentic era. [10:45] Sabastian Mugazambi: cognitive constraint [12:18] Sabastian Mugazambi: When you really think about low latency inference, you have to be efficient in how you moving data from the memory onto the computational units. In a typical GPU floor, you have data moving from the L2 cache to the L1 to the tensor cores where operations are happening on the small tiles. But on the other hand with TPUs, large inputs are copied to the shared vector memory. And these large inputs are further streamlined to the matrix multiplication units. [15:32] Sabastian Mugazambi: What this means is that you could actually take AI agents and have them operate and practice operating in this simulated real-time environment before you actually have them perform tasks in the real world where they're now more trained and more accurate.

Original transcript

[0:00] The rise of AI agents is pushing the [0:03] boundaries of AI infrastructure. [0:06] With the autonomous nature of these [0:07] agents, they require continuous [0:10] execution. [0:12] Think about thousands of agents that [0:14] require massive throughput in AI [0:16] accelerators, which is the foundational [0:18] enabler for continuous real-time [0:21] autonomy for complex AI agents. [0:23] Additionally, because these AI agents [0:26] are operating on long horizon tasks, it [0:29] becomes critical to enable vast context [0:32] windows. [0:33] These are critical memory workings that [0:35] grounds AI agents, enabling deep [0:38] situational awareness and the execution [0:41] of complex long horizon tasks. [0:44] Last but not least, there is a real-time [0:46] reasoning component to AI agents. [0:49] They require AI infrastructure that [0:51] provides high-speed inference and low [0:53] latency memory access that is necessary [0:56] for continuous real-time agentic [0:59] orchestration. [1:01] These demands, we're seeing them across [1:03] the entire model development life cycle [1:06] of the agentic era. Starting with [1:08] pre-training the mixture of experts [1:10] models that drive these AI agents. [1:14] Models are rising up to the 10 trillion [1:16] parameter sizes, and this requires [1:20] optimized hardware that is specialized [1:23] for high throughput pre-training. [1:26] And even before you launch the model for [1:28] serving, we're seeing the rise of [1:30] reinforcement learning techniques like [1:32] sampling, where the model goes through [1:34] several iterations of reasoning loops to [1:38] make it more efficient and more [1:40] accurate. [1:41] And then you serve hundreds of thousands [1:43] of AI agents that are continuously [1:45] prompting and waiting for responses, [1:48] requiring [1:49] efficiency in latency. [1:53] With these demands across this entire AI [1:56] agentic development [1:58] life cycle, we're seeing that the [2:00] bottleneck for innovation has shifted [2:03] from just being algorithmic design [2:06] efficiency to being both algorithmic [2:08] design and the underlying AI [2:11] infrastructure. [2:14] That's why today, I would like to talk [2:16] about architecting infrastructure for [2:18] the AI native future, specifically [2:21] focusing on how you can scale autonomous [2:24] agents on Google's TPUs. [2:27] When you think about specialized [2:28] hardware for AI ML workloads, [2:31] you've got CPUs that are generalist and [2:34] serial thinkers. [2:36] So, while they're versatile, they're not [2:38] very specialized for one specific tasks. [2:41] They're very flexible, but they're [2:43] bottlenecked by high throughput tasks. [2:47] And then on the other hand, you've got [2:48] GPUs. [2:50] They're also generalist, but they're [2:51] parallel thinkers. They excel at [2:54] parallel tasks, like graphics and AI ML [2:57] tasks. [2:59] They use massive parallelism to prevent [3:02] any [3:03] throughput bottlenecks. [3:05] Now, TPUs on the other hand, they're [3:07] specialist. They're specifically [3:10] specialized for the mathematics that [3:13] forms and powers the foundation of AI [3:16] models, specifically matrix [3:19] multiplications. [3:20] TPUs are ideal for high-performance [3:23] machine learning workloads. [3:25] At Google, we've been designing and [3:27] engineering Tensor Processing Units for [3:29] over a decade to specifically break [3:32] through the arithmetic intensity and the [3:34] interconnect walls that severely [3:37] restrict agentic systems. [3:39] Over the past 10 years, we've been [3:41] innovating across the entire TPU road [3:43] map since we launched our first TPU v1 [3:46] in 2015. [3:48] We've added technologies like [3:50] distributed shared memory across ICI [3:53] to efficient cooling with liquid cooling [3:56] to introduction of 3D Taurus [3:58] architecture and topologies so that we [4:00] build massive supercomputers that all [4:03] work together coherently. [4:05] But what I'm really excited about is [4:07] some of the technologies that we have [4:09] added to TPU 8T and TPU 8I, which is the [4:13] newly announced and newly launched TPUs. [4:16] So for the entirety of my talk today, [4:18] I'll be focusing some of those key [4:19] technologies and advancements that we [4:21] have added to TPU 8T and TPU 8I. [4:26] Now reimagining the accelerator [4:28] architecture [4:30] starts from the specialized TPU silicon [4:32] features themselves, extends to how you [4:35] innovate around the ICI interchip [4:38] interconnect topologies, [4:40] all the way to how you scale out data [4:42] center networking capabilities. [4:45] However, the story doesn't end there. [4:47] You really need to think about how you [4:49] deliver power to this TPU chips [4:51] efficiently, and with the TPUs is [4:53] vertical power delivery, and how you [4:55] efficiently cool the systems with [4:57] technologies like liquid cool cooling on [5:00] TPUs. [5:02] Now let's zoom in and start with TPU 8T, [5:06] which is the throughput optimized [5:08] silicon. With TPU 8T, you have a tensor [5:11] core that has a bunch of MXUs, AXUs, and [5:15] VPUs, which are computational units that [5:17] are optimized for matrix multiplication [5:20] that form the foundation of AI [5:22] computation. We support low precision [5:25] arithmetic from FP4, FP8, BF16, and also [5:29] support for mixed precision [5:31] optimizations as well. One of the things [5:33] that we've done to TPU 8T is move things [5:36] like block scale multiplication inside [5:39] of the MXUs from the VPUs. This allows [5:42] for zero VPU overhead for quantized [5:46] matmuls, and allows for better model [5:49] accuracy and better model flop [5:52] utilization. Another technology for TPUs [5:54] is systolic arrays. Rather than going [5:57] back and forth between memory and the [5:59] computational units, you actually open [6:02] up pipelines that enable the constant [6:05] flow of data from the memory into the [6:08] computational units, which ensures that [6:10] this MXUs constantly are fed with data [6:14] and continue computation and are not [6:17] waiting on data. [6:19] Additionally, we interconnect all these [6:21] chips to function as single unit in an [6:23] ICA domain using a proprietary ICI [6:27] interchip interconnect technology. [6:29] Another key technology for TPUs is [6:32] sparse core. These are specialized cores [6:34] that we've added onto the silicon. [6:37] And prior to TPU AT, [6:39] you could utilize sparse cores to [6:42] accelerate embeddings heavy workloads by [6:44] essentially overlapping the dense layer [6:48] onto the [6:50] as it is computationally heavy with the [6:52] embeddings layer offloaded onto the [6:55] sparse core. You could accelerate [6:57] embeddings heavy workloads. In TPU AT, [7:01] we've gone a step further and added what [7:03] we call an LLM decoder engine. [7:06] An LLM decoder engine enables [7:10] sparse cores to have the right amount of [7:13] arithmetic intensity such that we can [7:15] utilize the same technique we're doing [7:17] of [7:18] on embeddings models, [7:20] utilize that on LLMs when it comes to [7:23] overlapping prefill on the tensor cores [7:27] and decode on the sparse cores that are [7:29] supported by the LLM decoder engine. [7:34] When it comes to connectivity between [7:36] the TPUs, you have to think about how [7:38] you're transferring data between TPUs [7:41] that are coherently working together as [7:43] a single unit. That's where we introduce [7:46] TPU RDMA and TPU direct storage. [7:50] With TPU RDMA, you can directly transfer [7:53] data between two TPU HBMs via the [7:56] network interface cards, essentially [7:59] eliminating hops via the host memory. [8:02] This improves data transfer latency [8:06] drastically. [8:07] You can even extend that technology to [8:09] TPU direct storage, where essentially [8:12] you can copy data directly from managed [8:15] storage onto the HBM of the TPU chips. [8:18] This really works very, very well for [8:21] long context KV cache storing onto [8:24] managed storage, but yet utilizing that [8:27] in the TPU chips. [8:29] Because now you can directly access [8:31] those long context KV cache in managed [8:33] storage [8:35] by copying the data from managed storage [8:37] onto directly onto the TPU HBM. [8:41] And then when you go beyond data [8:43] transfer, and when you think about TPU [8:46] chips working coherently together, you [8:48] have to think about the scale-up [8:50] technology for these TPU accelerators. [8:53] That's where TPUs excel with the [8:56] throughput-optimized 3D torus ICI domain [8:59] technology. With 3D torus topology, [9:02] every single TPU chip is connected to [9:04] its nearest neighbor with wrap-around [9:07] links that form a cube of 64 TPU chips [9:11] in this 3D torus topology. Think of a [9:13] cube of TPU chips being a single rack [9:16] that is actually sitting on the data [9:17] center floor. [9:19] We then connect all of the single racks [9:21] with optical circuit switching using [9:24] optical fibers. [9:25] These interconnected racks form a TPU [9:29] supercomputer of 9,600 [9:32] TPU H chips. [9:34] This deliver 121 exaflops of FP4 [9:38] computes [9:40] with over 2 petabytes of shared HBM. [9:43] And over 19 terabits per second [9:46] bidirectional ICI bandwidth. [9:50] And remember I talked about optical [9:51] circuit switching. [9:53] This is such a unique and really cool [9:55] technology that we have added to TPUs. [9:58] So essentially, input fiber that is [10:02] coming from one cube [10:04] can be redirected without any human [10:07] interaction or automatically in [10:10] software. [10:12] That input can be redirected to either [10:14] cube Z or cube Y's output fiber without [10:18] any human interaction. [10:20] All happening within the software. [10:22] This eliminates any bandwidth [10:24] bottlenecks and delivers 30% lower cost [10:28] and 40% lower power needs. [10:32] With the topology flexibility and [10:34] diversity OCS delivers, [10:37] we can effectively support model and [10:39] data parallelism, [10:41] which is key to accelerating AI [10:44] workloads. [10:45] The story doesn't end with scale up. It [10:48] also extends to scale out. And you have [10:50] to innovate in how you scale out AI [10:54] accelerators into hundreds of thousands [10:56] of accelerators all working together on [10:58] a single workload. With that, we're [11:01] introducing Virgo Net architecture to [11:03] TPU AT. [11:05] With Virgo networking, a single fabric [11:08] scales to the entire data center [11:10] building. [11:12] Virgo Net architecture allows for the [11:13] connectivity of over 134,000 [11:16] TPU chips in a non-blocking single [11:19] fabric. [11:20] And if you connect these fabrics using [11:23] distributed global WAN, you can create [11:26] training clusters of over 1 million TPUs [11:30] across all of these fabrics. [11:33] This delivers a 40% lower latency when [11:37] it comes to distributed large-scale [11:39] training. [11:41] Now, the real hero of the agentic era is [11:44] lower latency inference. [11:46] If you think about latency, [11:49] it is transitioned from just being a [11:51] user experience or just another metric [11:54] into being really a cognitive [11:56] constraint. [11:58] In agentic workflows, latency it's not [12:01] just a user experience metric. It is the [12:03] foundational boundary that essentially [12:05] detects how efficient agents can operate [12:09] and can serve. [12:10] Coupled with that, there's the cost [12:12] iteration. [12:14] Agentic use cases often require multiple [12:16] internal reasoning loops to make the [12:18] agents more accurate and perform the [12:20] task more accurately in real life. [12:24] And last but not least, an agent is only [12:26] as reliable as the consistency of the [12:28] underlying model execution and the [12:30] infrastructure that is enabling it. [12:33] That's why we're introducing TPU AI, [12:36] which is the latency-optimized silicon. [12:39] Now, in addition to all the other good [12:40] stuff I've already mentioned in the TPU [12:42] 8i, we've done some amazing stuff to TPU [12:45] 8i to make it highly optimized for lower [12:48] latency inference. [12:50] Starting with a very large SRAM we've [12:52] added. [12:53] With a very large SRAM, you can keep [12:56] constantly fetch data very close to the [12:59] actual computational units. For example, [13:02] KV cache. [13:03] With a large KV cache that is stored [13:06] right next to the computational units, [13:07] you essentially reduce the overhead of [13:11] having to go fetch this KV cache from [13:13] memory. [13:14] We've also doubled the interchip [13:16] interconnect bandwidth and the speed for [13:19] chip-to-chip communication. [13:21] In addition to that, we have also added [13:24] a specialized Pascal-based collective [13:26] acceleration engine that will deliver an [13:29] additional 5x reduction in latency for [13:32] collective communication. [13:34] When you really think about low latency [13:36] inference, you have to be efficient in [13:38] how you moving data from the memory onto [13:41] the computational units. [13:43] In a typical GPU floor, you have data [13:46] moving from the L2 cache to the L1 to [13:49] the tensor cores where operations are [13:51] happening on the small tiles. [13:54] But on the other hand with TPUs, large [13:56] inputs are copied to the shared vector [13:59] memory. [14:00] And these large inputs are further [14:02] streamlined to the matrix multiplication [14:05] units. [14:07] And this allows for constantly feeding [14:10] these computational units with data and [14:13] allows for high model flop utilization [14:16] and constant and keeps these MXUs [14:18] constantly busy doing computation, which [14:21] is very, very efficient. [14:25] Even when it comes to the topology [14:27] itself of interconnecting these TPU [14:29] chips, you have to innovate. [14:31] That's why we introduced the board fly [14:33] topology to TPU 8 i. [14:36] This is a non-torus topology that [14:38] interconnects up to 1,152 [14:41] TPU 8 i chips. [14:44] The way to think about board fly [14:45] topology is that it is hierarchical in [14:48] nature. [14:49] It starts with four fully connected TPU [14:52] chips on each TPU board and eight of [14:55] those boards are fully connected into a [14:58] group, which is essentially the rack on [15:01] the data center floor. [15:02] And 36 of those groups are then also [15:05] connected via OCS [15:08] to form this pod of 1,152 [15:11] TPU chips. [15:13] This particular topology enables a 53% [15:17] reduction in network diameter between [15:19] any two TPU chips within the 1,152 [15:24] TPU port. This further enables a over [15:27] 50% improvement in latency. [15:32] Now at Google, we're always thinking [15:35] about the next frontier of AI models. [15:39] And we believe that the frontier is [15:41] going to be low latency inference. [15:43] We're already seeing the raw LLM AI [15:46] agents are playing in AI advancement. [15:50] We believe that LLM agents will continue [15:53] to require more inference cycles [15:56] to perform this deeper reasoning per [15:59] second. [16:00] But in addition to that, we see this [16:02] rise of world models. These are models [16:05] that can essentially in real time create [16:09] simulated environments of the real [16:11] workings of the real world and allows [16:14] for interactivity at a frame-by-frame [16:16] basis. [16:18] What this means is that you could [16:19] actually take AI agents and have them [16:23] operate and practice operating [16:27] in this simulated real-time environment [16:31] before you actually have them perform [16:33] tasks in the real world where they're [16:35] now more trained and more accurate. [16:39] And then on the other hand, we see the [16:40] rise of physical AI, specifically [16:43] robotics and autonomous systems that [16:45] require reaction in the milliseconds [16:48] regime, which makes low latency [16:51] inference key. [16:54] And even with the intersection of [16:56] physical AI and world models, where [16:58] we're seeing robots and autonomous [17:00] systems first practice how to act in the [17:03] real world, essentially practice that in [17:06] real-time simulated environments [17:09] developed by world models. [17:12] We're super excited at Google in terms [17:15] of the trends of AI development and [17:18] where AI is going, and we're super [17:20] excited to continue innovating our AI [17:23] infrastructure and TPUs to meet these [17:26] demands and needs for this frontier AI. [17:30] Be sure to check out more information on [17:34] the TPU website, and thank you.