Google Splits TPU Line as Mega AI Clusters Suffer Hourly Faults

谷歌分拆TPU产品线：超大规模AI集群每小时现故障 理论FLOPS已沦为虚荣指标，因为前沿集群每小时都会出现多次组件故障，迫使基础设施设计将实际交付的有效吞吐量（goodput）和电网功率限制置于芯片峰值速度之上。 Amin Vahdat阐释了为何在10万卡加速器规模下，原始FLOPS只是一个不切实际的虚荣指标，并主张采用“有效吞吐量”（goodput）——即在硬件和光纤持续出现故障的情况下实际交付的有用计算。他详细说明了谷歌将TPU产品线分拆为8i（推理）和8t（训练）的决策，并介绍了DeepMind研究人员与硬件工程师如何在流片前动态协同设计芯片。讨论还分析了长周期智能体工作负载如何对主机CPU和存储构成重压、光线路交换机如何在毫秒级内重新路由流量，以及为何并网电力的可用性仍是超大规模数据中心的最终硬性约束。 在10万卡加速器规模下，每小时都会发生多次硬件故障，这使得理论峰值FLOPS相较于有效吞吐量（goodput）成了虚荣指标——后者指在真实故障条件下、按实际自然时间完成的有效计算。 (5:30) 谷歌首次将其TPU产品线分拆为专用的训练（8t）和推理（8i）ASIC，原因在于推理需求已激增至占整个集群计算需求的30%到60%，这使得在保留基础可替代性的同时进行针对性专业化设计变得合理。 (15:32) 长周期智能体工作负载正推动非加速器需求暴涨，将大量的解析、推理和上下文检索开销转移至主机CPU、DRAM和存储系统上。 (34:16) 电力可用性已超越算力技术障碍，成为数据中心的首要硬性约束，促使谷歌倾向于电网并网以及与公用事业公司共同投资，而非采用需要昂贵1+1冗余的“表后”自发电方案。 (42:35)

Amin Vahdat, Google's AI infrastructure lead, argues that managing massive capital outlays requires shifting focus from theoretical peak FLOPS to usable goodput per watt across tightly co-designed hardware, network, and power layers. Goodput and Resilience at Hyperscale (5:30) Vahdat rejects peak FLOPS as a vanity metric, pointing out that theoretical arithmetic capacity in a lab ignores practical bottlenecks like high-bandwidth memory, host pipelines, and networking fabrics. Instead, Google optimizes for goodput: the volume of useful computation delivered to actual workloads divided by wall-clock time. Because frontier training and low-latency inference are tightly synchronized across 10,000 to 100,000 accelerators, a single dropped packet or hung chip can bring the entire cluster to a standstill. Recovering requires identifying the faulty node, rewinding state to a checkpoint, reallocating spares, and restarting, which turns raw throughput without successful completion into wasted effort. At a scale of 100,000 chips, continuous failure is treated as an inevitable physical reality. Running billions of microscopic components at thermal limits triggers anomalies multiple times per hour across an ever-shifting long tail of causes, from optical signal drift to compiler and runtime synchronization bugs. Google addresses this fragility through automated telemetry and dynamic re-routing, including optical circuit switches using micro-electro-mechanical systems (MEMS) mirrors. These switches bypass faulty racks by redirecting optical paths to standby hardware within milliseconds, keeping systemic goodput stable without requiring technicians to manually rewire connections. The Economics of Specialization and Co-Design (20:00) Hardware specialization delivers superior speed and energy efficiency, but it risks obsolescence if model architectures shift quickly. Vahdat highlights this dynamic in Google's custom silicon roadmap. In 2013, launching the original TPU was a contrarian bet against general-purpose CPUs riding Moore's Law. By the TPU v8 era, inference demand expanded from a negligible fraction to an estimated 30% to 60% of Google's compute fleet footprint. This scale justified splitting development into two purpose-built ASICs: the TPU 8i for inference and the TPU 8t for training. To hedge against operational rigidity over a multi-year hardware lifespan, both chips retain baseline fungibility so that surplus capacity on one variant can absorb workloads from the other. While designing generalized, decoupled infrastructure across multi-cloud environments provides portability, Vahdat contends it incurs severe efficiency penalties due to abstraction mismatches. Google instead pursues tight vertical co-design between hardware engineers and Google DeepMind researchers. Because model researchers and silicon architects interact continuously, algorithmic shifts can trigger mid-flight modifications to hardware definitions or brief delays in tape-out schedules to capture compounding gains. At the same time, Vahdat notes the TPU's foundational instruction set—relying on matrix multiply units, sparse cores, and custom interconnects—has stayed durable across algorithm generations, with modern Gemini models recursively assisting in the design of future silicon architectures. Power Limits and Modular Infrastructure Trajectories (42:35) Electrical power represents the binding constraint for hyperscale data centers, characterized by long utility lead times that contrast with rapid silicon turnover. To secure hundreds of megawatts up to gigawatt-scale supplies, Google works directly with regulated utilities, funding dedicated substations and transmission lines to avoid inflating residential electric rates. Facilities are intentionally kept grid-connected rather than isolated off-grid; relying entirely behind the meter would require costly 1+1 generation redundancy to guarantee high availability. This interconnected approach also enables bidirectional load shedding, allowing facilities to reduce consumption or export on-site power during extreme regional heatwaves. Looking toward future supercomputers in 2036, Vahdat forecasts that data center density will increasingly rely on pre-assembled, modular multi-megawatt racks containing 576, 1,152, or more accelerator chips. Rather than managing complex optical cabling webs on-site, facilities will deploy factory-integrated units requiring only power, liquid cooling, and consolidated fiber hookups. On more speculative horizons, orbital data centers offer continuous solar availability and higher solar irradiance outside atmospheric filtering, but Vahdat cautions that space-based compute faces distinct physical obstacles: heat dissipation must rely entirely on radiative cooling rather than airflow, maintenance demands autonomous robotics, and inter-node networking requires active laser steering over free space. While the race to build frontier infrastructure drives multi-megawatt facilities and specialized silicon architectures, long-term expansion remains governed by electrical grid constraints, thermal limits, and the economic necessity of supporting diverse workloads across decades-long real estate lifecycles.

## Background 本期节目由红杉资本（Sequoia Capital）合伙人 Sonya Huang 主持并发布于其播客频道，上线时间为2026年10月6日。节目对话 Google AI 基础设施首席技术专家 Amin Vahdat，重点围绕前沿 AI 领域的物理硬约束与经济学展开，探讨了在十万卡集群故障常态下为何应关注“有效吞吐（Goodput）”而非标称 FLOPS、TPU 产品线的分化演进、与 DeepMind 的软硬件协同设计，以及电网瓶颈、光路交换与未来轨道数据中心等核心议题。 ## TL;DR Amin Vahdat 深入剖析了超大规模 AI 数据中心的系统工程与算力经济学。面对十万加速器集群中每小时频发物理故障的严酷现实，Google 拒绝理论 FLOPS 这种虚荣指标，转而将穿透硬件与系统损耗后的“有效吞吐（Goodput）”作为核心考核标准。对话详细拆解了 TPU 架构从兼顾训练推理到在 TPU v8 拆分为专精推理的 8i 与专精训练的 8t 的决策逻辑，并揭示了基础设施团队如何与 DeepMind 在流片前夕“空中拦截”芯片设计。此外，Amin 还探讨了长思维链智能体对 CPU 和存储架构的冲击、纯光路交换（OCS）实现毫秒级故障重构的机制、电力作为终极刚性约束下与公共电网的协同方案，并前瞻了2036年数兆瓦级即插即用机架与轨道数据中心的物理可行性。 ## Atmosphere & style 整场对话沉稳严谨且极具工程师洞察力，兼具宏大产业视野与底层物理细节的交织。Sonya Huang 的提问直击千亿资本支出（CapEx）下的工程痛点，而 Amin Vahdat 则以务实、冷静且谦逊的口吻娓娓道来，既有面对庞大基础设施系统故障常态的严谨理性，又流露出工程师在芯片底层电路与光纤分形美感之间探索极限的纯粹浪漫。听感上犹如置身顶级分布式系统与半导体架构师的闭门复盘会议，不仅让人深切感受到当前 AI 算力竞赛背后惊人的物理与资本重压，更传递出置身历史级技术变革中心的振奋与使命感。 ## Bulletin - `05:30` Google 基础设施负责人 Amin Vahdat 认为理论 FLOPS 是虚荣指标，在 10 万颗加速器规模下系统每小时都可能发生故障，唯有穿透硬件与软件故障交付的“有效吞吐”（Goodput）才是衡量真实产出的核心标准。 - `15:32` 随着推理在算力生命周期中占比预计飙升至 30% 到 60%，Google 在 TPU v8 世代首次打破统一芯片架构，分化出专攻推理的 8i 与专攻训练的 8t，但二者仍保留相互兜底以防止灵活性丧失。 - `34:16` 长思维链智能体（Long-Horizon Agents）移除了人类交互的限流屏障，使得推理调用与编排请求呈微秒/毫秒级暴增，不仅拉动加速器需求，更带动 CPU 与各级存储等传统基础设施需求全面井喷。 - `42:35` 电力已成为 AI 数据中心最根本的刚性约束；面对 1 吉瓦级电力缺口，Google 优先选择联合公用电网协同扩建，而非承担 1+1 冗余成本去在表后孤立自建微电网。 ## Key takeaways - `01:47` 现代 AI 数据中心在建筑与机架功率上与传统机房分道扬镳，单机架功耗从过去的 10-40 kW 飙升至数百 kW 并逐步迈向兆瓦级，促成了长达 30 年的物理设施与内部短周期硬件的端到端深度协同设计。 - `10:00` 超大规模集群的系统故障呈现长尾分布，从光纤、芯粒先进封装到编译器及操作系统皆可能中断同步流水线，因此基础设施必须具备近乎实时的毫秒级感知与自愈恢复能力以捍卫 Goodput。 - `11:52` Google 保持约每六个月服务 Token 容量翻番的激进节奏，其能效与性能跃迁的核心贡献主要来自模型架构与算法创新，系统软件优化次之，而底层芯片硬件则充当底座的乘数支撑。 - `20:00` TPU 架构演进的关键权衡在于专用化与通用性：尽管 Transformer 占据主流，但其底层仍是矩阵乘法（GEMM）与稀疏核心等线性代数原语，硬编码特定模型结构将带来极高的淘汰风险。 - `26:11` 芯片与模型的全栈协同设计能够消除跨抽象层间的阻抗失配；在 Google 内部，DeepMind 算法团队甚至可以在芯片流片前夕“空中拦截”硬件架构，为换取全局能效收益不惜延后流片数周。 - `32:41` 从 TPU v1 到最新架构，其核心由大型双矩阵乘法单元、稀疏核心以及片上芯片间互连（ICI）远程加载/存储等五至六个基石原语构成，历经多代深度学习算法变迁仍未发生根本动摇。 - `37:50` 光路交换技术（OCS）利用微机电系统（MEMS）微镜阵列直接在纯光域反射光束，完全免除光-电-光转换带来的功耗与延迟，使得网络拓扑能在数毫秒内动态将故障机架切换至备用机架。 - `40:00` 虽然理论上自由空间光通信具备无介质优势，但因空间发散与精密对准难度，数据中心内网仍需依赖光纤引导光束接入微镜阵列以完成毫秒级动态路由。 - `47:15` 单体数据中心的物理规划受到单点故障与物理互联时延的夹击，Google 采用梯度化配置：将逼近吉瓦级的训练集群高度物理集约，同时将数百兆瓦或数十兆瓦的推理与边缘节点全球离散化部署。 - `50:00` 训练集群与推理集群在物理架构上分道扬镳，推理集群单兆瓦成本与复杂度往往更高，必须在全球离散分布以贴近用户，深度混构计算、高吞吐网络与存储资源，无法仅靠退役训练集群填补。 - `52:30` Google 内部服役七八年的旧 TPU 依然保持 100% 满负荷运转，硬件退役与技术升级通常以包含数千颗芯片的完整 Pod 级别进行整体机房置换改造，而非单卡逐一替换。 - `54:00` 为避免供应商锁定并延续互联网协议（IP）的历史经验，Google 拥抱开放标准与互操作性，在推崇自研 JAX 的同时开发 Torch-TPU，在 Google Cloud 上兼备 TPU 与 GPU 双栈支持。 - `58:30` 轨道数据中心在物理上有望利用太阳同步轨道近乎 100% 的日照时效与增强 40% 的太阳辐射，但受制于真空环境下极其低效的热辐射散热机制与太空维护挑战，短期内仍需探索解决。 - `01:01:30` 展望 2036 年的超级计算机，机架集成度将迎来质的跃迁，单个机架功耗或达数个兆瓦，工厂高度预制使得现场部署极简化为“水冷、电力、光纤束”三条接口的即插即用流程。 ## Detailed walkthrough ### `00:00` 硬件专用化的权衡与史上最大规模 CapEx 建设 硬件架构设计中存在一个根本性的权衡取向：对特定工作负载（workload）做越深度的专用化定制，其灵活性就越低；但换来的回报是运算速度的大幅提升和能效（power efficiency）的显著优化。这本身是一门精妙的艺术和对未来的前瞻预测——设计者必须判断自己瞄准的工作负载究竟具有多强的持久性。如果某种模型结构或计算范式在两三个月后就会被淘汰，即便它在这三个月内规模庞大，留给硬件拦截（intercept）的窗口期也极其狭窄。因此，专用化芯片所针对的工作负载必须具备足够持久的生命力，同时设计团队必须能够精确预估为了这一特性进行专用化后所能换取的具体收益。 红杉资本（Sequoia Capital）合伙人 Sonya Huang 随后引出了本期对话的嘉宾——Google AI 基础设施首席技术专家（Chief Technologist for AI Infrastructure）Amin Vahdat。Sonya 指出，人类当前正处于有史以来最大规模的资本支出（CapEx）浪潮之中，而 Google 仅今年一年预计就将投入超过 2,000 亿美元的 CapEx，其中绝大部分直接砸向数据中心的建设。Amin Vahdat 处于这场风暴的最中心。Amin 感慨道，纵观人类工程史或 Google 自身的发展历程，整个行业正在经历的扩张规模与转型速度都是前所未有的。 ### `01:47` 传统数据中心与 AI 数据中心：25-30年寿命与专用化共同设计 传统通用数据中心与 AI 数据中心在底层物理形态上存在很多共通之处，二者并非截然对立： - 两者都由高强度的混凝土外壳、电气场站（electrical yards）、机械冷区（mechanical yards）和冷却管网构成； - 内部同样整齐排列着用于分配电力的配电排、连接大规模计算单元的庞大网络基础设施，以及底层的存储资源池； - 数据中心的建筑外壳本质上是一笔长达 25 到 30 年的长期基础设施投资，而机房内部署的服务器与硬件生命周期通常只有大约 6 年，这意味着一座数据中心必须能够承载其物理生命周期内多达 4 到 5 代硬件的平滑迭代。 两者的核心差异在于专用化程度（purpose-built）和共同设计（co-design）。传统的通用数据中心追求高度的可替代性与通用性（fungible），但为了兼顾所有负载往往会导致机房设计过于臃肿。而现代 AI 数据中心则是将建筑本体与内部运行的硬件进行端到端联合设计的产物。在功率密度与配电架构上，两者的反差尤为悬殊： | 维度 | 传统存储机架（Storage Rack） | 现代 AI 算力机架（TPU / GPU Rack） | |:--- |:--- |:--- | | **单机架功率密度** | 约 10 至 40 kW | 轻松达到数百 kW，未来数年正迈向 MW（兆瓦）级别 | | **单排空间承载** | 一整排可容纳约 30 个机架 | 相同配电与物理排位下仅能承载 1 至 2 个机架 | | **网络带宽需求** | 极小（尤其是机械硬盘存储） | 极其恐怖的网络互联密度，线缆与光纤交错复杂 | 如果试图在 30 年周期内保持完全的通用性，就会造成设施的严重过度建造。因此，现代 AI 数据中心从冷却方案、配电回路到物理承重，都与特定硬件高度绑定与协同优化。Sonya 提到，Google 此前向其被投企业 Ineffable Intelligence 交付了一组大型算力集群，其光纤布线呈现出的“分形几何美感”（fractal nature）极其震撼，犹如现代人类工程的一座丰碑。Amin 赞同并回忆道，哪怕只是几台机架内部的光纤分发走线，发到社交网络上都引发了极大的关注和轰动，这种极致工程本身就是一种独特的工业美学。 ### `05:30` 拒绝 FLOPS 虚荣指标：在故障常态下追求“有效吞吐”（Goodput） 在谈到衡量超大规模建设的核心标准时，Amin 明确指出 FLOPS（每秒浮点运算次数）本质上是一个“虚荣指标”（vanity metric）。无论是标称的 FLOPS 还是其他以芯片为中心的指标，描述的都只是“理论最大值”，即在理想受控条件下单颗芯片能发挥的纸面峰值算力。但在真实的生产环境中，真正决定业务成败的是特定端到端工作负载实际交付的性能。 AI 性能几乎从来不是由单张芯片决定的。尽管标称 FLOPS、高带宽内存（HBM）容量与 SRAM 容量等参数至关重要，但现实系统往往是由 2 颗、4 颗、8 颗、16 颗乃至数万到十万颗加速器（TPU 或 GPU）紧密组合而成的集群，同时还需要上游 CPU 不断供给数据，并由底层高速网络将其全部编织在一起。 由此，Amin 提出了“有效吞吐”（Goodput）的概念，将其与单纯的“吞吐量”（Throughput）进行了深刻对比： ```mermaid flowchart TD accTitle: Throughput vs Goodput Work Cycle accDescr: 展现同步任务中发生故障时，传统吞吐量与实际有效吞吐的区别 subgraph NormalFlow ["正常计算流程"] S1["步骤 1 计算"] --> S2["步骤 2 计算"] S2 --> S3["步骤 3 计算"] S3 --> S4["产出最终有效结果 (Goodput)"] end subgraph FailureInterruption ["大规模集群节点故障发生"] F1["单芯片/光纤故障"] --> F2["微秒级同步依赖被打破，全任务挂起"] F2 --> F3["故障定位与 Checkpoint 回滚"] F3 -.->|"浪费的算力：属于 Throughput 但不是 Goodput"| S1 end ``` - **吞吐量（Throughput）与有效吞吐（Goodput）的区别**： 许多前沿任务（不论是大模型预训练、长上下文推理还是 Agent 智能体工作流）本质上是高度同步的（synchronous），往往成千上万个计算单元需要以微秒或毫秒级的粒度保持步调一致。 就像用纸笔算题，你写下了第 1 步到第 4 步，中途突然算错，不得不全部擦掉倒退回第 1 步重新推导。在这整个过程中，你的大脑确实一直在消耗能量、手一直在演算——这就是 Throughput；但最终用于解出答案的实际有效耗时——才是 Goodput。排查故障、任务停顿、等待诊断以及读取 Checkpoint（检查点）重做的大量无效计算，全都无法转化为 Goodput。 - **10 万卡集群下的物理故障常态**： 在十万颗加速器（100,000 accelerators）的集群尺度下，“每时每刻都有硬件在发生故障”。当今的前沿算力芯片本身就是半导体制造的物理极限，往往采用先进封装工艺，由 2 个、4 个、8 个甚至更多的 Chiplet（芯粒）拼接而成，外挂多颗 HBM 堆叠体，并连接板载高速网络甚至共封装光学器件（Co-Packaged Optics, CPO）。在如此庞大的组件基数乘以十万倍之后，任何一个物理部件损坏都会瞬间打断同步计算链条。 因此，Google 内部衡量基础设施成败的唯一严苛标准，是系统在包含真实物理故障的严酷条件下，到底能稳定交付多少可用的“Goodput”，而不是在实验室无故障环境下跑出来的理论基准性能。系统必须具备近乎实时的毫秒级故障感知与自愈恢复能力，才能在故障不断发生的世界里守住真实的算力产出。 ### `10:00` 十万卡集群的常态故障与“有效算力（Goodput）”考核 在规模达到 100,000 颗加速器的超大规模集群中，故障监测就像在奔流不息的大海里捞针。故障不再是罕见的异常事件，而是跨越数秒、数分钟乃至数周的长时间运行作业中持续发生的必然现象。Amin Vahdat 指出，在这个体量下，系统“每天必然会发生多次故障，甚至根据具体配置的不同，可能每小时都会出现几次损坏”。当被问及最常见的故障诱因时，他坦言根本不存在单一的“典型故障”——如果真有，工程团队早就彻底解决并修复了；现实中面对的是一条深不见底的“持续发现的长尾（long tail of constant discovery）”。 由于硬件始终处于最前沿的技术极限，故障可能出在超高速网络互联上，但也极有可能来自软件层面。Amin 强调，硬件芯片在纸面上或许拥有惊人的理论 FLOPS，但只要遇到编译器缺陷、运行时错误、模型层 Bug 或底层操作系统异常，整套端到端系统的性能就会直接崩溃。正因如此，Google 拒绝将峰值 FLOPS 作为衡量标准，而是以内部术语“Goodput（有效吞吐 / 有效算力）”来严格要求自己。Goodput 衡量的是真实且重要的业务负载在穿透实际硬件故障与软件损耗后，最终真正交付的有效工作量。Sonya Huang 询问这是否是 Google 独有的概念，Amin 表示它确实起源于 Google，但如今正在逐渐被整个行业广泛接纳。 当谈到是否有像 NVIDIA 那样成熟的“参考架构堆栈（reference stack）”即可一劳永逸时，Amin 评价 NVIDIA 是一家令人惊叹的“全系统公司”，提供了极其强悍的参考堆栈；而 Google 自身的 TPU 同样拥有标准参考堆栈。不过现实情况是，虽然多数客户会直接采用参考设计，但许多团队仍必须为了特定用例和极限优化去自行定制设计数据中心。 ```mermaid flowchart LR accTitle: 端到端 Goodput 损失链路 accDescr: 展示从理论峰值 FLOPS 到实际有效算力 Goodput 之间经历的故障与损耗层级。 subgraph Hardware["底层硬件层"] A["峰值理论算力 (Peak FLOPS)"] --> B["超高速网络互联抖动 / 物理硬件长尾故障"] end subgraph Software["系统与软件层"] B --> C["编译器 Bug / OS 异常 / 运行时故障"] C --> D["模型实现与框架层损耗"] end subgraph Output["最终交付"] D --> E["实际业务 Goodput (有效算力)"] end ``` --- ### `11:52` 每六个月推理容量翻倍与“每瓦智能”的来源剖析 面对爆发的业务需求，Google 内部设立了一个激进的技术目标：大约每六个月就要将对外服务的有效容量（Serving Capacity）翻一番。Amin 澄清道，这绝不等于硬件底层的物理 FLOPS 每六个月翻一倍，而是指硬件生成 Token 的综合能力（Token generation capability）必须以这一速度倍增。实现这种狂飙突进的容量增长，软件层面的贡献甚至要大于硬件本身，它依赖于成百上千个不断落地的模型架构创新、算法剪枝以及运行时微观优化的持续累加。 关于在“每瓦智能（Intelligence per watt）”或“每瓦有效算力（Goodput per watt）”的能效提升中，不同技术栈究竟扮演了怎样的角色，Amin 作出了定性拆解： - **模型层改进（贡献最大）：** 占据了绝大部分收益。模型架构与算法层面的跃升，直接重塑了单位电力所能转化出的智能水平。 - **系统与软件优化（极其关键）：** 承上启下，核心使命是确保底层昂贵的物理硬件被极致榨干、高效利用，避免算力空转。 - **底层芯片硬件（提供坚实乘数）：** 硬件本身展现出了惊人的迭代速度，目前完全能够实现年化 2 倍以上的性能飞跃。硬件并非“免费的午餐”，但它像是一个稳固的基座乘数，每一年都将上层所有软件与模型的收益整体向上托举。 --- ### `15:32` TPU 豪赌回顾与 TPU v8 走向 8i/8t 分化的算力经济学 Google 自研定制芯片的决策始于 2013 年，在当时这被公认为一桩违背行业常识的逆向豪赌。回溯彼时的技术语境，业界泰斗与顶尖专家坚守着芯片领域的“苦涩教训（the bitter lesson of chips）”——即摩尔定律每 18 至 24 个月性能翻番，通用 CPU 配合标准 C++ 编程模型永远会击败专用硬件，因此“绝不应该为单一负载定制专用加速器”。然而 Google 内部敏锐意识到，如果语音识别和机器翻译等极少数关键应用完全依赖通用 CPU，所需的服务器规模将庞大到令人绝望。尽管当时公司内部很多人怀疑其可行性，首代推理芯片 TPU v1 仍应运而生并取得了空前成功。紧接着，Google 推出了兼顾训练的 TPU v2。随后，Transformer 架构横空出世，推荐系统（支持广告业务核心）也全面接入，TPU 由此从最初狭窄的推理尝试，演化为支撑通用人工智能与万卡集群的战略支柱。 随着工作负载的演化，芯片团队必须在“做一款兼顾所有任务的通用芯片”与“针对细分场景深度定制”之间做出权衡。在两年前左右规划面向 2026 年的技术路线时，Google 面临一个关键抉择：究竟是继续做一颗兼顾训练与推理的单一芯片，还是首次将产品线拆分为两颗专精芯片？团队最终的决策促成了今年（2026年）双芯片架构的推出：专攻推理服务的 **8i** 与专攻模型训练的 **8t**。 ```mermaid flowchart TD accTitle: TPU 架构分支决策逻辑 accDescr: 展示 Google 权衡单一通用芯片与训练/推理分化芯片的决策逻辑。 subgraph TradeOff["分化评估决策"] Q{"推理负载在生命周期中占比预测？"} Q -->|"预测占比仅 2% - 5%"| Single["单芯片方案 (兼顾训练与推理)<br/>保持硬件规格单一性，不针对推理极端定制"] Q -->|"预测占比高达 30% - 60%"| Dual["双芯片战略 (TPU v8 路线)<br/>走向架构分化以换取极致能效与速度"] end subgraph Split["TPU v8 双芯片落地"] Dual --> 8i["TPU 8i: 极致专精于推理服务 (Serving)"] Dual --> 8t["TPU 8t: 极致专精于大规模集群训练 (Training)"] end ``` Amin 揭示了这一分化背后的定量推演： - 早期如果预测推理在芯片生命周期内仅占整体算力市场的 2% 到 5%，哪怕定制推理芯片能快上 2 倍，工程上也完全不划算，因为保持硬件体系的单一性（Uniformity）带来的综合运维收益更高。 - 但团队在两年前就清晰预测到，进入 2026 年后，推理与服务（Inference and Serving）将迎来彻底爆发，在芯片生命周期内将吞下高达 30%、40%、50% 甚至 60% 的市场份额。 - 当一个专精负载不仅体量足够巨大、能实现翻倍以上的提速，而且具备持续数年的长期增长确定性时，将其拆分为两颗专属芯片在工程与经济层面上就成了毋庸置疑的最优解。 ### `20:00` 专有化的权衡与 TPU 8i/8t 的设计哲学 硬件设计存在一个根本性的权衡：硬件越针对特定工作负载进行定制与专有化（specialization），其灵活性就越低，但运行速度会更快、能效也更高。因此，硬件架构设计本质上是一门艺术，需要精准预判所针对的工作负载具有多大的持久性。如果一个工作负载只存在一到三个月就会消失，即便在那三个月内规模庞大，硬件截获（intercept）它的时间窗口也极其狭窄。支持一个全新的硬件项目存在高昂的固定成本，设计者必须确保未来有足够持久的需求来支撑这笔投资，并且能明确测算出专有化带来的收益。 针对这种权衡，Google 在划分 TPU 8i（侧重推理）和 TPU 8t（侧重训练）时采取了极具远见的设计策略：两者都保留了运行对方工作负载的能力。如果 8i 只能做推理而对训练的性能为零，反之 8t 只能做训练而完全无法做推理，那将带来致命的局限性。因为硬件的服役周期长达六年，团队不可能在六年前就百分之百准确预测出这两种需求的精确比例。8i 和 8t 虽然各自在专精领域表现更出色，但一旦某一侧出现闲置容量，另一侧即可灵活调度补位，从而避免过度专有化导致的硬件死板与不可替代性（fungibility 丧失）。 面对“如今 AI 市场绝大多数基于 Transformer，为何不直接把 Transformer 架构固化烧录到芯片中”的提问，Amin 解释道，Transformer 的底层本质上是一系列线性代数原语（Linear Algebra Primitives），如向量和矩阵乘法运算（GEMM）、Softmax 等。这些底层原语其实早就已经内嵌在 TPU 以及其他竞争对手的硬件之中了。真正的差异在于模型架构层面的具体细节： - 模型的层数是多少； - 数据如何穿过这些层、跨层如何连接； - 每一个维度所应用的矩阵和向量的具体形状（Shape）是什么。 进一步的专有化不仅是针对 Transformer 本身，而是针对某个特定模型架构进行深度定制。业界已有不少初创公司在探索这一方向，这是一个极其有趣但需要承担更高风险的研究路径。 ```mermaid flowchart TD accTitle: Hardware Specialization Tradeoff accDescr: Diagram illustrating the tradeoff between general purpose GPUs and specialized TPUs, as well as the balance in TPU 8i and 8t. subgraph FlexibilityVsEfficiency["硬件权衡谱系"] GPU["通用 GPU<br/>极高灵活性 / 适配广泛场景"] TPU_8i["TPU 8i<br/>专精推理（保留训练保底）"] TPU_8t["TPU 8t<br/>专精训练（保留推理保底）"] HardcodedModel["模型硬编码 ASIC<br/>极限能耗比 / 零灵活性风险"] end GPU -->|"逐步下沉原语"| TPU_8i GPU -->|"逐步下沉原语"| TPU_8t TPU_8i -.->|"过度专有化风险"| HardcodedModel TPU_8t -.->|"过度专有化风险"| HardcodedModel TPU_8i <-->|"互相复用闲置算力"| TPU_8t ``` --- ### `23:30` 软硬件协同设计（Co-Design）的双刃剑 在评估芯片与工作负载的匹配度时，Google 内部与 Google Cloud 客户既使用 TPU 也大规模使用 GPU。GPU 相比 TPU 无疑更加通用（general-purpose），是极其出色的产品。Google 的核心策略并非强推单一架构，而是为客户提供多样化的选择，由客户根据自身工作负载的具体特征来评估何种芯片最能满足需求。 关于从芯片、网络到软件全栈推行协同设计（Co-design）的利弊，Amin 给出了深入的解构。不采用协同设计的路径是构建一个通用的 N 层抽象栈： - **通用抽象栈的优势（反对协同设计）：** 系统完全解耦，能够自适应任意硬件、软件和网络拓扑。当跨云运行或面对全新异构资源时，可以在完全不改动业务逻辑的情况下实现无缝迁移，“新容量一旦上线，系统一夜之间即可运转”。这种方案杜绝了供应商锁定（lock-in），获得了极致的便携性与灵活性。 - **通用抽象栈的劣势：** 每一层为了保持绝对的通用性，层与层之间会产生巨大的“阻抗失配”（impedance mismatch）。为了兼容所有基础设施，不可避免地会在计算、存储、网络等各个维度放弃潜在的深度优化空间。 - **协同设计的优势（赞成协同设计）：** 在垂直整合的协同设计体系下，每一层哪怕只榨取出 10%、20% 或 2 倍的优化潜力，将这些层层叠加的收益相乘（multiplied through），端到端累积起来就是一个惊人的全局优势。无论是以每瓦产出智能（Intelligence per watt）还是每瓦有效吞吐（Goodput per watt）来衡量，协同设计都能从供电架构、硬件布线一路穿透到顶层软件栈，带来巨大的飞跃。其代价则是放弃了跨平台的无缝通用性。 --- ### `26:11` 与 DeepMind 并肩作战：在流片途中拦截芯片架构 Amin 强调，在 Google 工作最令人振奋且欣慰的事情之一，就是能够与 Google DeepMind 团队肩并肩（shoulder-to-shoulder）进行硬件与模型的全栈协同设计。这种协同不仅限于两支团队，还无缝延伸至消费者级服务与 Google Cloud 基础设施。 在实际协作中，常有戏剧性的技术突破场景发生。例如，DeepMind 的研究人员在 Transformer 或特定数学算法上推导出了某种模型优化，而此时基础设施团队正在研发的一款新芯片尚未完全定型。DeepMind 会提出：“如果我们能为这种计算提供专门的硬件加速支持，端到端的训练或推理能效将实现大幅跃迁，现在修改芯片设计需要付出什么代价？” 面对这种契机，芯片工程师与算法研究员会立刻全员集结在同一间会议室，进行长达数天甚至两周的高强度闭门研讨。双方不断寻找帕累托最优解（Pareto frontier）： - 基础设施团队可能会指出：“我们无法百分之百按照你提议的方式落地，但我们可以换用另一种硬件实现方案；相应地，你们需要把模型架构微调一下，这样依然能获得 90% 到 98% 的预期加速收益。” - 基础设施团队会做出大胆决断：“为了这项巨大的端到端收益，我们完全愿意将流片（tape-out）推迟一周或两周，在半空中直接拦截（intercept）并重构芯片架构。” 这种深度的临门调整在跨公司的合作中几乎是无法想象的。如果相隔公司边界，在芯片距离完成交付仅剩几周或几个月时，叫停整个项目并打乱进度进行协同大修，即便并非绝对不可能，其阻力和沟通成本也会高到无法执行。而在 Google 内部，团队能够实时掌握模型内部的数学流动，也能掌握芯片底层电路的每处微结构。 ```mermaid flowchart LR accTitle: Chip Lifecycle and DeepMind Co-Design Integration accDescr: Diagram showing the multi-stage pipeline of chip development at Google and how DeepMind collaborates at each stage. subgraph Pipeline["芯片研发五阶段全景"] direction LR S5["1. 概念构想<br/>(Concept)"] S4["2. 架构设计<br/>(Design)"] S3["3. 逻辑实现<br/>(Implementation)"] S2["4. 回片调试<br/>(Post-Silicon)"] S1["5. 生产集群<br/>(Production)"] S5 --> S4 --> S3 --> S2 --> S1 end subgraph Collaboration["DeepMind 深度介入节点"] direction TB C_Design["前瞻协同：<br/>输入未来 2-3 年算法走向，<br/>借助大规模仿真预判匹配度"] C_Tapeout["空中拦截 (Intercept)：<br/>在流片前最后关头修改硬件，<br/>必要时推迟流片 1-2 周"] C_Prod["生产优化：<br/>软硬件全栈联调，<br/>最大化 Goodput per Watt"] end S4 <-->|"多套架构评估"| C_Design S3 <-->|"紧急闭门协商"| C_Tapeout S1 <-->|"端到端调优"| C_Prod ``` Google 任何时刻都维持着一条横跨数年、涵盖多个代际的芯片研发管线： 1. **概念构想阶段（Concept phase）：** 停留在长远设想与预研中； 2. **架构设计阶段（Design phase）：** 与 DeepMind 共同探讨两到三年后的模型演化方向，利用强大的仿真基础设施（Simulation infrastructure）对候选架构进行建模与工作负载映射； 3. **逻辑实现阶段（Implementation phase）：** 即将流片并送交台积电等代工厂制造，是执行“中途拦截”与架构微调的关键窗口； 4. **回片调试阶段（Post-silicon debugging）：** 芯片已从代工厂返回，正在进行工程调试以验证各项功能完备并准备量产； 5. **大规模生产阶段（In-production）：** 芯片已部署到数据中心集群，双方继续围绕运行指标联合调优，榨取每一瓦电力所能交付的最大智能。 ### `30:00` 软硬件协同设计的现实张力：在同一个房间里拦截芯片架构 在 Google，硬件与软件团队绝非各自为政、隔绝运作，而是同处一栋楼甚至常常坐在同一间会议室里，保持着高频且深入的日常交流。Amin Vahdat 提到自己每周都会与 Oriol Vinyals 或 Demis Hassabis 等 DeepMind 核心领袖多次沟通，这种紧密协作是整个工作中最具乐趣的部分。更令人兴奋的是，Google 已经在形成闭环：他们正直接运用当前的 Gemini 模型来协助设计下一代 Gemini 所依赖的底层芯片硬件。 面对主持人关于“软件快速试错与硬件极长周期之间是否存在阻抗失配（Impedance Mismatch）”的提问，Amin 坦言硬件规划周期确实极其漫长。在团队对外正式发布 TPU v8i 与 TPU v8t 的当下，TPU v9、v10 甚至更后继的架构早已处于概念落地与工程执行的不同阶段，动辄布局两到五年之后。然而，Google Research、DeepMind 以及 Transformer 架构的诞生自始至终都在这一批交织的会议室中演进，因而整整一代算法研究员都深度习惯于向硬件团队传导需求，并清晰知晓芯片运作的多年节奏。 这种默契带来了一种极具理性的权衡机制： - 研究员深知硬件流片（Tape-out）不等于软件代码提交通道（CL），绝不可能“改几行代码两周后就推上线”，任何叫停流片的决定代价都极其巨大。如果某项微调只能在即将流片的芯片上带来 0.5% 或 1% 的微弱边际收益，他们绝不会贸然介入。 - 反之，一旦发现能带来数倍提升（Multiplicative Gains）的突破性机会，软硬件团队会立即并肩评估能否在设计中途强行拦截并集成该特性。硬件作为“承载所有船只的底层潮汐”，一旦做出改动就能让全公司的所有负载同步加速。许多 DeepMind 研究员正是怀着“我两年前提出的一个架构构想，如今已在生产集群中让全 Google 的计算飞速运转”的成就感，持续深入参与硬件路线图的定义。 --- ### `32:41` 历经数代模型而未改的底层骨架：TPU 的核心原语 面对“提前五年预测算法突破是否属于不可能的任务”这一质疑，Amin 给出了出人意料的判断：从适度抽象的层级来看，TPU 的核心架构从初代 TPU v1 至今从未发生根本性颠覆。这就像 CPU 的指令集架构（ISA）一样，数十年来虽然底层指令集始终围绕加载（load）、存储（store）、加减分支（branch）等基础逻辑构建，但其上层软件形态已发生翻天覆地的跃迁。 TPU 的指令集与基石原语同样具有极强的普适生命力，其核心由五到六个决定性原语锚定： - 面向专用数值格式的大型双矩阵乘法单元（Large Matrix Multiply Units, MXU）； - 负责向量运算以及发散/汇聚（Scatter-Gather）操作的稀疏核心（Sparse Core）； - 具备强劲远程加载/存储（Remote Load/Store）能力的片上系统，能够通过芯片间互连网络（Inter-Chip Interconnect, ICI）直接高速读写跨节点的远程内存。 尽管每一代都会针对数值格式和硬件规模进行扩展升级，但这套基础原语成功贯穿了过去数代深度神经网络、算法演进以及模型结构的剧烈变迁。 --- ### `34:16` 长思维链智能体（Long-Horizon Agents）对数据中心形态的重塑 近年来尤其是今年年初以来，长思维链/长流程智能体（Long-Horizon Agents）的崛起彻底打破了原有大语言模型简短问答（Quick-turn LLM conversations）的服务范式，引发了数据中心需求的底层剧变。这种变化主要体现在两个维度： 1. **去除了人类交互的时间限速：** 过去用户在网页端或手机上输入提示词后，模型响应返回，人类需要数秒乃至数十秒的时间去阅读、思考、打字并进行追问。但在端到端自主运转的长流程智能体中，人机交互环节被剥离，没有任何人类节奏去天然限流（Rate-limit）。智能体解析输出、展开逻辑推理并生成下一步请求的耗时从原本的数十秒骤降至毫秒级，推理请求的涌入密度呈量级式暴增。 2. **CPU 与传统基础设施需求全面爆发：** 所有的结果解析、状态推理与工具编排几乎都在 CPU 上发生。智能体在每一步获取新认知后，必须在下一次调用模型前收集状态上下文——这些数据可能存放在本地内存（DRAM）、其他节点的 CPU 内存、固态硬盘（SSD）乃至远端机械硬盘（HDD）中。这带来了一场庞大的跨层级编排风暴。 因此，AI 数据中心并不仅仅是加速器算力的单边飙升，传统数据中心的 CPU 计算、高吞吐存储与低延迟网络需求同样在被推向天花板。 ```mermaid flowchart TD accTitle: 长思维链智能体引发的数据中心组件交互与开销示意图 accDescr: 展示了从模型加速器输出到 CPU 毫秒级编排、再跨越本地与远端存储收集状态，最终再次循环调用加速器的全过程。 subgraph AccelCluster["加速器计算集群 (TPU / GPU)"] Infer["模型推理与生成 (毫秒级输出)"] end subgraph CPUOrch["CPU 编排与解析层"] Parse["结果解析与逻辑推理 (毫秒级决策)"] StateReq["状态收集与下一轮 Prompt 构建"] end subgraph StorageHierarchy["多层级数据与存储网络"] LocalMem["本地 DRAM"] RemoteMem["跨节点 CPU 内存 (通过网络)"] DiskStore["分布式 SSD / HDD 存储"] end Infer -->|"极速返回响应 (无人类等待延时)"| Parse Parse --> StateReq StateReq <-->|"读取状态/环境上下文"| LocalMem StateReq <-->|"跨节点低延迟拉取"| RemoteMem StateReq <-->|"非结构化上下文检索"| DiskStore StateReq -->|"高频发起下一次模型调用"| Infer ``` 这给物理布局带来了两难抉择：TPU 机架相比 CPU 机架具有高得多的功耗密度与网络需求。如果将两者混合，整体机房与电力设计将极其复杂；如果维持物理单一性，将 TPU/GPU、CPU 和机械硬盘分别置于不同建筑物内，跨建筑的网络复杂度、可靠性压力与成本就会急剧上升，原本极低的时延可能会因队列积压而恶化至数百微秒甚至更久。 --- ### `37:50` 纯光域数据搬运：光路交换技术（OCS）的原理与演进 面对跨节点与跨机架通信的巨大挑战，Google 在网络技术特别是光互连领域始终保持领先。大约 16 年前，Google 就成为业内首批将波分复用（Wave Division Multiplexing, WDM）引入数据中心内部的先行者，实现单根光纤并行传输多路信号，并将其作为机架间通信的基石。伴随该技术同步落地的，是革命性的光路交换技术（Optical Circuit Switching, OCS）。 Amin 细致剖析了传统电分组交换机（Electrical Packet Switching）与 OCS 的本质差异： | 交换技术路线 | 核心工作机制 | 数据处理代价与瓶颈 | 内部核心硬件组件 | |:--- |:--- |:--- |:--- | | **传统电分组交换**<br>*(Packet Switching)* | 将光信号转化为电信号，逐包解析包含目标 IP 等元数据的报头（Header），在芯片内部查表后将数据包转发至对应端口。 | 面对每秒数十亿乃至数万亿个超高速数据包，电域解码、查表和重转发带来极高的功耗、串扰开销及延迟瓶颈。 | 电交换 ASIC、查表引擎、高频电收发器。 | | **全光路交换**<br>*(Optical Circuit Switch, OCS)* | 完全不碰触电域中的任何比特，数据全程以纯光形态传输；直接在物理层面将指定输入端口的光束无损反射导向目标输出端口。 | 彻底免除了光-电-光（O-E-O）转换的功耗损耗，对速率与编码格式完全透明，实现真正零丢包转发。 | 微机电系统（MEMS），通过微型电机驱动多维可旋转微镜组。 | Google 最早落地的 OCS 技术基于微机电系统（MEMS Switch）。其内部由高精度的三维微电机驱动反射微镜。整个交换单元通常提供 128 端口或 256 端口等规格，当携带巨量数据的光束从光纤射入输入端口时，微电机通过可编程方式动态调整微镜的物理倾角与旋转姿态，直接将光束折射投向指定的输出光纤端口，从而在物理微观层面实现了极致纯粹的毫秒级全光动态路由。 ### `40:00` 光学电路交换（OCS）与网络拓扑重构 光学电路交换（Optical Circuit Switching, OCS）在数据中心网络中发挥着革命性作用。最初引入这项技术有两个核心动机： 1. **建立机架群之间的局部性（Locality）：** 假设存在一个计算集群和一个存储集群（例如共同支撑搜索业务），已知它们之间会频繁通信，系统可以通过微镜调整，在两个机架群之间直接建立纯光学的快速直连通道。 2. **免插拔光纤的动态网络伸缩：** 无需任何运维人员手动插拔光纤，通过中央控制器指挥 OCS 重新配置网络脊干（Spine），就能按需扩展或缩减网络规模。 ```mermaid flowchart TD accTitle: 光学电路交换（OCS）在故障发生时的毫秒级重构流程 accDescr: 展示 TPU 机架发生故障时，OCS 内部微镜阵列偏转光束，快速将流量重定向至备用机架的过程。 subgraph TPUCluster["TPU 环形拓扑集群 (Torus Topology)"] RackA["正常运行 TPU 机架 A"] RackB["故障 TPU 机架 B"] SpareRack["常备闲置机架 (Spare Rack)"] end subgraph OCSNetwork["光学电路交换中心 (OCS)"] Mirror["微机电微镜阵列 (MEMS Mirrors)"] Controller["中央控制器"] end RackB -.->|"硬件/链路故障"| Controller Controller -->|"电控信号指示偏转"| Mirror RackA -->|"光信号"| Mirror Mirror ==>|"毫秒级偏转光束"| SpareRack ``` 在 TPU 架构中，全互连的环形拓扑（3D/4D Torus）直接连接所有加速芯片。Amin Vahdat 指出，提升系统有效产出（Goodput）的关键手段之一在于：集群中时刻保留一个备用机架（Spare Rack），一旦某个 TPU 机架发生硬件故障，控制器在数毫秒内偏转 OCS 内的微镜角度，将光线重定向至备用机架，瞬间完成整机替换。 针对 Sonya Huang 提出的疑问——“既然可以在自由空间反射光，为什么还需要光纤？”Amin 解释道，如果彻底摒弃光纤，光在空气中的衰减损耗会急剧增加，可用带宽会发生毁灭性骤降；此外，在超大规模厂房中，要在三维空间内精准对准所有光路而不依赖光纤介质几乎是不可能的。光纤将光束引导至 OCS 后，光线直接照射到微小的硅基微镜阵列芯片上完成路由调度。 - **动态路由时延**：通过 OCS 重定向光路并接入备用机架可在毫秒（Milliseconds）级完成。 - **物理实现约束**：由于光束衰减与三维空间对准的极高工程难度，数据中心无法采用纯自由空间光通信（Free-space optics），必须结合光纤引导与微镜偏转。 --- ### `42:35` 终极硬约束：电力、吉瓦级扩容与电网协同 在数据中心的各类瓶颈中，算力每瓦性能（Goodput per Watt）成为决定性指标。Amin 明确指出：虽然数据中心面临供应链、散热、网络等各种不断转移的挑战，但**电力是当前也是长期来看最根本的约束（Binding Constraint）**。其他工程瓶颈在可预见的时间周期内都有清晰的解决路径，唯独电力缺乏立竿见影的灵丹妙药。核能或许是未来充沛清洁电力的终极答案，但在何时落地、以何种规模落地仍是未知数。 当被问及建设一个需要 1 吉瓦（GW）负荷的数据中心是否意味着要自己造发电涡轮时，Amin 表示 Google 的首选模式始终是**并网运行（Grid-connected）**，深度依赖公用事业电力公司（Utilities）： * **提前数年联合规划（Co-planning）：** 无法简单要求电力公司“明天送来 1 吉瓦”，通常需要提前许多年启动协同设计。 * **全额承担配套基建成本：** 为避免电网因大型数据中心的建设而转嫁成本、导致居民电价上涨，Google 承诺全额垫付新建或升级的输电线路、变电站等基础设施费用。 * **过渡期与缺口应对：** 假设 Google 在 2028 年需要 1 吉瓦电力，而电力公司在 2028 年仅能交付 700 兆瓦（MW），剩下 300 兆瓦要到 2029 年到位。Google 的选择要么是推迟等待，要么是在本地部署部分太阳能或储能电池进行局部自建补充。 * **反哺电网能力：** 数据中心与电网建立双向供电机制。在全年用电高峰期（如盛夏最热的两周居民空调负荷激增时），数据中心可以依靠本地储备，反向向公共电网输送电力，缓解社区压力。 ```mermaid flowchart LR accTitle: Google 与公共电网的大数定律协同模型 accDescr: 对比孤岛式自建电力的高昂冗余成本与并网统计复用带来的双赢效益。 subgraph FullyVertical["完全垂直自建模式 (孤立微电网)"] Req1["需求: 1 GW"] --> Redundancy["1+1 冗余备份"] Redundancy --> Cost1["需自建 2 GW 装机<br/>极高资本开支与选址难度"] end subgraph GridCooperation["电网协同模式 (统计复用)"] Req2["需求: 1 GW"] --> UtilityGrid["公共电网基础供电"] LocalAsset["本地清洁能源/电池储能"] <-->|"波谷储能 / 高峰反哺"| UtilityGrid UtilityGrid --> Benefit["大数定律平滑负荷<br/>全社会电网与 Google 双赢"] end ``` 针对“为何不直接脱离电网、全流程自建发电机组”的疑问，Amin 从统计复用（Statistical Multiplexing）与大数定律的角度进行了算力与能源的类比：如果数据中心在表后（Behind the meter）孤立自建电力系统，要达到 99.99% 或 99.999% 的工业级可靠性，就必须配置 1+1 冗余，即为了 1 吉瓦需求必须实打实建造 2 吉瓦的发电机组，并且就近获取如此庞大的清洁能源极其昂贵和困难。与公用事业公司协同，依托更大的负荷基数互相调峰，是更具经济性和弹性的最优解。 - **规划周期与交付缺口**：以数年为周期推演，常出现多阶段分批交付（例如先通 700 兆瓦、次年补齐至 1 吉瓦）。 - **极高可用性代价**：脱离电网自建系统要达到 99.99%（4 个 9）或 99.999%（5 个 9）的高可用性，必须承担 1+1 冗余带来的翻倍建造成本。 --- ### `47:15` 数据中心尺寸规划与集群生命周期 关于“一个数据中心到底应该建多大”的辩论，在 Google 内部已经持续了超过 15 年： 早在 10 到 15 年前，内部曾有过激烈的争论，探讨是否应将所有算力集中在一个单一的 1 吉瓦数据中心内。然而，这面临两难抉择： * **单点故障风险（Single Point of Failure）：** 从 30 年的长期基础设施视角来看，集中在一个篮子里是巨大的灾难隐患。 * **物理集中优势：** 对应前沿大模型训练负载，“规模越大越好”（Bigger is better）。出于通信延迟和光学互联距离的要求，硬件集中在物理距离越小的范围内越有利。 * **物理上限封顶：** 如今 Google 的总算力需求规模已极其庞大，全球没有任何一个单一物理站点能够容纳整套系统的能耗。 因此，现实中的数据中心规模是基于模拟仿真与业务属性梯度化设计的动态组合： * **边缘节点与本地驻留（Edge / In-country）：** 仅数十兆瓦（Tens of megawatts），甚至在与 ISP 合作的机房中仅仅只有单一机架大小。 * **超大规模训练集群（Training Clusters）：** 物理集约度极高，单体规模逼近吉瓦级（Gigawatt scale）。 * **区域性综合站点：** 规模通常在数百兆瓦（Hundreds of megawatts）不等。 对于集群的生命周期管理（Lifecycle Management），Sonya 提出了行业常见的假设：最大最新的集群优先用于训练前沿基座模型，随着硬件老化再降级淘汰用于运行推理任务（Inference）。Amin 证实了这种资产轮转框架的合理性，但随即指出当前的产业现实正在颠覆这一逻辑——由于现实世界中推理计算的爆发式需求极其庞大，仅仅依靠退役训练集群转作推理，其算力规模已远远无法满足实际需求。 ### `50:00` 训练集群与推理集群的分化：地理分布与软硬件混构 Amin Vahdat 深入剖析了训练集群（Training Clusters）与推理集群（Serving Clusters）在构建逻辑上的根本差异。在特定年份中，Google 会倾向于将训练集群集中放置在少数几个超大型站点中，因为缩短这些训练节点之间的网络通信物理延迟能够带来巨大的系统收益。然而，这种集中化策略也带来了空间上的妥协：集群往往会被压在同一个大洲甚至同属一个大洲的某一特定区域内，导致其他大洲面临推理算力供给严重不足的局面。虽然数年之后退役的训练集群确实可以转去跑推理，但实际业务根本无法等待这种时间差，Google 必须同步在全球其他区域专项建设分布式的推理集群。 从架构上看，推理集群单兆瓦（per Megawatt）的建设成本不仅没有比训练更便宜，反而更加复杂。训练集群追求的是整齐划一的高密度算力部署；而推理集群则高度依赖将算力、网络与海量存储紧密融合（co-locate）。此外，推理需要极度贴近最终用户，必须在全球不同区域支持模型终端（Endpoints）及其衍生变体，并满足严苛的数据本地化（Locality）要求。这意味着推理集群的规模往往更小、更离散，在系统结构上也比训练集群更少采用极端的纵向集成（Vertically Integrated）设计。 ```mermaid flowchart TD accTitle: Training vs Serving Cluster Architecture accDescr: 展现训练集群集中化大密度与推理集群全球分布式混构的部署差异。 subgraph Training["训练集群架构 (集中化)"] T1["少数大型站点 (同一大洲/区域)"] T2["超高密度算力池"] T3["极度均一的专用硬件"] end subgraph Serving["推理集群架构 (全球分布式)"] S1["小规模、全球离散化节点"] S2["计算、网络与存储高密度混构"] S3["低延迟终端服务与数据本地化"] end Training -->|"数年后硬件折旧/降级"| Serving ``` --- ### `52:30` 7至8岁高龄 TPU 仍全负荷运转：Pod 级生命周期与整机替换的艺术 谈到硬件的折旧与淘汰，Amin 澄清了外界热议的“老旧芯片寿命”话题。他此前公开表示“Google 内部七八年前的旧 TPU 依然保持着 100% 的利用率”，这一事实在业界引起了不小的轰动。芯片的账面折旧周期大约为 6 年，在此期间老一代 TPU 和 GPU 依然被大量实际工作负载填满。当 Google 最终决定淘汰升级这些旧算力时，他们并不是一颗一颗地去拔插芯片，而是以系统级单元——Pod 为单位进行整机替换。 例如一个典型的 TPU 8t Pod，可能包含 9,600 颗芯片、分布在 152 个机架上。当决定拉出这整个 Pod 并用未来的 TPU v12、v13 或 v14 Pod 取代它时，新硬件的占地空间、供电和散热规格与旧机架留下的物理空位（Vacancy）往往无法完美吻合。硬件工程团队根本不可能提前几年准确预测后续数代 TPU 的具体形态，因此退役旧设备、做机房改造（Retrofit）并以最快速度换上新 TPU，不仅是一门精密的技术活，更是“一种充满挑战的艺术”。 - 硬件经济账：芯片折旧周期通常在 6 年左右，但七八年前的老 TPU 依然处于全额满载（100% 满负荷利用）状态。 - 替换颗粒度：退役和升级是以整个 Pod 维度进行（如单个 TPU 8t Pod 拥有 9,600 颗芯片与 152 个机架），而非单个芯片级别的替换。 --- ### `54:00` 坚持开放标准：从 JAX、PyTorch 到互操作性的“沙漏模型” 针对垂直整合与开放生态的关系，Amin 强调 Google 绝不想构建一个锁死开发者的“围墙花园”（Walled Garden）。尽管 Google 内部深度使用并非常推崇自研的 JAX 框架，但许多外部开发者依然首选 PyTorch。Google 的态度并非“想要跑 TPU 就必须用 JAX，因为 JAX 最好”，而是打造了 `Torch-TPU`，让开发者无需修改模型代码就能在 TPU 上平滑运行。 Amin 将这一理念类比于 1970 至 80 年代互联网协议（IP）的历史胜利。当时存在多种网络协议竞争，但 IP 凭借开放标准成为了整个通信体系中至关重要的“沙漏窄腰”（The narrow waist of the hourglass）：任何软件都可以跑在它之上，任何硬件都可以支撑在它之下。只要支持 IP，任何人都能插进路由器端口并融入全球网络。AI 基础设施的体量过于庞大、重要性过高，绝不能沦为某一家供应商封闭的私有技术栈。Google 致力于在各个接驳点支持开放标准与开源生态，这也是为什么 Google Cloud 同时全力提供 TPU、GPU 等多样化算力选择。 --- ### `56:30` AI 正在深刻重塑硬件研发与数据中心规划 当谈及团队的日常工作如何被 AI 改变时，Amin 指出，软件工程方面的 AI 代码辅助早已广为人知，但硬件端发生的变化同样深刻。从内部 Token 消耗等指标来看，硬件工程师使用 AI 的频率已与软件工程师不相上下。从芯片设计启动（Design Kickoff）到流片（Tape-out）的时间周期正在被显著压缩，芯片点亮（Bring-up）阶段的耗时也在缩短，生产力得到了实质性跃升。 更为隐秘但同样关键的变革发生在数据中心的设计规划中。以往在评估是建设单体吉瓦（GW）级园区，还是规划 100 兆瓦、200 兆瓦的分布式园区时，团队极度依赖繁复的电子表格和纯人工推演。如今这一过程大量引入了 AI 工具辅助，极大地精简了规划与开发流程。Amin 澄清道，目前这套工具还算不上纯粹的“推理模型”（Reasoning Models），它并没有完全取代人类的最终决策，但它能极其高效地整合海量信息，把最核心的全局数据清晰推送到决策者面前。 - 硬件开发提速：从项目立项、逻辑设计到 Tape-out 及 Bring-up，硬件全流程正因 AI 辅助而大幅加速。 - 选址与设计智能化：在兆瓦与吉瓦级园区的架构取舍上，AI 大幅简化了复杂因果推导，取代了以往繁重的人工表格规划流。 --- ### `58:30` 轨道数据中心：太空能源与日照时效的物理底牌 在对话尾声，话题转向了看似遥远却备受关注的未来概念——轨道数据中心（Orbital Data Centers）。Amin 直言，人类之所以开始严肃测算太空计算的可行性，正是因为地面上的能源供应正遭遇最硬核的物理天花板。如果从底层物理常识去推演，太空中蕴含着地面无法比拟的能量优势。 首先，由于没有地球大气层对光照的吸收与衰减，太空中的单位太阳辐射能量要高出大约 40%（即地面的 1.4 倍）。其次更为致命的是时间利用率：在地面上，受制于昼夜交替与天气，光伏面板的有效利用率仅有 28% 至 35% 左右；但如果将数据中心部署在太阳同步轨道（Sun-synchronous Orbit），太阳能电池板将拥有高达 98% 至 100% 的连续日照时间覆盖。将 1.4 倍的辐射强度与近乎 3 至 4 倍的有效时长相乘，太空环境展现出的能源捕获潜力是地面的数倍之多，这正是它吸引基础科学与工程界深度推演的核心驱动力。 ### `01:00:00` 轨道数据中心的机遇与工程挑战 将数据中心送入太空轨道在理论上极具吸引力。最直接的优势在于能源：在轨道上，设备几乎可以全天候沐浴在恒定的太阳能辐射中，无需像地面光伏那样受制于昼夜交替，从而从方程式中基本剔除了庞大且沉重的储能电池需求，实现了彻底的零碳排放。然而，这一构想背后的工程现实却充满重重阻碍。 Amin Vahdat 指出，人们直觉上常误以为太空极其寒冷，因此散热应该更简单，但事实恰恰相反——在真空中缺乏空气对流和传导媒介，废热只能完全依赖效率极低的热辐射散发，这让热管理变得极其棘手。其次是可靠性与维护问题：地面上由数万乃至数十万加速卡组成的集群每小时都会发生硬件故障，而在太空中，任何硬件维修都难如登天。虽然未来可能会有小型维修机器人在集群中巡检更换部件，但在极高冗余度设计真正成熟之前，空间运维仍是一大难题。此外，太空中的网络互连方式也将发生根本性颠覆，各计算模块之间不可能通过繁复的实体光纤相连，自由空间光通信（Free-space optics）将成为必然选择——系统必须通过极高精度的激光发射端实时校准并精准对准移动中的接收端。不过，Amin 强调，尽管工程难度极为苛刻，但这里并不存在任何物理层面上“根本不可逾越的致命障碍（no fundamental showstoppers）”。 - **光照与能源**：轨道太阳能摆脱了昼夜循环与储能电池的束缚，提供全天候持续供给的无碳电力。 - **热辐射散热瓶颈**：缺乏对流与传导介质，太空散热纯靠辐射散热板，工程散热难度远高于地面水冷或风冷。 - **自由空间激光通信**：放弃物理线缆，依赖微弧度级的高精度动态对准激光链路实现组件间的高速通信。 ### `01:01:30` 展望2036：超高集成度与数兆瓦级的未来超级计算机机架 当被问及十年之后（2036年）最前沿的超级计算机会是什么形态时，Amin 笑着坦言，十年跨度的“不确定性锥形（cone of uncertainty）”实在太宽了，极少有人能真正推演那么遥远的细节，但从宏观物理演进趋势来看，“集成度（integration）的跃升将是颠覆性的”。目前的超算集群机架内部依然充斥着密集而繁复的跳线光纤，而在未来的设计中，机架内部的集成度将显著提升，外露光纤会大幅缩减，整个机架可能只向外延伸出一小束紧凑的光纤束。 从模块化制造的角度设想，2036年的超级计算机机架很可能会在高度集中的专业工厂内完成深度预制。机架内部将极其紧密地集成大批量芯片——无论是72、144、288块，还是翻倍至576乃至1152块 GPU 或 TPU。在如此夸张的芯片密度下，单个机架的额定功耗甚至可能直接达到“数个兆瓦（multiple megawatts）”。未来的现场部署也将被简化为一种极度模块化的流程：卡车将整机架推入数据中心机房，技术人员仅需插上三样东西——水冷管道、动力电源与一束高带宽光纤，整套兆瓦级算力单元便能立刻全速上线。面对主持人开玩笑追问这是否会变成一个飘在太空中的“外星机械球体”，Amin 幽默地回应称，到了2036年，这种高集成机架甚至可以直接发射升空，由空间站上的机械臂直接抓取并插接入对应的轨道母舱模块中。 ```mermaid flowchart LR accTitle: 2036年超算机架的模块化交付流程 accDescr: 展示从中心化工厂制造集成度达千卡的数兆瓦机架，到现场即插即用部署，再到远期轨道自动化交付的架构流程。 subgraph Factory["中心化超级工厂制造"] direction TB A["高密度芯片深度互连<br/>(576 / 1152+ 核心)"] --> B["超紧凑内部总线封装<br/>(大幅压缩外露光纤)"] end subgraph Deployment["地面即插即用部署"] direction TB C["数兆瓦级一体化机架<br/>(Multi-Megawatt Rack)"] D["现场三大接口接入<br/>(极简光纤束 + 电力 + 水冷)"] C --> D end subgraph Orbital["未来轨道数据中心构想"] direction TB E["模块化机架整体火箭发射"] --> F["轨道空间站机械臂抓取并盲插"] end Factory --> Deployment Deployment -.->|"2036年长期演进"| Orbital ``` - **单机架兆瓦级功耗**：算力密度成倍压缩，未来单机柜功率预计将迈入数兆瓦级别。 - **极简化三线接入**：部署逻辑收敛为“推入机架、连接供水、接通电源、插入光纤”的即插即用模式。 - **深度集中制造**：由传统机房现场繁重的跳线组装，彻底转向工厂端超高集成度模块化预制。 ### `01:03:30` 结语：历史上最大规模的资本支出与技术革命 对谈进入尾声，主持人 Sonya Huang 总结道，当前围绕 AI 算力展开的基础设施建设是整个人类科技史上规模最大、强度最高的资本支出（CapEx）浪潮，但它绝非冰冷的数字堆叠，而是一场蕴含着极致工程美感的技术革命。无论是攻关底层物理学限制、平衡供电与通信的刚性束缚，还是驾驭多级网络与异构芯片的协同，这套系统工程展现出了前所未有的宏大与严密。Amin 对此深有感触地表示，这场基础设施竞赛最令人振奋之处在于，他们这代工程师不仅有幸亲身经历这一历史性巨变，更能够置身其中亲手去定义未来算力的形态。 ## Notable quotes > `05:30` At 100,000-accelerator scale, something fails multiple times an hour, which is why Google's Chief Technologist for AI Infrastructure Amin Vahdat thinks FLOPS is a vanity metric. The metric that matters is what he calls goodput: the useful work a workload actually delivers through real failures. — Amin Vahdat > `10:00` 每天必然会发生多次故障，甚至根据具体配置的不同，可能每小时都会出现几次损坏 — Amin Vahdat > `10:00` long tail of constant discovery — Amin Vahdat > `15:32` the bitter lesson of chips — Amin Vahdat > `26:11` 如果我们能为这种计算提供专门的硬件加速支持，端到端的训练或推理能效将实现大幅跃迁，现在修改芯片设计需要付出什么代价？ — Google DeepMind > `26:11` 为了这项巨大的端到端收益，我们完全愿意将流片（tape-out）推迟一周或两周，在半空中直接拦截（intercept）并重构芯片架构。 — Amin Vahdat > `42:35` 电力是当前也是长期来看最根本的约束（Binding Constraint） — Amin Vahdat > `52:30` Google 内部七八年前的旧 TPU 依然保持着 100% 的利用率 — Amin Vahdat > `54:00` The narrow waist of the hourglass — Amin Vahdat > `01:00:00` no fundamental showstoppers — Amin Vahdat ## Mentioned Google, Google DeepMind, Google Cloud, Sequoia Capital, Sonya Huang, Amin Vahdat, Demis Hassabis, Oriol Vinyals, Ineffable Intelligence, NVIDIA, TSMC (台积电), TPU (Tensor Processing Unit), TPU v1, TPU v2, TPU 8i, TPU 8t, GPU, CPU, Gemini, Transformer, JAX, PyTorch, Torch-TPU, OCS (Optical Circuit Switch / 光路交换技术), MEMS (Micro-Electro-Mechanical Systems), WDM (Wave Division Multiplexing) ## Video description At 100,000-accelerator scale, something fails multiple times an hour, which is why Google's Chief Technologist for AI Infrastructure Amin Vahdat thinks FLOPS is a vanity metric. The metric that matters is what he calls goodput: the useful work a workload actually delivers through real failures. Amin walks through the calculus that split the TPU line into 8i and 8t for the first time, why the TPU's core primitives haven't changed since v1, and how Google and DeepMind co-design in the same rooms, intercepting chip architectures mid-flight before tape-out. He explains why long-horizon agents are sending demand for CPUs and storage through the roof alongside accelerators, how optical circuit switches reroute light to a spare rack in milliseconds, and why Google would rather wait on a utility than build its own gigawatt. We also cover orbital data centers and the multi-megawatt rack of 2036. Hosted by Sonya Huang, Sequoia Capital 0:00 – Introduction 1:47 – What makes a data center an AI data center 5:30 – Goodput, not FLOPS: holding yourself accountable when something fails every hour 11:52 – Doubling token capacity every six months, and where the gains actually come from 15:32 – The TPU bet: from a contrarian call in 2013 to splitting 8i and 8t 23:30 – The case for and against co-design 26:11 – Shoulder to shoulder with DeepMind: intercepting chips mid-flight 34:16 – Long-horizon agents change the shape of the data center 37:50 – Optical circuit switching and the state of networking 42:35 – Power is the binding constraint: utilities, gigawatts, and sizing a data center 49:23 – Training vs. serving clusters, seven-year-old TPUs, and open standards 58:29 – Orbital data centers and the supercomputer of 2036 [Source](https://pscrb.fm/rss/p/traffic.megaphone.fm/CPUAI7791161404.mp3)

[5:30] Amin Vahdat: At 100,000-accelerator scale, something fails multiple times an hour, which is why Google's Chief Technologist for AI Infrastructure Amin Vahdat thinks FLOPS is a vanity metric. The metric that matters is what he calls goodput: the useful work a workload actually delivers through real failures. [10:00] Amin Vahdat: 每天必然会发生多次故障，甚至根据具体配置的不同，可能每小时都会出现几次损坏 [10:00] Amin Vahdat: long tail of constant discovery [15:32] Amin Vahdat: the bitter lesson of chips [26:11] Google DeepMind: 如果我们能为这种计算提供专门的硬件加速支持，端到端的训练或推理能效将实现大幅跃迁，现在修改芯片设计需要付出什么代价？ [26:11] Amin Vahdat: 为了这项巨大的端到端收益，我们完全愿意将流片（tape-out）推迟一周或两周，在半空中直接拦截（intercept）并重构芯片架构。 [42:35] Amin Vahdat: 电力是当前也是长期来看最根本的约束（Binding Constraint） [52:30] Amin Vahdat: Google 内部七八年前的旧 TPU 依然保持着 100% 的利用率 [54:00] Amin Vahdat: The narrow waist of the hourglass [60:00] Amin Vahdat: no fundamental showstoppers

[10:00] Amin Vahdat: multiple times an hour, something is going to fail. [10:00] Amin Vahdat: If there were a common failure reason, we'd have figured it out and fixed it. [11:52] Amin Vahdat: 2x or more year-over-year performance improvements [26:11] Amin Vahdat: They have come up with model optimizations, let's say, to transformers or to particular math that they might want to do. And we might have a chip in progress—not quite done—but then they say, 'Oh my gosh, if we had hardware support for this, our end-to-end training or serving might get significantly faster, more efficient. What would it take to actually now change our hardware definition that we might be in execution on to accommodate?' [51:48] Amin Vahdat: a bit of an art [53:57] Amin Vahdat: narrow waist of the hourglass

[0:00] In hardware, again, as you know, there is this opportunity where the more you specialize to a particular workload, the less flexible it is. the faster. the more power efficient the hardware is going to be. So it is this art and it's this projection of what are you designing to and how persistent is that workload. In other words, if it's going to go away after a month or two months or three months, even if it's big for those three months, you got a really narrow window to intercept it. So it has to be somewhat durable and you have to be able to project exactly what win you can get for specializing to it. Bye. Thrilled to welcome Amin Vadat to the show. Amin, thank you for joining us for today. It's really exciting to be here. I'm really looking forward to it. I am very excited for today's topic because we are in the middle of the biggest CapEx... build out in human history. You are in the middle of it. Google alone is expected to spend more than $200 billion on CapEx this year, most of which is going into building data centers. You're at the center of it all. You were named the head of Google's AI Infra at the end of last year. So you are the man spearheading the efforts of one of the most capital-intensive build-outs in human history. And so I am really excited to get into it with you today. It's a huge year, for sure. Of course, across the industry, including at Google. Really, I don't think we've ever seen anything like this, honestly. Certainly not at Google, but as you said, I think in the history of humanity in terms of the build-outs and the pace of transformation, really unbelievable. Before we get into it, maybe just crash course for the audience. What is an AI data center and how is it different from a non-AI data center? It's a very good question in that AI data center and a non-AI data center do have a lot of similarities, actually. And they're not radically different. I mean, it consists of concrete. An enclosure, it consists of electrical yards, mechanical yards, cooling. You have row after row of power being distributed. There's a huge amount of network infrastructure there. In other words, we connect large amounts of computes to one another. A fair amount of storage infrastructure goes in there. 30 year building. investments. And we're thinking about how it's going to be evolving over that 2025, 30-year period. You might have servers go into it. You might have networking storage. You might have some accelerators, GPUs, TPUs, whatever they might be going into it. But it's a 25, 30-year planning horizon. And the lifetime of the hardware might be six years. So many generations that we have to plan for. An AI data center oftentimes is going to be more purpose-built. So in other words, we are... oftentimes, co-designing the building with the hardware that might go into it. We might be saying, you know what, actually we're not going to put a lot of storage into this building. Why? Because a storage rack might have 10, 20, 30, 40 kilowatts of power. You put that up next to a TP rack or a GP rack that easily is hitting hundreds of kilowatts today. might hit megawatts, people are talking about that in the next few years. Designing a building that might take 30 storage racks in a row versus One. or two AI racks in that same row. Very, very different design. Just think of it in terms of the size. Think of it in terms of the power and how that would be distributed across the building, etc. Networking would be the same. The amount of networking that you would need for a storage rack... Tiny. Tiny. I mean, especially if it's hard drives, compared to an AI rack. So if you're trying to make it fully fungible, you'll probably make it too big and too overbuilt. fungible over certainly a 30-year period. an AI data center likely is going to be much more purpose-built, co-designed with the hardware, even to the point of, Cooling. and power distribution, et cetera. So a lot more co-optimization. You all delivered a big Verirubin cluster to one of my portfolio companies, Ineffable Intelligence, and I saw the photographs of it as it went out. That is a... It's a thing of beauty. It is. It is. It's one of those things where, similar to looking at the biggest construction projects in human history, you look at that data center and it's like, wow, that is a monument to what mankind can do. No, we got a nice picture of it and this is just a couple of racks. The cabling, the fiber distribution for it is really beautiful. I mean, beauty is not the beholder. [5:00] But for folks like me and probably yourself and others in the audience as well, it is a thing of beauty. We put this picture up on social media and people love the fact that Ineffable, but they're a huge fan of what they're doing. Fantastic team. That got a lot of attention. Okay, great rack for Ineffable, but actually people... just loved seeing the pictures of the fiber and the sort of the fractal nature of the fiber. One of the most popular posts that we've ever had actually was around that. So, so yeah, absolutely. We're really excited about that. Yeah. I had goosebumps seeing the picture. How do you measure and hold yourselves accountable in this, in the middle of this big build out? We were talking right before the show about how flops is a vanity metric and you prefer an alternate metric. Can you, can you say more? Yeah, so I think one thing to note is that whether it's in flops or pick your other favorite chip-centric metric, These are in theory. In other words, under some conditions, for whatever chip you have, this is the maximum amount of flops that you can deliver. What we really care about in the end is what's the performance delivered by workload. That's what we're looking at. And rarely, it happens, but actually relatively rarely, is that performance determined by a single chip. It's flops, it's HBM, it's SRAM capacity, et cetera. Those are all super important, but it might be how two, four, eight, 16, a thousand, ten thousand, more of these chips composed together with not just the accelerators, again, whether TPUs or GPUs, but then the CPUs that might feed them the data, the network that connects them all together. So, what is the workload? that you're running and what is the performance of that workload. One measure that is interesting is what's your flops utilization? So for example, for a particular workload, if you have in theory, a teraflop, or a petaflop capable for your workload, which fraction of that are you delivering? Now, That's a measure of good put. What is good put? So you can think of throughput, which is a well-known term. That's the throughput that is possible. But now let's consider some other considerations. One is, what is the slowdown of the workload? Again, just inherent to the workload, but other aspects of it that really hits are reliability. So in terms of how we hold ourselves accountable, if we have a chip failing for a synchronous workload, and many of these workloads, whether training or serving, or agentic workloads, they're synchronous. There's many, many components that are working together simultaneously. Now, if you have let's say a thousand, ten thousand, a hundred thousand of these components working together simultaneously, and they're really needing to coordinate at microsecond or millisecond granularity. One of them fails. It might actually bring the whole thing to a stop. because everyone is counting on everyone else to do that part of the job in order to come up with the answer. to a really tough question. One of them stops. Okay, now we have to figure out what happened, which one stopped. what's the checkpoint that we have of the computation at some previous state and time? How do we restore that checkpoint? How do we restart? Worst case, we have to restart from the beginning. That would be really bad, but that can happen in certain cases, especially on the inference side. The point here is that if you now then have to go back and redo a bunch of computation, if you then have to pause and wait to figure out what happened for the failure, All that work is work. It's not actually helping you get the answer. right? In terms of, let's say that you're working out a problem on paper. Step 1, Step 2, Step 3, Step 4. If you have to go back to step one, Because you've made a mistake. Yes, you're still doing work. That's throughput. You're doing work, but what is the good put in delivering your answer? It's basically the total amount of time it took you to solve the problem. That's what matters. And if you have failures, if you have failure recovery, if you have whatever it is that's interrupting your work, that's that's part of the issue so now how do we hold ourselves accountable it's delivered good put, not theoretical benchmark throughput or goodness in theory, but actually for a workload for real failure conditions was happening. And the unfortunate reality is that at 100,000 accelerator scale, I was going to say, at that scale, something's failing all the time, right? All the time. And each one of these, as you're aware, Each one of these chips is a wonder of nature. I mean, they're at the very bleeding edge of what's possible to manufacture. And so, again, to me, it's stunning. And now, actually, these chips aren't just one chip. They're packages made up of, in many cases, two, four, eight, Maybe more chiplets that are composing together. And then, of course, HBM off to the side, network connectivity, maybe things like co-packaged optics. No criticism. Lots of things can fail. And now if you've got 100,000 of them, something is going to fail. Yes. You have to be prepared for that, detect that in near real time. [10:00] Cover from that in near real time. doing all that it's really like finding the telemetry problem is massive it's like finding a needle in the haystack continuously across certainly seconds and minutes but for some of these jobs hours and days and even weeks it's just continuous and online how we hold ourselves accountable is What's the good put that we deliver for the workloads that actually matter in the data center? Is good put a Google term or is it an industry term? It's a Google term, but I think more and more people across the industry are starting to pick it up. And then just to calibrate me, 100,000 accelerator scale, are we talking it's going to fail... once a minute, once a day? How- - 100,000 accelerators, so let me see. I think that it is definitely going to be at the scale multiple times a day, and perhaps, depending on the exact configuration, multiple times an hour, something is going to fail. - And what are the most common failure reasons? This is the problem. Actually, it's a very good question. If there were a common failure reason, we'd have figured it out and fixed it. It is a long tail of constant discovery. Whenever we have a new product that's being introduced, there's going to be something that hits us. In many cases, frankly, because again, it's at the very bleeding edge, it might be network related. It might be how we connect these things together at super high speed. That might be something to but then a lot of the issues could be software. So this is the other aspect of how we hold ourselves accountable. Again, the chip might be capable of a certain level of flops, but if you have a, compiler bug, a runtime bug, a model issue, something else, operating system issue, it doesn't matter, that's going to impact the N10 system performance. So you might have perfect hardware, fully reliable, but then you might have software issues that hurt you. Is there a standard reference stack from the accelerator companies, from NVIDIA or from the TPU team of, you know, here is the optimal system to build around our accelerators, and as long as you build to that system, you're good? Or how much of your own kind of data center design do you have to do above and beyond what the… what the semiconductor companies give you? Right. So there's a reference stack. And I mean, I would say that NVIDIA is an incredible whole systems company. I mean, they're obviously a semiconductor company, but they're not just a semiconductor company. They give you a really, really strong reference stack. But I would say that many, we find that the way I'd put it is most of our customers leverage that reference stack, but many also specialize. So in other use case, they have an optimization opportunity or something different that they need to do. And they're going to do that. Similarly, for the TPU side, we have a reference stack. But then, and most people will leverage that, many will specialize it as well. Got it. And then I guess in terms of overall capacity, you've told your teams that Google has to roughly double the serving capacity every six months or so. Is that right? Well, so to be clear, this is in terms of effective available capacity from the way in the end that we look at it is from a serving perspective, it's token generation capability. So capacity, and this is where I would say, it's a combination of software and hardware. the hardware might have some "inherent level" of flops. What I'm saying here is not necessarily you have to double the number of flops every six months. That's one path. You have to double the capability of that hardware to generate tokens every six months. and as much or more of that is going to come from software. as it is from hardware. So in other words, it could be a model optimization that delivers that. It could be some runtime optimization that delivers that. It's really probably going to be dozens or hundreds of individual optimizations that are just landing again and again and again to make it all possible. But yes, the rate of capacity improvement is incredible. I guess we've had a few years of data center build out now, and a few years of model progress and software progress. What has been the empirical breakdown of how much capacity, maybe as measured by intelligence per watt, how much capacity increase has come from the silicon itself, versus the models. versus other software and any other kind of big components. Yes, it's a really good question. I don't have the exact breakdown, but I would say that in our experience, most of the benefits come from model side improvements in terms of intelligence per watt. And by the way, intelligence per watt is a fantastic metric. Really focusing. We also, for us, it's good put per watt. And we can come back to why the watts need to be the denominator. Good put could well be a measure of the intelligence delivered per watt. In other words, good put is a workload specific metric. come from the model side. Software system [15:00] um quite a bit. Why? Because they're able to actually make sure that the hardware that you have is being used effectively. And then the hardware, it is pretty stunning. In other words, we're living in a world right now where 2x or more. year-over-year performance improvements is absolutely possible. And so the hardware does support it, and it is a, if you want, it's a free multiplier, not free, but everyone else above it can count on it in terms of that being a multiplier that lifts everyone else up year over year. I'd love to talk a bit about the TPU program and co-design. So Google started building custom silicon more than a decade ago. That was a contrarian call at the time. How has the TPU program evolved? Quite a lot. I mean, when the program started 2013, it was really a contrarian call. And I mean, I think that at the time, it's hard to put yourself back into the moment 2013. Conventional wisdom, you know, all the smartest, wisest people would say, you don't build a custom built accelerator for a single workload. Why? Because Moore's Law is there, doubling of performance every whatever it is, 18 or 24 months is there. You get to leverage standard programming models, all your C++ code, Like the bitter lesson of chips. Yes, exactly. It's like the bitter lesson of chips is that specialization never wins. But in this particular case, it was we have this one application, or a few, a small number of them, that would benefit tremendously and that would require an unimaginable amount of general purpose CPU to support. So really in 2013, it was a bet. And there were quite a few people, even within the company, I would say, who were not sure that it would work out. So it was a bet. It turned out to be a massively successful bet. So the first chip was all about inference. The second chip was, hey, actually we can take the same idea and build a training chip. Thank you. From there, it got picked up for more and more use cases. Around the time the second chip came out, transformers were invented. This was a major, major moment, and that actually completely shifted the TPU program. We, of course, learned that recommender systems could run really, really well on the TPUs, and so ads and related use cases came along. So I would say that change in evolution has been one of expansion in scope and impact. In other words, it started with a really impactful use case of a couple of applications for inference, serving. This was language translation primarily and voice recognition. to then training to then transformers, recommender systems, and continuing to generalize as the Gen.AI moment took hold to ever larger, ever more scalable systems as well. You mentioned the moment the transformer came through being a major moment. I'm curious to explore The TP was like specific architecture, but it wasn't specific to the Transformers specific. How do you straddle the fine line of how specific you want your chip to be for the workload? - This is a great question, and I think that it really comes down to the applicability. I mean, I think that of the chip to a particular workload. So in other words, in the end, we've been thinking about this for every generation. Would we further specialize? The question that we were facing, let's say, two years ago or a little bit more was, In 2026, should we have two chips? or one. We could have one ship that could do inference and training simultaneously and do both quite well. Or we could have two chips, one that was further specialized for inference and the second one that was further specialized for training. This analysis and this work led to the release of two chips this year, 8i for inference, 8t for training. And in the end, what we realized is that It wasn't the case necessarily a couple of years ago, but By 26, we saw inference and serving really taking off. And so having a chip that would be significantly faster for serving that we thought might be 30, 40, 50, 60 percent of the market in its lifetime. Thank you. started making a lot of sense for us. Relative to if we projected inference to be 2% or 5% of the market, even if that specialized chip is, let's say, 2x faster, into it might not make sense, because you'd actually go for the one general purpose chip, That's not fully optimized, but that's okay because you now have the uniformity, etc. So really, it does come down to a calculus question of, okay, would you further specialize? How big is that workload? How big is that workload projected to be in a couple of three, four years? Is this a sustained growth for a particular workload? In hardware, again, [20:00] As you know, there is this opportunity where The more you specialize to a particular workload, the less flexible it is. the faster. the more power efficient the hardware is going to be. So it is this art and it's this projection of what are you designing to and how persistent is that workload. In other words, if it's going to go away after a month or two months or three months, even if it's big for those three months, you got a really narrow window to intercept it. So it has to be somewhat durable. And you have to be able to project exactly what win you can get for specializing to it. So the rough trade-off is there's a large fixed cost to support a new program. Yes. And you basically have to think that there's going to be enough demand for that specific program to justify... Exactly. And to some extent, for example, for our 8i and 8t chips, Both chips can do the other work with it. This is also key if 8i could only do inference. and I couldn't do training at all. In other words, I had zero performance for training. and then vice versa, if AT was amazing at training, had zero performance for inference, that would have also been a key limitation. Why? Because we would have to predict ahead of time, over a six-year period, lifetime of the hardware, exactly how much we need of each. In this case, it was nice because both chips are better at what they're specialized for, but if you needed to, if you had leftover capacity in one place or the other, both can actually do the other's job. Depending on the specialization you do, you might get so yourself from being flexible, fungible. And if the rough tradeoff then is kind of size of program on the other end, like it seems to me that so much of the modern AI market is transformer based. And so maybe just ask a provocative question, why not just burn the transformer architecture into the chip? Yeah, so I think that it's transformer based, but then the next level of question would be transformers in the end are about the vector and matrix multiply operations. And, you know, a soft max, there's a range of essentially linear algebra primitives. We have those and others do as well, roughly baked into the hardware. It's all transformer-based, but then your model architecture is, okay, how many layers do you have? How do you go through the layers? How do you go across the layers? For each dimension, what... exact shape of matrices and vectors are you applying, you could go further. and specialize to not just the transformer, but your model. That's the next level of specialization. I think there are a number of companies out there that are thinking about that. I think it's a very, very interesting direction as well. Okay, so at this point in time, do your customers view TPU and GPU as roughly fungible equivalents, or is there a certain set of problems that is better suited for one or the other? There's for sure a better set of problems that are better suited for one or the other. I mean, GPUs are more general purpose than TPUs, for one. That is clear. And they're incredible products at Google and Google Cloud. We sell a lot of GPUs. We use GPUs internally. the specifics of your problem. So I think that our customers, there is overlap between the two, but our customers then evaluate their workload and evaluate their options. What we like to do at Google is give our customers choice. In other words, we want to have the right solution for their needs and of course provide the solution that best meets their needs for them. What is the case for co-design from the chip to the network to the software? And then what is the case against co-design? Yeah, so there's huge optimization opportunities with co-design. And so what you can imagine is that if you have a n-layer stack, and you want to be able to pick and choose whatever component you want. Let's say you're running across many clouds or you're running across many pieces of hardware, many pieces of software. You could design abstraction layers. That would basically say I can run on any hardware, I can run on any software, I can run on any network topology, any amount of network that you give me. No problem. And my system is going to be fully adaptive. Very likely, so now you have a great capability. You can move anywhere. Like new capacity becomes available. Overnight, you're up and running. because you've designed your system that way to actually be able to take advantage of anything. You have not hard-coded or specialized at all to anyone's particular infrastructure. Downside is you probably leave a lot of performance on the table. If you become fully fungible, if you become fully flexible to anybody's hardware, software, network, storage, computes, etc stack. So the case for co-designing is between each one of those layers, um, There's a big impedance mismatch if you try to get full generality. And so there might be 10% 20% 2x across each of these layers. [25:00] You start multiplying those optimization opportunities through, and all of a sudden you're left with a big end-to-end opportunity in terms of whatever you want to say, intelligence per watt or good put per watt. that you can leverage, even again, all the way down to power delivery and power availability, software optimizations, etc., etc. So pros are... You can run anywhere, anytime. you have no lock-in Thank you. Cons are you're leaving significant, in all likelihood, significant amounts of performance on the table. Hmm. So my understanding is that if you take open AI and anthropic, OpenEye was primarily building on a homogenous compute stack and Anthropic was building a roughly more heterogeneous compute stack. Do you think that's part of the reason that code design is part of the reason why they converged on what is rumored to be very different architectures for their models? I can't speculate. I don't want to speculate in terms of what OpenAI and Anthropic are doing. It's just one possibility. But I think that without knowing the details of what they're doing, I would imagine that there could be many reasons for why they wind up with different architectures. Well, what about for Google then? I'm curious what the working relationship looks like between your team and DeepMind, kind of who's in the room, at what stage of model development, and how are you making decisions together on co-design? And frankly, gratifying parts of being at Google is the opportunity to work really shoulder-to-shoulder with the DeepMind team in terms of co-design of our hardware and models. There's a third element to it in terms of that we also get to extend that with the consumer services and cloud. I'll put that aside for a moment. I can come back to that. But with respect to DeepMind, it really is a deep partnership. So let me, I mean, from the past, I can give you examples where... uh, They have come up with model optimizations. let's say, to transformers or to particular math that they might want to do. And we might have a chip in progress. Not quite done. Thank you. But then they say, oh my gosh, if we had hardware support for this. Yes. our end-to-end training or serving might get significantly faster, more efficient? What would it take to actually now change our hardware definition that we might be in execution on to accommodate? And this then leads to our engineers and researchers getting together intensely in the same room for... a few days, a week, two weeks, saying, Yes, we can do this. More likely, We couldn't quite do what you wanted. but we can do this other thing. And then maybe you can change your model architecture in this other direction. That gives you 98% that gives us 90% of what we're looking for. And yes, we can then go back and intercept the hardware. And or we can say, you know what, we're going to delay the tape out by a week. or two weeks because wow for this level of benefit It's totally worth it. You know, similarly, when we're... projecting our roadmap out and it's time for, you know, we have many generations of chips that are essentially in progress at any point in time. We have the chips that are in production. That's one. We have the chips that we're working on getting into production. They're already back from the manufacturer and we're debugging them, making them work. We have the chips that are in implementation. that are about to tape out and go to the manufacturer. We have the chips that are in design phase. And then we have the chips that are in concept phase. So it's really this... five, six stage, many, many year pipeline from in production to in your mind, etc. The collaboration with DeepMind in production is significant. because we get to work together in maximizing delivered intelligence or delivered good put per watt. And we know exactly what's happening in the model, and we know exactly what's happening in the hardware and everything in between. We also get to collaborate deeply on, though, the chips that are actually just about to tape out. Why? Because we can intercept. and we can make changes to the chip, literally chip architecture in flight, which would be somewhere between hard and impossible to do if we were working across company boundaries. Not impossible, but it would be much harder for us to say, "Oh my gosh, we're a few weeks or a few months away from getting this chip done, and now let's get in the same room, shoulder to shoulder, and figure out if we should disrupt the program." It's possible, but harder, is what I would say. And then, of course, for the chips that are in design, we have many architectures we can evaluate together. We can then ask our colleagues in DeepMind, Where do you see model architectures going? in two, three years time. Here's the Pareto of things that we can do. Here's the Pareto of where model architecture is going. We have actually deep and significant simulation infrastructures that can predict how the workloads are going to map to [30:00] different hardware architectures. deep, deep and fast iteration. It really isn't The teams are working separately. It's in the same building, same rooms many times, deep daily interaction. I talk to whether it's Cori or Demis multiple times a week, et cetera. So it really is a super fun aspect of the work. That's awesome. And eventually their models will help with chip design. We're using Gemini to design hardware for future Geminis as well. That's really cool. I want to come back to that a little bit later. I guess, is there an impedance mismatch still of just, I figure the cycle times are probably just slower? when you're dealing with hardware than your deep mind folks get to deal with on the software side. And I figure your planning cycles are much further ahead. For sure. And so how much room do you actually have to adjust? Yeah, it's a really good question. We are planning hardware two, three, four, five years in advance. No question. In other words, we talked about TPU-8, I, and AT just now that we announced, but you can imagine that 9, 10, and maybe some others are in concept execution to something else. And so they might be years out. If you're working with model architecture, by default, you're not going to be thinking years out. But I think this is also the great thing about how the company has grown up together. And, you know, in other words, Google Research and DeepMind and Transformers being invented. all of this was happening in these overlapping rooms as well. So in other words, there's a whole... generation of researchers who've grown accustomed to being able to influence the hardware. and knowing that the hardware is operating on multi-year cycles. And they also know that, look, if they have a small tweak that's going to deliver like 1% or 0.5% or something like that for a chip that's about to tape out, they're probably not going to come to us. Because they know enough to know that actually it's not like software where you can just do a change list and it's going to ship to production in two weeks' time. Like, actually stopping a tape out is a... Big deal. But they also know if they've got a big like a really good opportunity and Yes, we're absolutely going to work together to figure out if we can get it in there. So as I said, it's multiplicative in terms of where the benefits come from. The hardware does lift all... all the tides, right? And so significant portions of the DeepMind team, we're so grateful for it, it's an amazing team, but significant portions of it are thinking about how do I influence the roadmap because it's actually a pipeline. Like the idea I had two years ago, like it's in production now. And it's helping all the workloads at Google go faster. Like that's a good feeling. Totally. It seems like an impossible task though to predict in five years what workloads will be most common and what algorithmic breakthroughs will have occurred. Doesn't it seem like an impossible task? I hear you that it would seem like an impossible task, but here's the awesome thing. And actually we're working on writing this in great detail and it's a lot of fun to do it. The stunning thing is that the TPU architecture at a medium level of detail, not at a super high level of detail, at a medium level of detail, hasn't really changed since TPU v1. One way to look at it is the instruction set architecture for a CPU. Like you've got loads and you've got stores and you've got ads and you've got subtracts and branches and like... And yet, what has the software on top of it done over that period of time? Same thing with TPUs. We have some fundamental instructions and some fundamental primitives. Of course we've extended it. It's not like the instruction set hasn't gotten changed at all. But the primitives there in terms of specializing the numerics, two very big, very large matrix multiply units, a sparse core that manages vector operations and scatter gather operations, etc. There's five or six things that really define a remote load store. That's another one actually. We can read and write remote memory through our ICI network. The fundamentals have been there, and they've extended to many generations of models, many generations of even deep neural network algorithms and model structures. Okay, speaking of shifting workloads, it seems like one of the biggest changes in workloads over the last year or so, I think it really started at the beginning of this year, this calendar year, was kind of the rise of the long horizon agent. Yes. And... I would guess that's a very different shape of workload than the quick turn LLM conversations of years past. What does that mean in terms of data center needs? Yeah, so I think there's two huge aspects to this. One is it's no longer human to, whatever you want to say, accelerate interaction. In other words, when you are typing at prompt on a web browser or on your phone or whatever, of course, in response to your prompt, There's going to be a bunch of work that happens. But then when the response comes back, you've got to read it. [35:00] You've got to think about it. And then maybe you have a follow up. that's going to be multiple seconds of interaction time. Now in Deep Horizon Agents, there's no human loop that is going to naturally rate limits how quickly requests are going to go to the model. So what went from seconds, maybe tens of seconds in terms of interaction time, is now going into perhaps milliseconds. As soon as I get a response back, I can parse it. I can reason about it perhaps a bit, and I can figure out what my next request is going to be. So that's big change one. Big change too is all that reasoning and all that parsing is going to probably happen on a CPU. And that CPU is then going to have to probably think about, okay, what other state do I need to go gather before making my next prompt back into the model? Like, in other words, I've learned something from this response. I'm going to take another step, but I actually need to go grab some context. Maybe from DRAM, local to me. Maybe from someone else's DRAM on another CPU. Maybe on SSD or maybe on HDD somewhere else. So now a huge amount of orchestration has to take place as well. So the design actually is changing pretty significantly where the demand for accelerator compute is going up. I But the demand for CPU and networking and storage, the traditional data center, CPU, et cetera, is also going through the roof. Does that mean you're putting more CPUs alongside your GPU racks then? So this is the GPU, TPU racks. This is the key question is going back to this question of optimization and specialization. density and network requirements, let's say, of a TP rack relative to a CPU rack. A TP rack is going to be more dense than a CPU rack. It's probably going to need more networking than a CPU rack. So in other words, now we're building design. is going to change. Another option is maintain your uniformity, put your TPUs or your GPUs all in one building. and the building next door, maybe, put your CPUs. And maybe, by the way, the hard drives have to be in another building on the other side because they have yet another set of requirements. Now you need networking between these buildings at a pretty significant level. So once you leave a building, the networking complexity goes up significantly. From a reliability perspective, from a cost perspective, latency goes up. probably acceptably, but still now you might go to hundreds of microseconds, potentially more with queuing between the components. So the considerations do change in pretty interesting ways. Your job is hard. It's fun. It's fun. Yeah. What's happening on the networking side? And I've heard that Google's always been at the forefront of the newest in networking, including optical. Can you say a word on what is the state of optical networking? 16 years ago, were among the first to bring essentially what's called wave division multiplexing, where you could put multiple signals on a single fiber within the data center. and we actually leveraged that for all of our communication between racks. At the same time, we introduced technology along with wave division multiplexing called optical circuit switching. And essentially what optical circuit switching does is in contrast with traditional electrical packet switching. What it does is it transmits and moves data entirely in the optical domain. The great thing about that is, and so let me describe the technology that underpins it in a packet switch. You would take a packet that has a header. You would look at it in the electric... code domain, figure out where the packet is headed. In this header, it might have an IP address that says, okay, where do I head it? You look up, okay, for that destination, you look up in a table, what ports do I forward it along? So basically billions, billions and billions, perhaps trillions of packets coming through at super high speed per second. You're forwarding them along. Optical circuit switching says I'm not touching these bits in the electrical domain. What I'm going to do is I'm going to figure out for an input port which output ports to send the light to. Okay, and now I have, there's multiple ways to do this. The one that we started with is called MEMS switches, microelectrical motors that basically control mirrors in 3D. So we can now programmatically take a box that might have you pick 128 ports, 256 ports, some number like that. And we can configure every input port for a fiber that comes into it. map it to an output port. then change the rotation of mirrors where the light literally shines down on these mirrors and gets reflected [40:00] to the right at the port. Initially, the reason we did this twofold was to create locality. between groups of racks. Let's say I had a Compute cluster. and a storage cluster. and both of them were in support of, again, making it up search. We knew that these two clusters would talk to each other a lot. So we would configure the mirrors. to create shortcuts purely optically between those two clusters. clusters of racks. That was reason number one. Reason number two was we wanted to be able to expand the network and contract the network. without actually moving any fiber. and without working into the details, I could draw this on a board, the optical circuit switch would allow you to actually reconfigure the spine of the network. to expand it or to shrink it without a human being having to do anything that'd be literally a controller. that would manage that. Now fast forward to TPUs, TPU It has a Taurus topology that connects all the TPUs to one another directly. I talked about throughput and goodput earlier, one of the things that we can do is if we have a TPU rack that fails. We can replace it with another TP rack. without moving any fiber. Again, I'd have to wave my hands or draw the whiteboard. But essentially, we can say we have a spare rack available at all times. And when a rack fails, we redirect the light to that new rack. And that can be done in milliseconds. Why do you have the fiber at all then? As opposed to fully free space. Yeah, it's a very good question. Attenuation loss and the bandwidth would drop dramatically. Furthermore, across the range of a very large building, Aiming everything without the benefit of fiber to connect it all together in 3D would be challenging to possibly impossible. We've talked about it. We've talked about it, actually, and there have been some really interesting discussions. But yes, it's mostly in fiber. But then when they hit the optical circuit switch, essentially the light then literally shines down on these tiny chips. So that's one big direction of networking. But there's a lot. is exploding in terms of its capabilities and need, frankly, in the data center. So cool. We've got an entire conversation on that. Yes, it is very, very cool. Even like going back to the picture from Ineffable, it was all the cables that cut out of these eyes. Exactly. Yeah. And those cables, eventually some of them wind up at optical circuits, which are data centers. Makes sense. Okay. I want to flip to talking about power. You've been talking about good put per watt and other units per watt. It makes me think per watt means power is kind of the binding constraints or the scarce constraints or the expensive constraints in some way. You know, so I get asked this question, what is the biggest constraint that we face? The reality is there is no single biggest constraint that we face. They're all constraints. They're all super hard. It could be – and they shift continuously. But if I had to answer fundamentally, I would say that power is the single most fundamental constraint that we face. Like everything else seems – We know how to solve them, and it's a question of solving them over some period of time. Power, I think the way that you put it is really nice. It's a binding long-term issue that we do not have a... I mean, nuclear. Perhaps nuclear is going to abundant clean energy, which would solve a lot of problems for sure. When that happens and when it happens at scale, still unknown. And so how does it work in practice? You know, you're standing up a new data center, It needs... gigawatt of power, I imagine you can't just call PG&E and say, hey, please send a gigawatt of power. So what does provisioning power actually look like? And are you having to actually vertically integrate all the way down to doing your own turbines? Or how do you solve the power bottleneck? Yeah. Again, it's a big question, an important question. Our preferred model at Google is always to be utility connected, to be grid connected. So it's the equivalent of calling your... Oh, so you can't just call PG&E. Exactly. You call very politely your favorite utility, wherever it is that your data center is. And of course, you give them many, many years of notice. So in other words, if we're talking about a gigawatt scale, it's not something that you can say, hey, I need a gigawatt tomorrow. When can you start billing me? It's something that we co-plan together. You know, for us, it's something that we also take very seriously from the perspective of ensuring that when we work with these utilities, the costs of putting that infrastructure in place. We could get into a long conversation just on this topic as well. that we cover those costs because with the way that billing works, it actually could be that by the act of the utility building up capacity for us, let's say, other people's rates could go up. In theory, what we ensure is that actually, let's say the transmission [45:00] - That have to be built, upgraded, additional utility base stations, et cetera, that we pay for those as well. But so it is a long planning process. It can absolutely be the case that let's say that we need a gigawatt in I'll make up a date, 2028. and the utility can get us a gigawatt in 2029. that can get us, let's say, 700 megawatts in 2028. So now we might be left with a question of how do we cover Those 300 megawatts. Well, one answer is just wait. Another answer is to say, okay, well, would we figure out how we generate some subset of that power ourselves? And maybe that would be with solar cells. or with batteries as a backup, et cetera, would it be for other sources? So then we again work with the utility. So it might also be a combination where we might maintain some power generation local and have the capability, even when the utility comes online fully, let's say the gigawatt scale, where we could actually provide power back to the grid. So having that when they need it as well. So in other words, there might be the one way to look at it is there might be the two weeks of the year. Maybe it's the hottest two weeks of the year. where there's a huge amount of residential demand. If we have some local generation, we can then provide that back to the grid as well. So it really is working over multiple years with the utilities for us. Why is your preference to do that versus to... kind of go, like, to work with the utilities as opposed to to kind of vertically integrate yourselves? Yeah. So the main reason is flexibility and uplift on both sides. So one way to look at it is statistical multiplexing, or if you want the law of large numbers, if we need to have a gigawatt of power, let's say we wanted to have that with 99.99 percent plus reliability. Pi means we have to build two gigawatts of power. At that level, 99.99 or 99.999, you have to have one plus one redundancy. Thank you. That gets expensive. And then making sure that that's going to be ideally a clean energy source. probably right next to our data center, that can also be challenging. Now if we partner with the data center, again maybe we have some amount of that that we can bring ourselves. that we can give to the grid when they need it less, that we can take their power. So in other words, by leveraging that statistical multiplexing over a much larger base, actually everybody wins. We win. the good ones. residences when etc. We prefer that. I mean in rare rare cases we will do it let's say behind the meter but even then Thank you. We're doing it under the assumption that we're going to work with the utility where it might be a year later. Yeah. We want to be connected to the grid. Again, it's an uplift for us. It's an uplift for the grid. Makes sense. How did you decide how big to make a data center? Yeah, that's an art and it's a source of significant debate. You know, at some point, you know, we had a debate, I remember, even 10 plus years ago, 15 years ago, big debate at Google. Should we just put everything into one data center? Like back then it was going to be a gigawatt, which was-- - Simpler times. - Yeah, different times, like 10, 15 years ago. We were going to have a gigawatt data center. So one obvious concern there is single point of failure, right? If you're looking at things from a 30-year perspective, that's a huge concern. On the other hand, if you're talking about training workloads today, bigger is better. In other words, having more, because from a networking perspective, actually, you want to have things concentrated in as small a distance as possible. But then again, two issues, single point of failure, but then also now power availability. Well, a gigawatt was huge 10 or 15 years ago, but imaginable. There's no way anywhere in the country or the world we're going to get the total requirements of Google built in one place. It's just not going to be possible. So now what's the optimum size? Again, we have models for this, simulators, etc. But this... It also depends on, look, in some places we're going to be at the edge of the network or we're going to be in country. That might be tens of megawatts. we actually partner with ISPs that might be Iraq. Like literally it could be Iraq. This is a training cluster. Okay, maybe that's going to be closer to a gigawatt. Some other sites might be hundreds of megawatts, et cetera. I love to understand how you think about the... portfolio lifecycle management. So I guess my guess would be that the biggest, newest clusters are used for training the latest frontier model. And then you kind of recycle the other stuff and run inference on it. Is that like a fair framework? Are you also standing up inference-specific clusters? How does that all work? Very reasonable framework that you're thinking about. And I think makes a lot of sense. But I would say that the demand for inference is such that we can't just that are no longer being fully used just for training. [50:00] As the basis for inference and also if you think about it, we might centralize again in a particular year into a small number of large sites, our training clusters. And there is benefits to keeping the network distance between them small. So now, wherever they're located in the world, they're probably going to be, let's say, on the same continent, or they might be even in the same portion of a continent in a particular year. So now you might be left with not enough serving capacity on the other continents, etc. So then, okay, actually, we then have to go build this specialized inference elsewhere across the world. wholly. In other words, it really does have to be, okay, here are the training clusters. Yes, probably they're going to be used for serving in some number of years. But then we're also having to build the serving clusters as well. And your serving clusters, are they different than the training clusters? Like, are they smaller? Are they cheaper per megawatt? Not necessarily cheaper, actually, because for serving, there is more need to co-locate compute networking and storage. So for training, you actually have this big density, uniform deployment, et cetera. Whereas for serving, you're going to have to have the mix of storage, compute, and accelerators. Furthermore, it's a very interesting aspect of this that we could go into more detail on as well. But for serving, you actually don't want to have too much in one place. You want to be serving your workloads from all over the planet. model endpoints, and there actually can be variations of models. So now we have to spread these models out across the world also accounting for locality. So they will be smaller. They'll actually be less vertically integrated, probably. and the inference slide. Super interesting. So if you made a data center five years ago, you know, state-of-the-art accelerator that five years ago was very different, vastly less efficient than the ones that are being made today. Yes. Do you actually go back and swap out the chips in those old data centers? And this kind of relates to there's been an ongoing debate, I think, of what is the actual useful life of a chip. Yeah. So I've been on record as saying this. And one of my more surprised by how much pickup this had. But I said that our seven and eight year old TPUs are still at 100 percent utilization. Wow. I think I saw this. I didn't know this from you. Yeah, I didn't mean for it to be a meaningful statement, but it was a meaningful statement, apparently. Yeah, so our older GPUs, but our older TPUs are seeing significant utilization. We do replace them. In the end, there's a question of, okay, depreciation lifetime is approximately six years. them and upgrade them. It's not that we do replace the chips, but it's really the systems. So in other words, we think in terms of pods. So a TPU, let's say AT pod might be 9,600 chips. and it might be 140 something, 152 racks, et cetera. So then we would say, okay, we're going to pull out that pod and we're going to replace it with a... whatever it is, TPU 12 or 13 or 14 pod. It might not be a perfect fit. In other words, the new pod's footprint might not be a perfect fit for the old pods. vacancy, we then just have to account for that. We have to figure out how we would retrofit. We can't plan that because we don't know what that many generations of TPUs are going to be. So then it's again a bit of an art to figure out how we would, and it's actually hard work, figure out how we would decom and then replace as quickly as possible with new TPUs. You've written publicly about open standards. Can you say a word on that? Yeah, so I think that's the, we talked about this interoperability question. And so for us, while we support vertical integration, and we make it possible to, you know, extract just as much performance as you want, for us, it's really important to not force lock in and not force a sort of walled garden in terms of how the system works end to end. So, I'll give one example. At Google, we've developed a model development framework called JAX. We like it a lot. We think it's really, really good. And we use it extensively internally. Many of our customers like PyTorch. And so one thing we could say is, hey, if you want to run on TPUs, you got to use JAX. "It's the best." I'm being facetious a little bit, but maybe it is, maybe it isn't. We think it's the best. And that's your only choice because it's so great. Or we could say, "Look, if you like Jacks, We like jacks. If you like jacks, you can use it. But if you like [55:00] which We have Torch.DPU. where your unmodified models, et cetera, can run. We've seen this throughout history, actually with many, many examples. I've used the example of IP. Why did the internet protocol win? There are actually many competing protocols to IP. This is ancient history in the 70s and early 80s. Why did IP win? Because it was... open standard and it was this narrow ways to the hourglass where anything could run on top in terms of software. and anything could run underneath it in terms of hardware. So open standard, interoperable, anyone who brought IP. could plug into a router port. and become part of the internet. It was beautiful. And that's what allowed the internet to explode in its growth and reach across the world. So we really do believe in those open standards in terms of plugin points. Now, if you want to plug in something highly specialized... You can. If you think that you've got a better thing to plug into our framework, you absolutely can. But we want to really support open standards, ideally open source around it as well. It's too big and too important to build out. to be any one vendor's closed proprietary stack. Yeah, we really believe that. And it's got to be one of choice. And that's also why, for example, we fully support and have TPUs, GPUs, other accelerators, etc. Yeah. How has day-to-day life for your team changed with AI? Where is it changing your function the most? So I think the Whatever you want to say, the easiest answer is on the software engineering side. where it's been documented externally quite a bit. So I think that we are at Google and my team using it to great effect in terms of our software development capabilities, but also, frankly, tests, rollouts, even helping with design, etc. But there have been on the hardware side, which has maybe gotten less external coverage, also significant change. In other words, my hardware engineers, I was just looking at the numbers earlier today, in fact, are using AI, using whatever the token counts, if you want, not the best metric, but it's still a metric. Our hardware engineers are using AI as much as our software engineers are. And productivity has gone up. Time from design kickoff to tape out. It's shrinking. Thank you. Time for bring up is shrinking. So in other words, productivity is going up significantly. But then other maybe less expected changes, how we do data center design. has changed significantly. In other words, how we evaluate, you mentioned, hey, do we do a gigawatt building or 100 megawatts campus or 200 megawatt campus? In the past, this would be very detailed, very sort of spreadsheet-driven, human-driven processes. And it still is to some extent, but there's a lot of AI now involved that really streamlines the process in terms of... Planning. and development as well. Interesting. There's like reasoning models or? I wouldn't say quite yet reasoning models. It's not replacing human judgment, but it's making it much easier to bring the necessary information together in one place and basically put the right information in front of people. the humans making the decisions. Okay, I'm going to bring us home with two kind of out there, kind of fun questions. Question number one. orbital data centers. I've seen that Google is thinking about this quite seriously, and you don't need to comment on that. But I mean, does the fact that people are seriously running the numbers on orbital computes mean that there are kind of serious binding constraints on Earth? And what do you think of orbital data centers? earlier part of the conversation where you raised, I think correctly, that in terms of fundamental constraints, energy and energy production is a key, key challenge. And so really, bottom line is from a fundamentals perspective in space. you have something like 40% more power available because of lack of attenuation in the atmosphere, etc. In other words, it's just more solar capacity, 1.4x. So that's one part of it. But in a sun-synchronous orbit, you have 98%. to 100% coverage. sunlight on your solar cells relative to 28 30, maybe 35% on land. So in other words, just this enormous amount of power, obviously the sun, you take the 1.4x and you take the... 3 to 4x. [60:00] terms of number of hours per day. You remove batteries more or less from the equation and now you have the potential for something that can really end and of course carbon-free and Lots of benefits. Lots of challenges. Lots and lots of challenges. So in other words, okay, now what about cooling? Naively, you might think cooling in space is easier. It's actually harder. Reliability. We talked about how these things fail sometimes. So repairs becomes harder. Not impossible, but it becomes harder in space. Networking. Some little robot up with the cluster. Exactly. So perhaps this model is a promising direction. Redundancy will probably be your on free space optics is now going to become reality. We're probably not going to be stringing fiber between these components, so it will actually be free space. and we're going to have the lasers pointing at the receivers and calibrating in real time. There are no showstoppers here, no fundamental showstoppers. Last question. I've had this image of the... ineffable big supercomputer that you built them in my head this whole time. And so here's a question. 10 years from now, What will the most frontier supercomputer look like? Oh my gosh, yes. So 10 years is right at that edge where it's I would say... In all sincerity, Probably some people are thinking about what the 2036 computer looks like. We might have a few thinking that far ahead. But wow, the cone of uncertainty there is way, way too wide. Like in those, if you look at the trends, the level of integration is going to be tremendous. So, you know, we saw the beautiful fiber and the VR 200 racks that we put together for ineffable. My guess is it's going to be more integrated, less fiber to, I won't say no fiber in 2036, but I think that probably from a rack perspective, the rack will look much more tightly integrated and the amount of fiber. there'll be maybe a small fiber bundle coming out of the rack. And then, so in other words, one view you might have from a modular manufacturing perspective is these racks likely in 2036 are, are going to be manufactured centrally. And so whether they have 72 or 144 or 288 or probably more, 576, 1152, pick your favorite multiple of GPUs, since we're talking about the ineffable case, but then be the same for the TPUs. Deeply integrated into a rack, would a rack be multiple megawatts? We don't know exactly how, but 2036, we get to imagine things. Could be multiple megawatts in a single rack. And now you bring water and you bring power. and you bring fiber. You wheel that rack in, you plug those three things in, and you're off to the races as well. Okay, so it's not going to be like a big alien orb then? Up in space. By 2036, I wouldn't say that it's not possible for 2036 for it to be then directly launched into space. And then maybe taken by a robotic arm at some space station and then plugged into the right module. Yeah, maybe that for 2036. Fun. Fun to think about. I really enjoyed this conversation. Yes, it was a ton of fun. This is... The most... intense, biggest cat-backs build-out in history, but I also think it's... It's a technology revolution that is a thing of beauty. You have such a grasp over the beauty of the technology and the binding constraints and how to balance this all. Google is in good hands. Thank you for taking the time to share what you're doing. It's a ton of fun. We get to live through this and we get to define it. So thank you very much. It was a great conversation. Thank you. Thanks. Music playing