Google AI Chief Rejects Flops Metric as Systems Fail Hourly at Scale

谷歌AI主管摒弃FLOPS指标：大规模系统每小时均现故障 在10万颗芯片的集群中，加速器每小时故障多次，使得理论峰值FLOPS沦为虚荣指标，迫使基础设施设计必须严格针对端到端有效吞吐量进行优化。 Amin Vahdat解释了为什么在10万颗加速器规模下，原始FLOPS是一种具有误导性的虚荣指标，并强调了“有效吞吐量”（goodput）——即在组件每小时发生多次故障的情况下依然交付的有用计算量。他详细阐述了Google的芯片演进，包括战略性拆分为TPU 8i和8t，并介绍了与DeepMind的深度协同设计如何让团队能够在流片前中途调整芯片架构。Vahdat还概述了长周期智能体工作负载如何重塑数据中心平衡并推高对CPU和存储的需求、Google为何依靠光电路交换（OCS）实现自动化故障转移，以及获取电力如何构成制约未来吉瓦级园区乃至潜在轨道设施的终极硬性瓶颈。 在10万颗加速器规模下，理论峰值FLOPS已不再适用，因为物理组件故障每小时都会发生多次，使得同步任务的“有效吞吐量”（goodput）成为唯一能反映实际进展的指标。 (5:30) Google每两年翻倍的token推理服务能力中，有一半以上源自软件、编译器、运行时和模型架构的创新，而非仅仅依靠硬件层面的升级。 (11:52) 随着推理需求增长至整个集群规模的30%至60%，TPU架构分化为TPU 8i和8t，但二者均保留了双向运行能力，以避免在六年的折旧周期内出现资本搁浅。 (15:33) 长周期自主智能体正在改变数据中心拓扑结构，它们大幅推高了对CPU、多层级内存和网络带宽的非加速器需求，以管理中间推理状态。 (34:16) 电力仍然是前沿AI基础设施的决定性瓶颈，这促使Google优先考虑公共电网互联，而非需要经济上难以承受的1+1冗余容量的表后离网建设方案。 (42:35)

Amin Vahdat, Google's head of AI infrastructure, argues that the physical and economic viability of frontier AI depends on shifting focus from peak chip metrics to end-to-end operational efficiency, co-designed hardware stacks, and overcoming grid-scale energy bottlenecks. Goodput and Fault Tolerance at Massive Scale (5:30) Theoretical peak FLOPS serves as an incomplete metric because production performance across large clusters depends on memory, interconnect fabrics, and host systems working in unison. Synchronous workloads operating across 10,000 to 100,000 accelerators can be halted by an issue in a single component, triggering costly rollbacks to stored checkpoints. Because hardware and software faults occur multiple times an hour at 100,000-chip scale, Google evaluates systems by 'goodput'—the rate of useful, valid computational work delivered to workloads rather than raw theoretical throughput. The Economics of Hardware Specialization and Co-Design (19:35) Specialized ASICs provide performance and energy efficiency advantages over general-purpose silicon, but they demand durable workloads to justify multi-year development cycles. Google split its eighth-generation TPU family into the inference-oriented TPU 8i and the training-focused TPU 8t because serving demands reached a projected 30% to 60% of fleet workload, yet both chips retain bidirectional capability to prevent stranded capital. To maximize goodput per watt, Google pairs hardware silicon with tight, full-stack co-design, collaborating directly with DeepMind researchers to alter instruction sets or adjust tape-out schedules when model algorithmic gains warrant it. Optical Switching and Shifting Data Center Architectures (37:51) Unlike standard data centers designed for 20- to 30-year operational lifecycles and fungible, low-density hardware, modern AI facilities are engineered around dense accelerator racks drawing hundreds of kilowatts toward megawatts. To mitigate component breakdowns in direct torus topologies, Google uses Optical Circuit Switching (OCS) powered by micro-electro-mechanical systems (MEMS) mirrors. OCS bypasses optical-electrical-optical conversions and redirects light paths to hot spare racks within milliseconds during physical faults. Meanwhile, the emergence of low-latency AI agents is shifting data center physical balance, driving surging demands for CPUs, DRAM, and tiered storage alongside matrix accelerators. Power Constraints, Sizing Limits, and Open Standards (42:35) Vahdat identifies power availability as the ultimate binding constraint on AI infrastructure expansion. Monolithic gigawatt-scale sites introduce single-point failure risks and face immense regional grid constraints. Operating off-grid behind the meter requires 1+1 redundancy—building two gigawatts of generation to guarantee one gigawatt of compute—which is economically prohibitive, leading Google to co-plan grid connections with utilities and finance dedicated substations. To sustain architectural flexibility across this massive capital expenditure, Vahdat rejects proprietary closed ecosystems, endorsing an open-standards 'narrow waist' approach exemplified by tools like Torch-TPU that decouple user software from specific underlying silicon. While Google explores orbital data centers to tap unimpeded solar power in sun-synchronous orbits, significant radiative cooling and optical networking challenges remain. For terrestrial systems over the coming decade, Vahdat projects that infrastructure will consolidate into modular multi-megawatt racks, though power availability and long-horizon grid integration remain unresolved bottlenecks.

## Background 该内容由红杉资本（Sequoia Capital）于 2026 年 10 月 6 日在 YouTube 频道发布，由合伙人 Sonya Huang 主持。视频围绕 Google AI 基础设施负责人 Amin Vahdat 的深度访谈展开，探讨了在 10 万卡集群故障成为常态的背景下，前沿 AI 基础设施的物理极限、经济账本、TPU 架构演进与未来数据中心形态。 ## TL;DR Amin Vahdat 直言理论算力（FLOPS）只是虚荣指标，在超大规模集群中真正关键的是克服高频硬件故障、持续交付实际有效算力的 Goodput。访谈深入剖析了 TPU 芯片从 2013 年的反共识下注到分化为 8i（推理）与 8t（训练）的底层逻辑，以及 Google 基础设施与 DeepMind 在流片前夕“半空拦截”架构的软硬件协同设计模式。此外，Amin 还详细拆解了长思考 Agent 对 CPU 和存储带来的巨大冲击、利用光路交换（OCS）实现毫秒级光路重构的黑科技，并指出电力已成为 Frontier AI 最核心的物理刚性约束，展望了轨道数据中心与 2036 年多兆瓦机架的前沿图景。 ## Atmosphere & style 整场对话在顶尖系统架构师的硬核工程严谨与顶级风险投资人的敏锐商业洞察之间碰撞，节奏紧凑且信息密度极高。Amin 展现出一种举重若轻的工程自信，不仅用推导数学题擦掉重来、光线变轨等生动隐喻拆解复杂的物理与网络瓶颈，更在谈及与 DeepMind 并肩作战和太空算力设想时流露出纯粹的技术浪漫。现场氛围从微观的硅片逻辑无缝延伸到吉瓦级电网与太空轨道的宏大叙事，让人深刻感受到置身于人类历史上最大规模工业革命前沿的震撼与激荡。 ## Bulletin - [05:30](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=330s) Google 视 FLOPS 为虚荣指标，强调在 10 万卡集群每小时发生数次硬件故障的常态下，衡量端到端不被打断的真实可用产出（Goodput）才是唯一标准。 - [15:33](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=933s) 因预测 2026 年推理负载将占整体算力预算 30% 到 60% 以上，Google 首次将 TPU 架构分拆为推理专用 8i 与训练专用 8t，但保留跨任务兼容以防资产闲置。 - [34:16](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2056s) 长思考 Agent 去除人机交互等待后调用延迟压缩至毫秒级，推动编排状态所依赖的传统 CPU 和高容量存储需求急剧飙升，重构了数据中心混合布局。 - [46:32](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2792s) 受限于大数定律与 1+1 冗余成本，Google 优先选择并网而非离网自建 1 GW 发电厂，同时作为终极登月项目正严肃论证太阳同步轨道数据中心的可行性。 ## Key takeaways - [01:47](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=107s) AI 数据中心不同于 30 年生命周期的传统机房，必须与特定芯片架构进行深度协同定制，无法兼顾完全通用性。 - [05:30](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=330s) 在 10 万卡规模下硬件故障每小时都会发生，真正决定系统交付能力的考核核心是剔除回滚与阻塞后的有效吞吐量（Goodput）。 - [11:52](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=712s) Google 算力基础设施的目标是每六个月将 Token 生成容量翻番，其中软件栈调优和模型架构革新贡献的增益甚至超越硬件本身。 - [19:35](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1175s) TPU 产品线分拆为 8i 与 8t 源于推理负载占比激增的经济确定性，但两者仍保留跨界跑训练与推理的互换弹性以规避 6 年生命周期风险。 - [23:30](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1410s) 全栈协同设计通过消除芯片、网络与软件之间的阻抗失配，能够将每一层 10% 到 2 倍的性能损耗转化为端到端巨大的能效跃升。 - [26:11](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1571s) Google 基础设施团队与 DeepMind 保持多代硬件流水线并进，甚至具备在临近流片前的工程实现阶段半空中动态拦截修改芯片架构的能力。 - [32:42](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1962s) TPU 自 2013 年初代以来的核心计算原语（MXU、Sparse Core、专用数值格式、远程 Load/Store）十年未变，支撑了深度学习多代架构演进。 - [34:16](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2056s) 长思考 Agent 引发系统调用频率从秒级压缩至毫秒级，不仅加速器用量激增，更导致编排决策所依赖的 CPU、DRAM 与底层存储负载冲破天花板。 - [38:58](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2338s) 光路交换（OCS）利用 3D MEMS 微镜在纯光域毫秒级重定向光路，无需电包解包即可绕过故障机架并无缝接入备用机架，大幅保障 Goodput。 - [42:35](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2555s) 电力是构建超大规模 AI 基础设施唯一的终极物理约束，Google 采用与公共电网深度协同规划并全额承担变电改造费用的并网模式。 - [46:32](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2792s) 脱离电网自建 1 GW 发电站需要配置 1+1 热备并承受高昂冗余成本，接入电网可利用大数定律平滑风险，并在民生用电高峰时反哺电网。 - [49:23](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2963s) 由于地理网络延迟以及推理需要紧密混合 CPU 与存储，退役的同质化训练集群无法直接接管全球分散的在线推理服务需求。 - [51:55](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3115s) Google 机房硬件以 Pod 规模整机退役与进场，得益于极度旺盛的算力需求，部分服役 7 到 8 年的老旧 TPU 至今仍保持 100% 满载利用率。 - [53:53](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3233s) 借鉴互联网 IP 协议的历史经验，Google 反对构筑专有围墙花园，主张以开放标准和对 PyTorch 等框架的无缝支持作为系统的狭窄腰部。 - [58:29](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3509s) 轨道数据中心可利用太空 1.4 倍光照强度与太阳同步轨道近乎 100% 的日照时间彻底摆脱庞大电池储能，在基础物理学层面不存在致命障碍。 - [01:01:00](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3660s) 预测到 2036 年，前沿超级计算机机架将演进为单柜数兆瓦能耗与上千颗加速卡的高密集成模块，仅需连接水冷管、高压电缆与光纤束即可部署。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=0s) 专用硬件的权衡与人类历史上规模最大的资本开支 硬件架构的设计本质上是一场精妙的艺术与前瞻性预测。芯片越是针对特定工作负载进行深度定制与专精化，其灵活性就越低，但相对地，在速度与能效表现上就会带来巨大的跃升。这种设计的核心难点在于：你究竟在为怎样的负载设计硬件？这项负载在时间尺度上是否具有足够的持久性？如果一个负载虽然在未来两三个月内规模庞大，但热度过去后便迅速消退，那么架构师能切入并捕获红利的窗口期将极其狭窄。因此，硬件定制必须瞄准具有长期生命周期的稳健工作负载，并且在立项之初就精准计算出专门化所能带来的确定收益。 红杉资本的 Sonya Huang 指出，当前全球正处于人类历史上规模最大的一场资本开支（CapEx）浪潮之中。以 Google 为例，其年资本支出预计将超过 2000 亿美元，其中绝大部分直接砸向了数据中心建设。作为去年底走马上任的 Google AI 基础设施负责人（Head of AI Infra），Amin Vahdat 正是站在这一浩大工程风口浪尖的核心掌舵人。Amin 坦言，这不仅在 Google 历史上前所未见，从全人类建设史与技术转型速度的角度来看，也是极其不可思议的壮举。 ```mermaid flowchart LR accTitle: 专用芯片设计权衡 accDescr: 展示芯片专用性程度对灵活性、能效和速度的双向影响 subgraph Tradeoff["设计权衡"] direction TB A["通用硬件架构<br>(灵活、兼顾多任务)"] B["高度专用硬件<br>(TPU / 专用加速器)"] end subgraph Outcomes["系统表现"] direction TB C["灵活性下降<br>适用窗口收窄"] D["计算速度跃升<br>能效比大幅提高"] end B --> C B --> D ``` --- ### [01:47](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=107s) 传统数据中心与 AI 数据中心的本质分歧：30 年通用性 vs. 深度协同定制 从土建与外围设施来看，AI 数据中心与传统数据中心有着诸多重合之处：它们都拥有厚重的混凝土外壳、庞大的变电站（Electrical yards）、机械冷却系统、密集的配电网络，以及将海量计算节点连接在一起的高带宽网络和底层存储。然而，二者在设计哲学上存在颠覆性的差异，这个差异就是“专用化”（Specialization）。 传统数据中心本质上是一项跨越 20 到 30 年周期的不动产与基础设施投资。在这个长达数十年的生命周期中，服务器硬件可能每 6 年就会迭代一次，因此机房建筑必须具备极高的“通用可互换性”（Fungibility），能够从容容纳数代不同的通用服务器、网络交换机、存储阵列或异构加速器。但如果将这种长周期的通用性生搬硬套到 AI 设施上，整个系统就会变得极其臃肿且严重超建（Overbuilt）。 AI 数据中心是高度“定制化”（Purpose-built）的产物，建筑本身在设计之初就需要与未来的芯片硬件进行深度“协同设计”（Co-design）。以机柜功率密度和散热为例： - 传统的存储机柜功率仅在 10 到 40 千瓦之间，网络需求极低（尤其是机械硬盘架构）。 - 现代的 TPU 或 GPU 机柜功率则轻而易举突破数百千瓦，业界甚至已经在规划未来的数兆瓦（Megawatt）级别机柜。 - 如果在一排机柜空间内，规划部署 30 个低功耗存储机柜与部署一两个超高功率的 AI 加速机柜，二者在空间占位、电力母线分配、液体冷却管道布局及网络光纤结构上完全是截然不同的物理体系。因此，现代 AI 数据中心在建筑层面就针对供电、散热和加速卡拓扑进行了全栈联合优化。 Sonya 提到，Google 此前向其投资的创业公司 Ineffable Intelligence 交付了一套 Vera Rubin 大规模集群。Amin 回忆道，那组机柜的内部光纤布线呈现出一种令人惊叹的“分形之美”（Fractal nature），现场照片在社交媒体上发布后引发了轰动，成为了最受关注的热门内容之一。 --- ### [05:30](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=330s) 拒绝虚荣指标 FLOPS：以真实故障下的 Goodput 作为第一衡量基准 在评估 AI 基础设施的实际产出时，Amin 直言行业内热衷讨论的“FLOPS”（浮点运算次数）或单芯片理论吞吐量只是一个“虚荣指标”（Vanity metric）。纸面上的 Teraflops 或 Petaflops 只是芯片在理想真空条件下的理论极限，而现实工程中最核心的指标是真实工作负载所交付的端到端性能。除了单芯片的计算力、HBM 显存容量与 SRAM 缓存之外，更关键的是成千上万颗芯片如何与 CPU 和高带宽网络大规模互联协同。 针对系统效率，Amin 引入了两个关键概念的对比： - **吞吐量（Throughput）**：系统在单位时间内完成了多少计算操作（包含无效或重复劳动）。 - **有效产出（Goodput）**：系统扣除故障、重试、等待与通信阻塞后，真正对生成最终答案做出实质贡献的有效工作量。 现代前沿 AI 无论是大规模预训练（Training）、在线推理服务（Serving）还是长程智能体（Agentic）工作负载，底层都是高度同步协同的（Synchronous）。数千、数万甚至十万张加速卡需要在微秒（Microsecond）或毫秒级的时间粒度上保持严格步调一致。在这个超大规模系统中，一旦有一颗芯片因硬件错误突然停机，由于其他节点都在同步等待该节点的中间结果，整个庞大的训练作业就会瞬间陷入停滞。 ```mermaid flowchart TD accTitle: 芯片故障对大规模同步集群的影响机制 accDescr: 描绘十万卡集群中单点故障如何导致计算停滞、检查点回滚与 Goodput 损失 subgraph SyncCluster["100,000 卡同步协作集群"] N1["计算节点 1"] --- N2["计算节点 2"] N2 --- N3["故障节点 X (硬件故障)"] N3 --- N4["计算节点 N"] end subgraph FailureImpact["故障级联代价"] F1["全局计算阻塞 (微秒/毫秒级依赖中断)"] F2["定位故障硬件 + 寻址隔离"] F3["读取存储检查点 (Checkpoint Restore)"] F4["回滚重算此前状态 (损失计算时间)"] end N3 -->|"单点崩溃"| F1 F1 --> F2 F2 --> F3 F3 --> F4 ``` 一旦发生故障，系统就必须暂停运转，排查故障源，定位并加载此前的存储检查点（Checkpoint），将集群状态回滚并重新启动；在某些严苛的推理场景下，甚至不得不完全从头开始计算。Amin 打了个生动的比方：这就像一个人在纸上推导复杂的数学题，做到了第四步发现第一步算错了，不得不擦掉回到起点重做——虽然你一直在纸上拼命书写（有很高的 Throughput），但对于算出最终答案而言，这些重算完全是无用功。 在 10 万颗加速芯片的恐怖规模下，物理现实是极其残酷的：芯片每一小时都会出现数次硬件级故障。这些现代芯片本身就是人类半导体制造工艺的物理极限奇迹，它们通常不再是单一硅片，而是由 2 个、4 个、8 个甚至更多的小芯片（Chiplets）与周围密集的 HBM 显存堆叠封装而成。在如此高密度的物理前沿上，组件故障是确定性的常态，而非偶然的意外。因此，Google 衡量自身工程能力的铁律，不是纸面上理论算力的多少，而是在高频硬件故障持续发生的环境中，系统究竟能交付多少不被打断的真实有效算力（Delivered Goodput）。 ### [09:48](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=588s) 10万卡规模的故障常态与“有效吞吐”（Goodput）衡量标准 在拥有 10 万颗加速器的超级数据中心中，故障不是“会不会发生”的假设，而是不可避免的日常规律。系统涵盖了共封装光学器件（Co-Packaged Optics, CPO）、成千上万的高速线缆和极端复杂的通信拓扑，任何一个微小组件都有可能损坏。在十万卡级别下，系统不仅是每天发生几次故障，依据具体配置的不同，甚至可能达到“每小时故障数次”的频率。 面对这种物理现实，运维体系必须具备近乎实时的感知与毫秒级的恢复能力。然而，大规模遥测监控（Telemetry）的难度犹如持续在每秒、每分、甚至持续数周的训练周期中“大海捞针”。因此，Google 不用峰值算力（FLOPS）这种虚荣指标来自我衡量，而是以**Goodput（有效吞吐量）**作为核心考核基准——即在发生真实硬件与软件故障的恶劣环境下，数据中心为核心业务负载实际交付的有效工作量。虽然 Goodput 最初是 Google 内部提出的术语，但如今已被整个 AI 基础设施行业广泛采纳。 ```mermaid flowchart TD accTitle: Goodput Telemetry and Recovery Loop accDescr: 10万卡规模下从故障检测到保障有效吞吐量的闭环机制 subgraph ScaleLayer ["100,000 加速器超大规模基础设施"] A["物理与软件层扰动<br/>（网络抖动 / 编译 Bug / 硬件衰退）"] end subgraph TelemetryLayer ["实时遥测与恢复系统"] B["秒级/分钟级高频遥测<br/>（大海捞针式异常检测）"] C["近实时自动切流与恢复<br/>（毫秒级光路重路由）"] end subgraph MetricLayer ["业务核心考核指标"] D["避免虚荣指标：FLOPS 峰值算力"] E["核心考核指标：Goodput 有效吞吐<br/>（真实交付的有效功耗比）"] end A --> B B --> C C --> E D -.->|"取代"| E ``` - **故障的长尾效应**：故障并非集中在单一常见元器件上，否则团队早已将其彻底修复；新产品导入期往往伴随大量未知长尾问题。 - **软硬件交织的根因**：很多故障源于超高速互连的物理网络极限制约，但更多问题来自软件栈——诸如编译器 Bug、运行时（Runtime）缺陷、操作系统故障或模型自身逻辑异常。即使硬件 100% 完美无损，软件层面的故障同样会直接拉低全链路吞吐。 --- ### [11:52](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=712s) 参考技术栈与每六个月翻倍的 Token 生成能力 对于大规模集群构建，半导体厂商通常会提供参考设计。Sonya Huang 提问行业客户是完全遵从厂商蓝图，还是必须进行大量自研定制。Amin Vahdat 指出，像 NVIDIA 这样的企业早已超越了单纯的半导体公司，而是能够交付极其强大的整机全栈参考系统（Reference Stack）。然而，无论是面对 NVIDIA 还是 Google 内部的 TPU 参考设计，大多数客户在采用官方标准栈的同时，只要业务具备独特性，就一定会针对自身特定用例进行定制化调优与重构。 关于 Google 算力基础设施的扩展速度，Amin 阐明了其核心目标：**每六个月左右将线上推理服务（Serving）的有效可用容量翻一番**。这里考核的不是单纯的物理硬件 FLOPS 翻倍，而是实际产生 Token 的输出能力（Token Generation Capability）。 - **算力提升的动力构成**：Token 生成能力的指数级增长，软件调优带来的收益甚至超越硬件升级本身。这并非依靠单点技术突破，而是数百项编译器重构、模型剪枝调优以及运行时流水线优化的持续累加。 - **“每瓦特智能”（Intelligence per Watt）与能效**：由于电力已成为数据中心建设的核心硬约束，所有有效吞吐都必须以“瓦特”为分母进行归一化度量。在模型、软件与硬件三大支柱中，模型侧的架构革新往往贡献了绝大部分能效增益；而系统软件则负责将底层算力彻底挤压出来。 - **底层硬件的倍增器效应**：当前硬件技术依然支持年化 2 倍以上的性能提升。对上层算法而言，底层硬件迭代就像一个近乎“免费的算力放大器”，持续抬升整体系统产出的底线。 --- ### [15:33](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=933s) 算力演进史：从 2013 年 TPU 孤注一掷到 8i 与 8t 的分道扬镳 2013 年 Google 启动 TPU 自研芯片项目时，这在业内是一项极其反传统的边缘决策。当时业界的普适共识遵循所谓的“芯片苦涩教训”（The Bitter Lesson of Chips）：通用 CPU 配合摩尔定律每 18 至 24 个月性能自动翻倍，并坐拥庞大的 C++、Java 与 Python 软件生态，针对单一业务构建专用加速芯片注定难以与通用技术曲线抗衡。然而，Google 意识到少数核心工作负载的算力需求呈指数级爆发，如果完全依赖通用 CPU，所需的服务器规模与能耗将达到难以承受的天文数字。 最初内部亦充满质疑的 TPU 赌注取得了巨大成功。芯片的发展路径经历了数次关键跃迁： | 发展阶段 / 芯片代际 | 核心定位与业务驱动力 | 技术演进特征 | |:--- |:--- |:--- | | **第一代 TPU (2013-2015)** | 纯推理芯片 (Inference) | 专为语音识别（Voice Search）与机器翻译设计，支撑初代深度学习服务。 | | **第二代 TPU (2017)** | 引入训练能力 (Training) | 验证了脉动阵列在训练场景的可行性；恰逢 Transformer 架构诞生，二者相互成就。 | | **生态扩张期** | 覆盖推荐系统与广告 | 深入 Google 核心商业引擎，全面承接广告点击率预估及大规模推荐算法。 | | **大模型时代 (TPU 8i / 8t)** | 架构分化：专用推理与专用训练 | 面对 2026 年算力格局，单芯片全能路线分拆为 **8i（推理专用）** 与 **8t（训练专用）** 两款独立硅片。 | 两年前规划新一代产品时，团队曾面临核心抉择：是继续设计一款兼顾训练与推理的统一芯片，还是将其彻底拆分？根据 Google 对 2026 年 AI 市场的测算，推理服务占总体算力预算的比例正在向 30% 到 60% 甚至更高快速攀升。如果推理需求仅占 2% 到 5%，制造专用推理芯片即便能带来两倍加速也没有商业合理性；但当推理工作负载成为数据中心的绝对主流时，打造一颗剥离训练冗余、在单位成本与能效上大幅超越通用的专属推理芯片（8i），便成为了极具经济确定性的必然选择。 ### [19:35](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1175s) 硬件专用化的权衡考量与 TPU 8i / 8t 的分化逻辑 在通用芯片与专用加速器之间进行权衡，核心是一个严格的计算与预测问题。芯片越针对特定负载进行专门化优化，其灵活性就越低，但能效和运行速度往往越高。然而，这种专用化要求工程团队必须精准预判：该负载到底有多大？在未来三到四年内，它能否保持持续的增长态势？如果一个工作负载仅仅在两三个月内规模庞大随后便迅速消亡，留给硬件工程团队拦截并适配它的窗口期就极为狭窄，根本无法支撑立项所需的庞大固定成本。 ```mermaid flowchart LR accTitle: Hardware Specialization Tradeoff accDescr: Comparison between specialization gains and workload risk subgraph Spec["高专用化 (Specialized)"] S1["能效极高"] S2["运算速度快"] S3["灵活性低 / 风险集中"] end subgraph Gen["可替代/通用 (Flexible)"] G1["跨负载复用能力"] G2["降低预测风险"] G3["留有一定性能损失"] end Spec -.->|"权衡考量"| Gen ``` 正是基于这种考量，谷歌将其 TPU 产品线拆分成了 8i（推理）与 8t（训练），但依然保持了二者之间的关键通用性。硬件的使用寿命长达六年，没有任何团队能够提前预判这六年内推理与训练需求的绝对配比。如果 8i 只能做推理而完全无法跑训练，或者 8t 在推理上性能归零，那么一旦出现某一端算力过剩，整个基础设施将陷入完全无法调度的僵局。因此，8i 与 8t 虽各自在目标负载上做到了更优的专门化，但依然具备跨界执行对方任务的能力，保留了必要的容错空间与灵活性。 - 启动全新芯片研发项目的固定成本极高，必须有足够持久且体量可观的特定负载需求作为支撑。 - 硬件生命周期通常长达六年，在这期间必须防范单一负载预测失误导致的资产闲置。 - TPU 8i 与 8t 兼顾了专用能效与“可替代性”（fungibility），若某一侧出现算力闲置，可无缝承接另一种负载。 --- ### [21:34](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1294s) 为什么不直接把 Transformer 架构“固化”进硅片？ 针对主持人提出的是否应该直接把 Transformer 架构“烧录”进芯片的问题，Amin Vahdat 剖析了其底层的技术本质。从数学实质来看，Transformer 归根结底是一系列线性代数原语的组合——主要是向量与矩阵乘法，以及 Softmax 等算子。谷歌当前的 TPU 以及业界的其他主流加速芯片，实际上已经将这些底层的线性代数原语固化在硅片硬件之中。 如果进一步走向专门化，就不仅仅是硬件针对 Transformer 本身优化的问题，而是必须针对特定模型的超参数与结构进行硬编码。例如：模型到底有多少层？层与层之间如何连接与穿越？每个维度的矩阵与向量具体尺寸是多少？这构成了下一阶段的极限专用化方向，虽然业界已有少数初创公司在探索针对特定模型定制硬件，但这同样伴随着极其高昂的架构僵化风险。 在谈到云端客户对 TPU 与 GPU 的认知时，Amin 明确指出二者并非完全等价的替代品。GPU 显然比 TPU 更加通用，谷歌自身以及 Google Cloud 既采购、自用也向外销售海量 GPU。然而对于特定问题集合，二者各有优劣，谷歌的策略是给予客户充分的技术选择权，根据具体负载的特征去匹配最合适的计算方案。 --- ### [23:30](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1410s) 跨层协同设计的利与弊：灵活性与性能阻抗匹配 关于从芯片、网络到软件全栈协同设计（Co-design）的争论，核心在于“可替代性与无厂商锁定”与“端到端最大能效”之间的取舍。如果不做协同设计，系统可以通过高度抽象的分层设计运行在任何公有云、任何硬件拓扑或网络环境中。这种方案的优势在于完全不受限于底层架构，一旦市面上出现新的算力资源，一夜之间就能部署上线。 ```mermaid flowchart TD accTitle: System Stack Co-design Impedance accDescr: Layers of the compute stack and optimization gains through co-design subgraph Stack["系统抽象分层"] L1["上层软件 / 算法模型"] L2["网络拓扑与通信协议"] L3["芯片架构与微体系结构"] L4["供电交付与物理基础设施"] end L1 <-->|"消除层间阻抗不匹配"| L2 L2 <-->|"消除 10%~2x 性能损耗"| L3 L3 <-->|"端到端乘数效应"| L4 ``` 然而，全面追求通用抽象的代价极其昂贵。在没有协同设计的情况下，软件与网络、网络与芯片、芯片与供电之间存在着严重的“阻抗不匹配”（impedance mismatch）。每一层抽象的泛化，都可能在实际运行中损失 10%、20% 甚至高达 2 倍的性能。当这些层面的损耗在全栈中逐级相乘时，被白白浪费的端到端系统收益将变得非常庞大。深入进行协同设计，能够把每一层挤压出的效率相乘，最终在关键指标上——无论是“每瓦交付智能”（intelligence per watt）还是“每瓦交付有效工作”（goodput per watt）——实现巨大的跃升。 - 追求完全可移植性：实现无厂商锁定、全环境部署，但会在每一层软件与硬件的抽象边界上损失大量性能。 - 全栈协同设计：通过消除层间阻抗不匹配，将供电、硬件、网络拓扑到软件优化的乘数效应转化为极高的端到端综合能效。 --- ### [26:11](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1571s) 肩并肩作战：谷歌基础设施与 DeepMind 的“半空拦截” 谈及基础设施团队与 DeepMind 算法团队的日常协作，Amin 称这是在谷歌工作“最有趣、最令人欣慰的环节之一”。双方并非甲乙方的交付关系，而是真正处于同一间会议室内的紧密研发伙伴。当 DeepMind 团队在 Transformer 架构或底层数学算子上提出全新优化思路时，他们会直接与硬件团队研判：如果硬件加入特定支持，整套端到端训练或推理的能效会提升多少，而当前正在推进中的芯片架构需要做出何种调整。 两边的工程师和研究人员经常连续数天甚至数周关在一间会议室里反复推演。现实中的妥协往往非常务实：硬件团队可能会说“我们无法完全照搬你的提议，但可以换一种硬件实现方案”，DeepMind 团队则会评估“这样调整模型架构，是否依然能拿到原本预期中 90% 到 98% 的收益”。如果收益极其显著，团队甚至会果断拍板：“我们愿意把流片（tape-out）推迟一到两周，因为这种级别的架构收益完全值得等待。” ```mermaid flowchart LR accTitle: Google TPU Multi-Generation Pipeline accDescr: Five active stages of TPU chip hardware pipeline S1["概念构想<br/>Concept"] --> S2["逻辑设计<br/>Design"] S2 --> S3["工程实现 / 待流片<br/>Implementation"] S3 --> S4["回片调试 / 准备投产<br/>Bring-up / Debug"] S4 --> S5["产线在役运行<br/>In Production"] ``` 在任何给定时间节点上，谷歌的芯片团队都同时推进着跨越多个世代的五到六级流水线： 1. **在役运行阶段**（In production）：团队与 DeepMind 深度配合，深挖模型细节与微体系结构，榨干每一瓦功耗下的有效算力（goodput）。 2. **准备投产阶段**（Getting into production）：芯片已从代工厂制造返回，工程师团队正进行系统级调试和 bring-up。 3. **工程实现阶段**（Implementation）：处于流片前夕，设计即将送往晶圆代工厂。 4. **逻辑设计阶段**（Design phase）：正在确定芯片内部微结构细节。 5. **概念构想阶段**（Concept phase）：针对数年后的前瞻技术路线进行探索。 这种紧密的组织架构带来了决定性的竞争壁垒：基础设施团队甚至能够在临近流片前的工程实现阶段，在“半空中”（in flight）直接拦截并改动芯片架构。Amin 指出，如果双方分属两家不同的独立商业实体，受制于商业壁垒、沟通周期和合同约束，这种在硬件流片前夕动态响应算法变革的操作“即便不是完全不可能，也绝对会艰难得多”。 ### [29:20](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1760s) 软硬件协同设计：与 DeepMind 并肩作战与 Gemini 反哺芯片 对于即将流片（tape-out）的芯片，如果突然发现问题，硬件与模型团队要临时聚在同一个房间里决定是否推倒重来，这种情况虽然可能发生，但代价极高且难度极大。而对于尚处于设计阶段的芯片，团队拥有极大的空间来共同评估各种候选架构。硬件团队会直接询问 DeepMind 的研究员：“在未来两到三年内，你们认为模型架构会走向何方？”硬件团队随后会给出可实现的帕累托最优边界（Pareto frontier），并与模型架构的演化路径相对齐。Google 内部构建了极为深厚且庞大的硬件仿真基础设施，能够高度精准地预测未来的 AI 工作负载在不同硬件架构上的运行效率。这种迭代不是部门间各自为政的交接，而是深度的日常协作——团队成员就在同一栋大楼甚至同一个房间里办公，Amin Vahdat 本人每周都会与 Demis Hassabis 以及 Jeff Dean（或 Koray Kavukcuoglu）进行多次深入沟通。Sonya Huang 补充道，未来这些前沿模型还将反过来协助芯片设计；Amin 则证实，Google 目前已经在利用 Gemini 来设计未来服务于下一代 Gemini 的底层硬件。 关于硬件与软件之间周期节奏不匹配（阻抗失配，impedance mismatch）的问题，Amin 坦承，硬件的规划周期必然比软件更超前，通常需要提前 2 到 5 年。例如，在刚刚发布 TPU v8i 和 TPU v8t 的当下，TPU v9、v10 乃至更长远的代际已经在概念与工程执行阶段铺开。而模型研究员在探索架构时，通常不会提前数年去思考。然而，Google 长期以来的团队融合（包括 Google Research、DeepMind 以及 Transformer 架构的诞生皆源于这种紧密交织的研讨环境）培养了一整代非常了解底层硬件的研究人员。他们既深知硬件遵循长周期规律，不会为了仅带来 0.5% 或 1% 收益的小改动去冒叫停流片的风险；但同时也明白，一旦发现重大的架构级机遇，硬件团队绝对会全力协同，设法将其做进芯片。这种硬件红利具有乘数效应（multiplicative）——正所谓“水涨船高”（lifts all tides），两年前在白板上构想的一个设计一旦落入芯片并投入生产，便能让整个 Google 的所有工作负载全面提速，这为整个团队带来了巨大的成就感。 - **多年前瞻规划**：当 TPU v8 系列正式对外公布时，下一代的 TPU v9、v10 等更远期的硬件架构已经在深度研发与概念验证之中。 - **自研闭环反哺**：Google 正运用现有的 Gemini 大模型来设计与优化面向未来代际 Gemini 的 AI 芯片。 - **仿真基础设施**：高度精密的硬件级性能模拟器，使得软件团队能在流片前数年验证不同模型结构映射到特定硬件上的能效比。 --- ### [32:42](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1962s) TPU 架构的基石：十年未变的计算原语 面对 Sonya 提出的疑问——预测 5 年后的主流工作负载与算法突破似乎是一项不可能完成的任务，Amin 透露了一个令人惊叹的事实：从中等颗粒度（medium level of detail）的架构视角来看，TPU 的核心架构从 2013 年发布的 TPU v1 至今，根本基石并没有发生颠覆性改变。这就好比经典 CPU 的指令集架构（ISA）一样，几十年过去，底层依然由 Load（加载）、Store（存储）、Add（加法）、Subtract（减法）、Branch（分支）等基础指令构成，而上层软件却诞生了翻天覆地的演进。TPU 同样遵循这一逻辑，它依靠少数几项极其扎实的核心计算原语，支撑了深度学习模型从 CNN、RNN 一路演进到 Transformer 乃至混合专家（MoE）的多代飞跃。 ```mermaid flowchart TD accTitle: TPU 核心原语架构图 accDescr: 展示构成 TPU 历代架构不变的五至六项核心原语及互连方式 subgraph Primitives["TPU 核心计算与通信原语"] MXU["超大规模矩阵乘法单元<br/>(Matrix Multiply Unit, MXU)"] Sparse["稀疏计算核心 (Sparse Core)<br/>向量运算与 Scatter/Gather 操作"] Numerics["专用数值格式单元<br/>(专用浮点/量化运算加速)"] RemoteLS["远程内存存取 (Remote Load/Store)<br/>跨节点直连读写机制"] end subgraph Network["片间互连层"] ICI["ICI 网络 (Inter-Chip Interconnect)<br/>极低延迟芯片间光电混合通信"] end RemoteLS <--> ICI ``` - **矩阵乘法核心（MXU）**：历代芯片均保留极大规模的矩阵乘法单元，作为密集张量运算的核心加速引擎。 - **稀疏处理核心（Sparse Core）**：专门负责管理向量运算、Scatter/Gather（分散/收集）等非结构化或半结构化内存访问。 - **专用数值表示（Specialized Numerics）**：在硬件层面对低精度数值计算提供原生支持与定制化优化。 - **远程读写（Remote Load/Store）**：通过芯片间直连互连网络（ICI, Inter-Chip Interconnect）直接读写远端芯片的高带宽内存，突破单芯片内存墙。 --- ### [34:16](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2056s) 长思考 Agent（Long-Horizon Agents）对数据中心架构的冲击 2024 年初以来，AI 工作负载形态发生了剧烈变化——长思考智能体（Long-Horizon Agents）的崛起打破了过去单次简短人机对话的模式。以往人类在浏览器或手机上输入 Prompt 时，虽然模型会密集计算，但人类阅读、思考、输入追问的过程天然引入了“数秒乃至数十秒”的延迟瓶颈（rate limit）；然而长思考 Agent 是端到端自动运转的，去除了人机回路（no human in the loop），推理响应一返回，系统在毫秒级内就会完成解析、推理并生成下一次模型调用，使得系统调用频率直接从秒级压缩至毫秒级。 更深远的影响在于，这种推理、解析和复杂的决策编排主要由传统通用 CPU 承担。在每一次触发新的模型 Prompt 之前，CPU 必须梳理状态、收集外部上下文——这些数据可能存储在本机 DRAM、另一台服务器 CPU 的内存中，或是分布式 SSD 及机械硬盘（HDD）之中。这导致整个系统的算力需求被彻底重构：虽然加速计算（GPU/TPU）的用量仍在持续飙升，但通用数据中心 CPU、存储以及数据编排网络的负载更是直接“冲破天花板”（going through the roof）。 这种负载形态的剧变引发了极其棘手的数据中心物理拓扑权衡： | 方案路径 | 部署方式 | 核心优势 | 面临的工程代价与物理瓶颈 | |:--- |:--- |:--- |:--- | | **混合部署**<br/>(Co-located Racks) | 将大批 CPU 机架紧挨着 TPU/GPU 机架混合部署 | 缩短 CPU 与加速器之间的通信物理距离，降低链路延迟 | 破坏机房专有化设计；TPU 机架功率与网络密度远高于 CPU，统一混合布局会导致供电、制冷及布线无法达到极度特化，降低空间利用率 | | **物理隔离部署**<br/>(Segregated Buildings) | TPU/GPU 机架统一放在专属机房栋；CPU 与高容量存储分别部署在相邻建筑中 | 保留机架高度特化与极致密度的优势；机械硬盘也无需受高能耗制冷气流与振动干扰 | 跨建筑网络复杂度激增；跨栋互连成本与故障率上升，伴随排队延迟（queuing delay），端到端延迟可能上升至数百微秒（$\mu s$）级别 | 面对 Sonya “你的工作真艰难”的感叹，Amin 笑着回应：“这正是好玩的地方所在。” --- ### [37:51](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2271s) 光学交换前沿：波分复用与光路交换（OCS）的技术底层 在探讨数据中心网络变革时，Amin 指出 Google 在底层网络创新上一直处于行业前沿。大约在 15 至 16 年前，Google 就率先在数据中心机架间网络中引入了波分复用（Wave Division Multiplexing, WDM）技术，实现在单根光纤中并行传输多路不同波长的光信号，大幅提升了光纤带宽密度。 紧接着波分复用技术的落地，Google 进一步研发并规模化部署了光路交换（Optical Circuit Switching, OCS）技术。传统的电分组交换机（Electrical Packet Switch）在转发数据时，必须首先在电域（electrical domain）对数据包头（packet header）进行解包、分析路由，再进行电-光转换发出；而 OCS 彻底打破了这一模式，它直接在纯光域（optical domain）内完成光束物理通路的偏转与重定向，从根本上省去了传统交换机反复进行光电（O-E-O）转换的巨大能耗与延迟开销。 - **波分复用（WDM）机架互连**：Google 早在约 15 年前便将电信级的 WDM 技术引入数据中心内部，解决机架之间爆炸式增长的吞吐需求。 - **全光域转发机制**：OCS 完全避开了电分组交换的报头解析与电路切换瓶颈，让光信号在物理层面通过微反射镜或光路变换直接打向目标光纤。 ### [38:58](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2338s) 光路交换（OCS）：用 3D 微镜与毫秒级重构掌控光流 在传统的电域数据包交换中，每秒都有数十亿甚至数万亿个高频数据包穿梭而过。交换机必须拆开每一个数据包的包头（Header），查看其 IP 目标地址，然后在庞大的路由表中检索匹配的端口，再将电信号转发出去。Amin Vahdat 指出，光路交换（Optical Circuit Switching，OCS）的核心哲学完全不同：“我根本不去碰电域里的这些比特数据。”OCS 直接在光学层面工作，它做的事情纯粹是为特定的光纤输入端口寻找并连接正确的光纤输出端口。 实现 OCS 有多种物理机制，Google 最初采用的是基于微机电系统（MEMS）的开关架构——利用微型电机在三维空间中动态微调成百上千面微型反射镜的角度。在一台具有 128 或 256 个端口的光交换机内部，光束从输入光纤直射到这些微镜上，系统通过软件程序调整微镜的角度偏转，将光束精确反射进指定的目标光纤输出端口。Google 早期在数据中心引入这项技术主要基于两大考量： 1. **构建计算与存储集群的拓扑就近性**：例如让支持搜索业务的计算集群和存储集群在需要海量数据互访时，利用微镜旋转在全光层建立点对点的“通信捷径”（Shortcuts）。 2. **免人工干预的网络动态弹性伸缩**：网络管理人员无需跑到机房人工插拔光纤，纯粹通过自动化控制器重新编排网络脊干（Spine），就能按需扩容或收缩网络。 ```mermaid flowchart TD accTitle: 光路交换（OCS）故障容错架构 accDescr: 展示 OCS 如何利用 MEMS 3D 微镜在毫秒内将发生故障的 TPU 机架旁路并重定向至备用机架。 subgraph Fabric["OCS 核心光层"] OCS["MEMS 3D 微镜交换矩阵<br/>（毫秒级重定向光路）"] end subgraph ActiveCluster["活动 TPU 环形拓扑 (Torus)"] RackA["TPU 机架 A"] RackFail["TPU 机架 B<br/>（突发硬件故障！）"] RackC["TPU 机架 C"] end subgraph SparePool["容灾冗余池"] SpareRack["备用 TPU 机架"] end RackA <-->|"光纤直连"| OCS RackFail -.->|"切断光路连接"| OCS RackC <-->|"光纤直连"| OCS OCS ==>|"毫秒级重路由重定向"| SpareRack ``` 当这套系统引入现代 TPU 超级计算机后，其价值发生了飞跃。由于 TPU 之间通过紧密耦合的环形拓扑（Torus）直接互联，一旦其中一个机架损坏，传统的全集群训练任务就会被迫中断。而借助 OCS，Google 在集群中始终保留在线的备用机架；当某个机架出现故障时，中央控制器可以在毫秒级内旋转微镜重定向光路，直接把备用机架无缝接入环形拓扑中，整个过程无需触碰任何一根物理线缆。 当 Sonya Huang 追问为什么不彻底抛弃光纤、改用全自由空间光通信（Free-space Optics）时，Amin 解释道，光在机房开阔空气中传输会面临剧烈的衰减损耗（Attenuation Loss），带宽也会发生断崖式下跌；更关键的是，在足球场般巨大的现代数据中心建筑内，要在无光纤导向的三维空间里精准瞄准数以万计的光束几乎是不可能的。尽管团队在白板上热烈讨论过这种构想，但光纤负责“跑长途”、OCS 微镜芯片负责“节点变轨”依然是目前最成熟可靠的方案。 - **MEMS 核心机制**：通过微电机精确控制三维微镜旋转角度，将输入端口光束原样折射至目标端口。 - **典型端口规格**：单台 OCS 设备通常承载 128 至 256 个光纤端口。 - **拓扑重构速度**：检测到机架级故障后，光路重定向可在毫秒（Milliseconds）级完成，大幅提升整集群有效吞吐量（Goodput）。 --- ### [42:35](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2555s) 电力： Frontier AI 唯一的物理终极约束 当被问及构建超大规模 AI 基础设施的最大瓶颈时，Amin Vahdat 直言，虽然日常工程中瓶颈总在算力、网络、散热和存储之间不断动态漂移，但如果退后一步看物理现实，“电力是我们面临的唯一最根本的约束条件（Fundamental Constraint）。”其他几乎所有系统层面的难题，工程团队都已具备清晰的技术演进路径，只是时间与执行的问题；但能源瓶颈是全行业共同面对的“刚性约束”（Binding Constraint）。虽然业界寄希望于核能带来充沛且零碳的清洁能源，但核反应堆的大规模落地与商用并网时间表依然充满变数。 针对 Sonya 提出的“Google 筹建吉瓦（Gigawatt, GW）级数据中心时，能否直接打电话给公用事业电力公司要求供电，抑或是被迫走向端到端垂直整合、自己建发电机”的疑问，Amin 阐述了 Google 深思熟虑的供电哲学：Google 的首选模式始终是与公共电网并网（Grid-connected / Utility-connected）。 与电力公司的合作不是打一通电话那么简单，而是一个跨度长达数年的联合协同规划过程： - **提前数年联合规划**：吉瓦级负荷不能凭空索取，双方需要从变电站选址、高压输电线路建设到发电机组调配全流程协同规划。 - **承担电网升级成本**：公共事业单位的基础设施建设费用若处理不当，可能被分摊到普通居民的电费账单上。Google 确立了一项严格原则，主动出资全额覆盖因自身负荷而产生的所有高压输电线路改造和专用变电站（Substations）升级成本，避免转嫁给居民。 - **弥补交付时间差的混合策略**：假设 Google 在 2028 年急需 1 吉瓦电力，而电力公司在 2028 年仅能交付 700 兆瓦、剩余 300 兆瓦要等到 2029 年，Google 会因地制宜配置自建的本地化资产（如光伏面板、储能电池等）作为缓冲过渡；但其底层逻辑依然是与公共电网保持深度的动态互动。 --- ### [46:32](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2792s) 大数定律与容量博弈：为什么 Google 不脱离电网自建闭环？ 针对为何不直接在数据中心旁边“圈地单干”、全套自建电厂的问题，Amin 从统计复用（Statistical Multiplexing）和大数定律（Law of Large Numbers）的角度算了一笔硬核经济与可靠性账。 如果一家科技巨头完全脱离电网、在“表后”（Behind-the-meter）自建一座 1 吉瓦的独立发电站，为了达到现代数据中心所要求的“四个九”甚至“五个九”（99.99% 至 99.999%）可用性，按照工业冗余标准，企业必须配置完整的 1+1 热备系统——这意味着你必须投入巨资建设整整 2 吉瓦的发电产能以防单点宕机。此外，在紧邻机房的方圆几公里内独力找到 100% 稳定的清洁能源也是极大的物理挑战。 相反，选择与公共电网并网，能带来巨大的系统级双赢红利： - **反向反哺电网**：在一年中最炎热、居民空调用电达到峰值的两周内，Google 可以调动自有的本地备用发电设备，将多余电能直接反向注入公共电网，缓解民生危机。 - **均摊可靠性成本**：接入宏观电网意味着可以在极大规模的底座上享受大数定律的平滑效应，无需让单一企业承担 1+1 冗余的畸高成本。唯有在极少数极端过渡期，Google 才会临时部署表后发电，但最终归宿依然是整体并网。 在探讨“一个数据中心园区究竟该建多大”这一终极架构抉择时，Amin 回忆道，早在 10 到 15 年前，Google 内部就爆发过异常激烈的技术路线争论：团队当时讨论是否应该把所有算力全部塞进唯一一个超大集中式数据中心里。当时大家设想的目标恰好也是“1 吉瓦”规模。 这种单体巨型数据中心的最大死穴在于单点故障（Single Point of Failure）风险，以 30 年为维度的长期系统容灾视角来看，把所有鸡蛋放在一个篮子里极其危险。然而在当前的 Frontier AI 时代，大模型分布式预训练（Training）的工作负载特性却展现出了截然相反的物理定律：跨节点的张量并行与流水线通信对极低延迟网络极度敏感，在网络拓扑允许的物理边界内，“单体集群规模越大，训练效率与通量表现就越好”。系统架构师必须在训练通信效率的极致扩张与全系统单点容灾风险之间，寻找精密的动态平衡。 ### [48:29](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2909s) 数据中心选址的物理与电力权衡：从单个机架到吉瓦级园区 从纯网络与延迟角度来看，算力越集中、距离越近越好；但这种做法立刻会撞上两大瓶颈：单点故障风险（SPOF）以及现实中的电力可获取性。Amin Vahdat 指出，在 10 到 15 年前，一个 1 GW（吉瓦）的数据中心听起来规模庞大但尚可想象；而如今，要在全美或全世界任何一个单一地点满足 Google 的总体算力电力需求，是“绝对不可能办到的”。因此，确定最优园区规模需要依赖专门的仿真器与容量模型，并根据业务定位做弹性划分： - **边缘与本地节点**：为了贴近驻在国网络或处于网络边缘，规模可能只有数十兆瓦（tens of megawatts）；在与某些 ISP 合作的边缘接入点，甚至可能小到“只有单个机架”。 - **区域大型站点**：根据电网条件与用地，单园区规模在数百兆瓦左右。 - **超大规模训练集群**：需要极致互联密度的集中训练基地，规模才会逼近 1 GW 级别。 ```mermaid flowchart TD accTitle: 数据中心物理规模层级 accDescr: 展示从吉瓦级训练集群到机架级边缘节点的不同部署形态与约束 subgraph Central["集中核心层"] Train["1 GW 训练集群<br/>超密互联 / 紧凑物理距离"] end subgraph Regional["区域与汇聚层"] Campus["数百 MW 园区<br/>电力供给与容灾平衡"] end subgraph Edge["网络边缘层"] EdgeDC["数十 MW 驻在国节点"] ISP["单机架 ISP 合作点"] end Train --> Campus Campus --> EdgeDC Campus --> ISP ``` --- ### [49:23](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2963s) 资产全生命周期管理：训练集群退役能直接用于推理吗？ Sonya Huang 提出了一种常见的直觉推测：最新的庞大集群用来训练前沿模型，随着硬件老化，顺势降级拿去跑推理（Inference）。Amin 回应称这种想法“合情合理，但在现实中无法完全成立”。 首先是**推理需求极其庞大**，光靠退役的旧训练集群根本无法填补算力缺口。其次是**地理覆盖与网络距离的冲突**：训练集群为了保证集合通信性能，在某一年份往往高度集中在少数几个大型站点（通常在同一大洲、甚至同一大洲的特定区域）；如果仅靠老旧训练集群来接管推理，其他大洲将面临灾难性的推理服务容量缺口。因此，Google 必须在世界各地主动兴建专用的推理集群。 在架构设计上，推理集群与训练集群存在本质区别，甚至“单位兆瓦成本并不见得更低”： - **训练集群**追求极致同质化、高密度与垂直整合，几乎全部由统一的算力与互连网络构成。 - **推理与服务集群（Serving Clusters）**则必须在空间上紧密混合部署通用计算（CPU）、网络和存储资源，难以做到极致单一的硬件特化； - **全球模型端点分散部署**：推理业务不希望把所有鸡蛋放在一个篮子里，而是需要面向全球用户提供低延迟访问。全球各地运行着大量不同的模型端点及变体，使得推理集群的物理规模更小、更离散，整体系统在纵向整合度上相对训练集群更低。 --- ### [51:55](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3115s) 硬件折旧与 Pod 级整机替换：7 至 8 年老旧 TPU 仍维持 100% 利用率 谈及芯片的实际使用寿命时，Amin 提到自己此前公开表达过的一个观点曾引发广泛讨论：“我们 7 到 8 年前的老旧 TPU 至今仍处于 100% 的满载利用率。”尽管业界技术迭代极快，但由于算力需求极度旺盛，较早期的 TPU 和 GPU 依然在承担海量任务。 关于硬件的退役与翻新，Google 遵循严密的工程与财务逻辑： - **折旧周期**：硬件的财务折旧生命周期约为 6 年。一旦完全折旧，考虑到新一代芯片在能效比（Perf/Watt）上的巨大跃升，继续占着机房电力和空间就不再划算，必须进行换代升级。 - **Pod 级别整机置换**：机房升级从来不是一颗一颗去拔插芯片，而是以 Pod（算力机群）为单位整体淘汰与进场。例如一个 TPU v8 Pod 可能包含 9600 颗芯片、约 152 个机柜。 - **机房逆向适配改造（Retrofit）**：拆除旧 Pod 并换入全新的 TPU v12、v13 或 v14 Pod 绝非直接“拼乐高”。新 Pod 的占地面积、供电与水冷规格与被腾出的旧空间往往无法严丝合缝匹配。由于无法提前五六年预知未来芯片的物理形态，如何快速退役旧机群、因地制宜改造基础设施并以最快速度上线新系统，是一门“极具挑战性的硬核工程艺术”。 --- ### [53:53](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3233s) 拒绝围墙花园：从开放标准到网络沙漏模型（IP 协议的历史启示） 在生态战略上，Amin 强调 Google 虽然支持深度的纵向整合以压榨极致性能，但坚决反对构筑专有锁定的“围墙花园（Walled Garden）”： > “在 Google 内部，我们开发了模型开发框架 JAX，我们非常喜欢它，在内部大量使用。但很多客户喜欢 PyTorch。我们本可以说：‘想在 TPU 上跑，你就必须用 JAX，它是最好的。’……或者我们也可以选择说：‘如果你喜欢 JAX，那很好；如果你喜欢 PyTorch，我们提供 Torch-TPU，你的模型无需修改就能直接跑。’” Amin 以 20 世纪 70 年代至 80 年代初的互联网协议（IP 协议）竞争为例阐释开放的力量：当时存在众多竞争性网络协议，但 IP 最终胜出，正是因为它作为开放标准构成了**沙漏架构（Hourglass Model）的“狭窄腰部（Narrow Waist）”**——向下可以承载任何异构底层硬件，向上可以支撑任意上层软件应用。只要设备支持 IP 协议，插入路由器端口就能立刻融入全球互联网。 AI 基础设施的建设规模过于宏大、意义过于重大，容不下任何单一供应商的封闭专有技术栈绑架。Google 的底层接入点全面拥抱开放标准与开源生态，并在云端同等支持 TPU、GPU 以及第三方加速器，将选择权交还给用户。 --- ### [56:23](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3383s) AI 反哺基础设施工程：软件、芯片设计与吉瓦级机房规划 AI 的爆发正在深刻重构基础设施团队自身的日常工作范式，这种变革不仅限于软件层，更深入到了芯片与数据中心规划物理层： 1. **软件工程全流程渗透**：AI 已深度参与 Google 基础设施团队的代码编写、测试发布流程以及早期架构设计。 2. **硬件芯片研发周期大幅压缩**：Amin 透露，硬件工程师的 AI 使用量（以 Token 消耗量计）丝毫不亚于软件工程师。从芯片设计启动（Design Kickoff）到流片（Tape-out）的研发周期正在显著缩短，硬件点亮（Bringup）速度大幅提升。 3. **数据中心规划从电子表格转向 AI 赋能**：在以往，评估一个站点究竟该建 1 GW 的超大单体，还是建 100 MW、200 MW 的分布式园区，高度依赖人工在复杂的 Excel 电子表格中反复推演。如今这一规划决策流程已大量引入 AI，实现了跨学科信息的快速整合与流转。Amin 澄清道，目前这套系统并非完全交给“推理模型（Reasoning Models）”去替代人类的最终商业与工程判断，但它极大地加速了海量异构信息的归纳与方案模拟。 ### [58:29](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3509s) 轨道数据中心：将算力送入太空的物理学与工程现实 访谈进入尾声，主持人 Sonya Huang 提出了两个天马行空又颇具趣味的终极问题。第一个问题聚焦于“轨道数据中心”（Orbital Data Centers）。Sonya 指出，Google 正在非常严肃地测算并考量轨道算力的可行性；如果顶尖科技公司都开始认真推演在太空中建数据中心，这是否反向证明了地球上的电力和空间正面临极其严苛的刚性约束？ Amin Vahdat 确认了这一点，并直言这确实是 Google 内部一个毫无戏谑成分、正在全力投资的重大登月项目（moonshot）。他从最基础的物理学定律拆解了太空计算的核心优势： ```mermaid flowchart TD accTitle: 轨道数据中心的能源增益与工程挑战 accDescr: 对比地面与太空数据中心在能源获取上的巨大差异，以及太空环境带来的冷却与通信工程挑战。 subgraph PowerAdvantage["太空能源核心增益"] A["大气层无衰减"] -->|"太阳能利用率提高"| B["光照强度 1.4x"] C["太阳同步轨道 (SSO)"] -->|"光照覆盖率"| D["98% - 100% 全天覆盖 (地面仅 28%-35%)"] B & D --> E["告别庞大电池储能储电"] E --> F["巨量全天候零碳能源"] end subgraph EngineeringBottlenecks["严峻的太空工程挑战"] G["散热与冷却 (真空环境缺乏传导介质)"] H["硬件可维护性与故障修复困难"] I["极度依赖空间光通信 (Free-Space Optics)"] end PowerAdvantage -.->|"权衡与落地瓶颈"| EngineeringBottlenecks ``` - **光照强度提升**：脱离了地球大气层的吸收和散射衰减，太空中的太阳辐射利用率大约可提升 40%（即地面的 1.4 倍）。 - **极高的日照时长**：在太阳同步轨道（Sun-Synchronous Orbit, SSO）中，太阳能电池板能够获得高达 98% 至 100% 的连续日光照射，而地球表面由于昼夜交替和天气影响，平均有效日照覆盖率仅有 28% 到 35%。 - **剔除储能瓶颈**：1.4 倍的光照强度叠加上 3 到 4 倍的日照时间，使得轨道计算系统在很大程度上可以不再依赖极其沉重的化学电池储能设施，直接获取纯粹的、海量的零碳能源。 然而，太空中也伴随着巨大的工程挑战。很多人直觉上认为外太空极度冰冷、散热会更简单，但 Amin 指出“在太空中散热其实更困难”（因为缺乏空气对流介质，主要只能依赖辐射散热）。此外，面对每小时都会发生硬件故障的超大规模集群，太空环境下的硬件维修将极其棘手。应对策略必须依靠极高的冗余度设计，或者发射专用维修机器人伴随集群升空作业。同时，卫星节点之间不可能铺设物理光纤，必须依赖空间光通信（Free-Space Optics），由激光精准对准接收端并进行毫秒级的动态实时校准。Amin 总结道，尽管困难重重，但在物理学底层“并没有不可逾越的致命阻碍（no showstoppers）”。 ### [01:01:00](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3660s) 2036 年的超级计算机：多兆瓦机架与终极集成 Sonya 紧接着抛出了最后一个设想：站在十年后的维度看，2036 年最前沿的超级计算机将会是什么形态？ Amin 坦言，预测十年后的技术图景处于不确定性锥体（cone of uncertainty）最宽的边缘。虽然内部有少数研究员在推演 2036 年的机器雏形，但技术迭代的跨度难以精确预估。不过，从底层趋势来看，“集成度（integration）的提升将是极其惊人的”。以当前在数据中心内部铺设的庞大光纤网络和 GB200 等高密机架为例，未来的发展方向必然是更致密、更一体化： - **极致集成的物理机架**：机架内部铜缆或近距离光通信高度固化整合，外部裸露散布的光纤将急剧减少，未来整机柜外部可能仅需引出一束极细的高容量光纤束。 - **超高芯片密度**：未来的机柜将采用中央模块化制造，机架内容纳的加速芯片数量将大幅飙升，从目前的 72、144、288 颗，一路跃升至 576 颗、1152 颗甚至更高的整数倍 GPU 或 TPU。 - **单机柜多兆瓦（Multi-Megawatt Rack）能耗**：到 2036 年，单个计算机架的功耗可能直接达到数个兆瓦（Megawatts）级别。部署方式将变得极其纯粹且标准化——像推箱子一样把机架推进机房，只需插上三样东西：**水管（冷却水）、超大功率电源线缆、以及光纤束**，系统便能立刻满载运行。 当 Sonya 开玩笑问及 2036 年的前沿超算会不会变成太空里的一颗“巨大的外星发光球体（alien orb）”时，Amin 笑着表示并不能完全排除这种可能：或许届时标准化的多兆瓦计算模块会直接用火箭发射升空，再由轨道空间站上的机械臂抓取并精准插入相应的太空舱插槽中。 对谈最后，Sonya 总结道，AI 基础设施正处于人类商业历史上最大规模、最激进的资本支出（CapEx）浪潮之中，但这绝非单纯的烧钱竞逐，更是一场极具技术美感的工业革命。Amin 动情地回应：“我们不仅有幸亲身经历这个时代，更有机会亲手定义它（We get to live through this and we get to define it）。”双方在对计算技术演进的致敬与赞叹中结束了这场对话。 ## Notable quotes > [05:30](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=330s) At 100,000-accelerator scale, something fails multiple times an hour, which is why Google's Chief Technologist for AI Infrastructure Amin Vahdat thinks FLOPS is a vanity metric. The metric that matters is what he calls goodput: the useful work a workload actually delivers through real failures. — Amin Vahdat > [00:00](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=0s) 这不仅在 Google 历史上前所未见，从全人类建设史与技术转型速度的角度来看，也是极其不可思议的壮举。 — Amin Vahdat > [11:52](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=712s) 每六个月左右将线上推理服务（Serving）的有效可用容量翻一番。 — Amin Vahdat > [26:11](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1571s) 我们愿意把流片（tape-out）推迟一到两周，因为这种级别的架构收益完全值得等待。 — Amin Vahdat > [29:20](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1760s) 硬件规划周期必然比软件更超前，通常需要提前 2 到 5 年。例如，在刚刚发布 TPU v8i 和 TPU v8t 的当下，TPU v9、v10 乃至更长远的代际已经在概念与工程执行阶段铺开。 — Amin Vahdat > [32:42](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=1962s) 从中等颗粒度的架构视角来看，TPU 的核心架构从 2013 年发布的 TPU v1 至今，根本基石并没有发生颠覆性改变。 — Amin Vahdat > [38:58](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2338s) 我根本不去碰电域里的这些比特数据。 — Amin Vahdat > [42:35](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2555s) 电力是我们面临的唯一最根本的约束条件（Fundamental Constraint）。 — Amin Vahdat > [46:32](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=2792s) 在一年中最炎热、居民空调用电达到峰值的两周内，Google 可以调动自有的本地备用发电设备，将多余电能直接反向注入公共电网，缓解民生危机。 — Amin Vahdat > [51:55](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3115s) 我们 7 到 8 年前的老旧 TPU 至今仍处于 100% 的满载利用率。 — Amin Vahdat > [53:53](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3233s) 在 Google 内部，我们开发了模型开发框架 JAX，我们非常喜欢它，在内部大量使用。但很多客户喜欢 PyTorch。我们本可以说：‘想在 TPU 上跑，你就必须用 JAX，它是最好的。’……或者我们也可以选择说：‘如果你喜欢 JAX，那很好；如果你喜欢 PyTorch，我们提供 Torch-TPU，你的模型无需修改就能直接跑。’ — Amin Vahdat > [01:01:00](https://www.youtube.com/watch?v=bGph8GwB3Sk&t=3660s) We get to live through this and we get to define it — Amin Vahdat ## Mentioned Amin Vahdat, Sonya Huang, Sequoia Capital, Google, Google DeepMind, Google Cloud, Demis Hassabis, Jeff Dean, Koray Kavukcuoglu, TPU, TPU v1, TPU 8i, TPU 8t, TPU v9, TPU v10, Gemini, Transformer, NVIDIA, GB200, Ineffable Intelligence, Vera Rubin, JAX, PyTorch, Torch-TPU, Optical Circuit Switching (OCS), WDM, MEMS, Co-Packaged Optics (CPO) ## Video description At 100,000-accelerator scale, something fails multiple times an hour, which is why Google's Chief Technologist for AI Infrastructure Amin Vahdat thinks FLOPS is a vanity metric. The metric that matters is what he calls goodput: the useful work a workload actually delivers through real failures. Amin walks through the calculus that split the TPU line into 8i and 8t for the first time, why the TPU's core primitives haven't changed since v1, and how Google and DeepMind co-design in the same rooms, intercepting chip architectures mid-flight before tape-out. He explains why long-horizon agents are sending demand for CPUs and storage through the roof alongside accelerators, how optical circuit switches reroute light to a spare rack in milliseconds, and why Google would rather wait on a utility than build its own gigawatt. We also cover orbital data centers and the multi-megawatt rack of 2036. Hosted by Sonya Huang, Sequoia Capital 0:00 – Introduction 1:47 – What makes a data center an AI data center 5:30 – Goodput, not FLOPS: holding yourself accountable when something fails every hour 11:52 – Doubling token capacity every six months, and where the gains actually come from 15:32 – The TPU bet: from a contrarian call in 2013 to splitting 8i and 8t 23:30 – The case for and against co-design 26:11 – Shoulder to shoulder with DeepMind: intercepting chips mid-flight 34:16 – Long-horizon agents change the shape of the data center 37:50 – Optical circuit switching and the state of networking 42:35 – Power is the binding constraint: utilities, gigawatts, and sizing a data center 49:23 – Training vs. serving clusters, seven-year-old TPUs, and open standards 58:29 – Orbital data centers and the supercomputer of 2036 [Source](https://www.youtube.com/watch?v=bGph8GwB3Sk)

[5:30] Amin Vahdat: At 100,000-accelerator scale, something fails multiple times an hour, which is why Google's Chief Technologist for AI Infrastructure Amin Vahdat thinks FLOPS is a vanity metric. The metric that matters is what he calls goodput: the useful work a workload actually delivers through real failures. [0:00] Amin Vahdat: 这不仅在 Google 历史上前所未见，从全人类建设史与技术转型速度的角度来看，也是极其不可思议的壮举。 [11:52] Amin Vahdat: 每六个月左右将线上推理服务（Serving）的有效可用容量翻一番。 [26:11] Amin Vahdat: 我们愿意把流片（tape-out）推迟一到两周，因为这种级别的架构收益完全值得等待。 [29:20] Amin Vahdat: 硬件规划周期必然比软件更超前，通常需要提前 2 到 5 年。例如，在刚刚发布 TPU v8i 和 TPU v8t 的当下，TPU v9、v10 乃至更长远的代际已经在概念与工程执行阶段铺开。 [32:42] Amin Vahdat: 从中等颗粒度的架构视角来看，TPU 的核心架构从 2013 年发布的 TPU v1 至今，根本基石并没有发生颠覆性改变。 [38:58] Amin Vahdat: 我根本不去碰电域里的这些比特数据。 [42:35] Amin Vahdat: 电力是我们面临的唯一最根本的约束条件（Fundamental Constraint）。 [46:32] Amin Vahdat: 在一年中最炎热、居民空调用电达到峰值的两周内，Google 可以调动自有的本地备用发电设备，将多余电能直接反向注入公共电网，缓解民生危机。 [51:55] Amin Vahdat: 我们 7 到 8 年前的老旧 TPU 至今仍处于 100% 的满载利用率。 [53:53] Amin Vahdat: 在 Google 内部，我们开发了模型开发框架 JAX，我们非常喜欢它，在内部大量使用。但很多客户喜欢 PyTorch。我们本可以说：‘想在 TPU 上跑，你就必须用 JAX，它是最好的。’……或者我们也可以选择说：‘如果你喜欢 JAX，那很好；如果你喜欢 PyTorch，我们提供 Torch-TPU，你的模型无需修改就能直接跑。’ [61:00] Amin Vahdat: We get to live through this and we get to define it

[5:30] Amin Vahdat: something fails multiple times an hour [5:30] Amin Vahdat: goodput [11:52] Amin Vahdat: whole systems company [15:33] Sonya Huang: The Bitter Lesson [26:11] Amin Vahdat: shoulder-to-shoulder [53:53] Amin Vahdat: too big and too important [58:29] Amin Vahdat: no fundamental showstoppers [61:00] Amin Vahdat: We get to live through this and we get to define it.

[0:00] In hardware again as you know there is [0:02] this opportunity where the more you [0:04] specialize to a particular workload the [0:05] less flexible it is the faster the more [0:09] power efficient the hardware is going to [0:10] be. So it is this art and it's this [0:13] projection of what are you designing to [0:15] and how persistent is that workload. In [0:18] other words if it's going to go away [0:20] after a month or two months or 3 months [0:22] even if it's big for those three months [0:24] you got a really narrow window to [0:26] intercept it. So it has to be somewhat [0:28] durable. Uh and you have to be able to [0:31] project ex exactly what win you can get [0:34] for specializing to it. [0:52] >> Thrilled to welcome Amin Vad to the [0:54] show. Amin thank you for joining us for [0:56] today. It's really exciting to be here. [0:58] I'm really looking forward to it. [0:59] >> I am very excited for today's topic [1:01] because we are in the middle of the [1:03] biggest capex buildout in human history. [1:06] You are in the middle of it. Google [1:08] alone is expected to spend more than [1:09] $200 billion on capex this year. Most of [1:13] which is going into building data [1:15] centers. Uh you're at the center of it [1:16] all. You were named the head of Google's [1:19] uh AI Infra at the end of last year. Um, [1:22] so you are the man spearheading the [1:23] efforts uh of one of the most capital [1:26] intensive buildouts in in human history [1:28] and so I am really excited to get into [1:29] it with you today. [1:30] >> It's a huge year uh for sure of course [1:32] across the industry including at Google [1:35] uh really have I don't think we've ever [1:37] seen anything like this honestly [1:38] certainly not at Google but as you said [1:40] I think in the history of humanity in [1:42] terms of the buildout and the pace of [1:44] transformation really unbelievable. [1:46] Before we get into it, maybe just just [1:49] crash course for the audience. What is [1:51] an AI data center and how how is it [1:53] different from a non-AI data center? [1:55] >> It's a very good question uh in that AI [1:58] data center and a non-AI data center [1:59] have do have a lot of similarities [2:01] actually and they're they're not [2:02] radically different. I mean it consists [2:04] of concrete right an enclosure it [2:07] consists of uh electrical yards uh [2:09] mechanical yards cooling uh you have row [2:12] after row of uh power being distributed. [2:16] Um there's a huge amount of network [2:18] infrastructure there. In other words, we [2:19] connect large amounts of computes to one [2:21] another. Uh a fair amount of storage [2:24] infrastructure goes in there. I think [2:25] that the big difference that we're [2:27] seeing with AI infrastructure is [2:30] specialization. In the past, when we're [2:32] building a data center, it really is a [2:34] 20 25 30-year building investment. And [2:38] we're thinking about how it's going to [2:40] be evolving over that 20 25 30-year [2:43] period. You might have servers go into [2:44] it. You might have uh networking [2:47] storage. You might have some [2:49] accelerators, GPUs, TPUs, whatever they [2:51] might be going into it, but it's a 25 30 [2:54] year planning horizon. So we have and [2:56] the lifetime of the hardware might be [2:58] six years. So many generations that we [3:01] have to plan for. An AI data center [3:04] often times is going to be more [3:06] purpose-built. So in other words, we are [3:09] oftenimes [3:11] co-designing the building with the [3:13] hardware that might go into it. We might [3:16] be saying, you know what, actually we're [3:17] not going to put a lot of storage into [3:20] this building. Why? Because a storage [3:22] rack might have 10, 20, 30, 40 kilowatts [3:25] of power. You put that next to a TP rack [3:29] or a GP rack that easily is hitting [3:31] hundreds of kilowatts today. Might hit [3:35] me. People are talking about that in the [3:36] next few years. Designing a building [3:38] that might take [3:40] 30 storage racks in a row versus one or [3:45] two AI racks in that same row. Very very [3:48] different design. Just think of it in [3:50] terms of the size. Think of it in terms [3:52] of the power and how that would be [3:53] distributed across uh the building etc. [3:56] Networking would be the same. The amount [3:57] of networking that you would need for a [3:59] storage rack tiny. I mean, if especially [4:02] if it's hard drives compared to an AI [4:05] rack. So, if you're trying to make it [4:07] fully fungeible, you'll probably make it [4:09] too big and too overbuilt. Fungeible [4:11] over certainly a 30-year period. An AI [4:14] data center likely is going to be much [4:16] more purpose-built, co-designed with the [4:18] hardware, even to the point of cooling [4:22] and power distribution, uh, etc. So, a [4:25] lot more co-optimization. You all [4:28] delivered a big Ver Rubin cluster to one [4:31] of my portfolio companies, Ineffable [4:33] Intelligence, and I saw the photographs [4:34] of it as it went out. That is a thing of [4:37] beauty. [4:38] >> It is. It is. [4:39] >> It's it's I mean, that's one of those [4:40] things where, you know, similar to [4:42] looking at the biggest construction [4:43] projects in human history, you look at [4:45] that data center, it's like, wow, that [4:46] is a monu monument to what what mankind [4:48] can do. No, we got a nice picture of it [4:50] and it um and this is just you know a [4:53] couple of racks. You know the the [4:55] cabling the fiber distribution for it is [4:57] is really be I mean you got beauty is [4:59] not the beholder but for folks like me [5:01] and probably yourself and others in the [5:02] audience as well. Um it is a thing of [5:04] beauty. We put this picture up on um [5:06] social media and um people love the fact [5:09] that ineffable by their huge fan of what [5:11] what they're doing. Fantastic team that [5:13] got a lot of attention. Okay, great rack [5:15] for an effable. But actually, people [5:17] just loved seeing the the pictures of [5:19] the fiber and the sort of the fractal [5:21] nature of the fiber. One of the most [5:22] popular um posts that uh we've ever had [5:25] actually uh was around that. So, so [5:27] yeah, absolutely. We're really really [5:28] excited about that. [5:29] >> Yeah, I had goosebumps seeing the [5:30] picture. [5:31] >> How do you measure and hold yourselves [5:33] accountable in this in the middle of [5:35] this big buildout? Um we were talking [5:36] right before the show about how flops is [5:38] a vanity metric and you prefer an [5:40] alternate metric. Can you can you say [5:42] more? Yeah. So I think uh one thing to [5:45] um note is that you know whether it's [5:47] flops or pick your other favorite uh [5:49] chip centric uh uh metric these are in [5:53] theory. In other words under some [5:56] conditions for whatever chip you have [5:58] this is the maximum amount of flops that [6:00] you can deliver. What we really care [6:02] about in the end is what's the [6:03] performance delivered by workload. [6:05] That's that's what we're looking at. And [6:07] rarely it it happens but actually [6:09] relatively rarely is that performance [6:11] determined by a single chip. It's flops, [6:14] it's HBM, it's SRAMM uh capacity etc. [6:18] Those are all super important but it [6:20] might be how 2 4 8 16 a thousand 10,000 [6:25] more of these chips composed together [6:28] with not just the accelerators again [6:30] whether TPUs or GPUs but then the CPUs [6:33] that might feed them the data the [6:34] network that connects them all together. [6:36] So what is the workload that you're [6:40] running and what is the performance of [6:41] that workload? [6:43] One measure that is interesting is [6:45] what's your flop's utilization. So for [6:48] example for a particular workload if you [6:50] have in theory uh teraflop or a pedlop [6:54] capable for your workload what fraction [6:56] of that are you delivering? [6:58] >> Now that's a measure of goodput. [7:00] >> What is goodput? [7:01] >> So you can think of throughput which is [7:02] a well-known term. Yep, [7:04] >> that's the throughput that is possible. [7:06] But now let's consider some uh other [7:09] considerations. One is what is the [7:11] slowdown of the workload? Again, just [7:13] inherent to the workload, but other [7:15] aspects of it that really hits our [7:17] reliability. [7:19] So in terms of how we hold ourselves [7:20] accountable, if we have a chip failing [7:23] for a synchronous workload and many of [7:24] these workloads, whether training or [7:26] serving or agentic workloads, they're [7:28] synchronous. There's many many [7:30] components that are working together [7:32] simultaneously. Now if you have let's [7:35] say a thousand 10,000 100 thousand of [7:37] these components working together [7:38] simultaneously and they're really [7:39] needing to coordinate uh at microscond [7:42] or millisecond granularity. One of them [7:44] fails it might actually bring the whole [7:46] thing to a stop right because everyone [7:48] is counting on everyone else to do their [7:50] part of the job in order to come up with [7:52] the answer to a really tough question. [7:55] One of them stops. Okay now we have to [7:58] figure out what happened which one [7:59] stopped. What's a checkpoint that we [8:01] have of the computation at some previous [8:03] state in time? How do we restore that [8:05] checkpoint? How do we restart? Worst [8:07] case, we have to restart from the [8:09] beginning. That would be really bad. But [8:11] that that can happen in certain cases, [8:13] especially on more on the inference [8:15] side. The point here is that if you now [8:17] then have to go back and redo a bunch of [8:19] computation, if you then have to pause [8:21] and wait to figure out what happened for [8:23] the failure, all that work is work. It's [8:26] not actually helping you get the answer. [8:28] Right? the in terms of let's say that [8:30] you're working out a problem on paper. [8:32] Step one, step two, step three, step [8:34] four. If you have to go back to step one [8:38] because you you made a mistake, yes, [8:40] you're still doing work. That's [8:42] throughput. You're doing work, but what [8:44] is the goodput in delivering your [8:46] answer? It's basically the total amount [8:47] of time it took you to solve the [8:48] problem. That's that's what matters. [8:51] And if you have failures, if you have [8:53] failure recovery, if you have whatever [8:55] it is that's interrupting your work, [8:58] that's that's part of the issue. So now, [9:00] how do we hold ourselves accountable? [9:01] It's delivered goodput, not theoretical [9:04] benchmark throughput or goodness in [9:07] theory, but actually for a workload for [9:09] real failure conditions, what's [9:11] happening? And the unfortunate reality [9:13] is that at a 100,000 accelerator scale, [9:16] >> I was going to say at that scale, [9:17] something's filling all the time, right? [9:19] >> All the time. and and these each one of [9:20] these as uh you're aware each one of [9:23] these chips is a u wonder of nature. I [9:27] mean that they're at the very bleeding [9:29] edge of what's possible to manufacture, [9:32] right? And so again to to me it's [9:33] stunning. And now actually these chips [9:35] aren't just one chip, they're packages [9:37] made up of in many cases two, four, [9:40] eight, maybe more chiplets, right, that [9:43] are composing together. And then of [9:44] course HPM off to the side, network [9:46] connectivity, maybe things like [9:48] co-ackaged optics. No criticism. Lots of [9:51] things can fail. And now if you got [9:52] 100,000 of them, something is going to [9:54] fail. [9:56] You have to be prepared for that. Detect [9:57] that in near real time. Recover from [10:00] that in near real time. Doing all that [10:03] it's it's really like finding the the [10:04] telemetry problem is massive. It's like [10:06] finding a needle in the haststack [10:08] continuously across certainly seconds [10:11] and minutes. But for some of these jobs, [10:13] hours and days and even weeks like just [10:16] continuous and online. How we hold [10:18] ourselves accountable is what's the [10:20] goodut that we deliver for the workloads [10:22] that actually matter in the data center. [10:24] Is goodput a Google term or is it an [10:26] industry term? [10:27] >> It's a Google term but I think more and [10:29] more people across the industry are [10:30] starting to pick it up. [10:31] >> And then just to calibrate me 100,000 [10:34] accelerator scale are we talking it's [10:36] going to fail once a minute once a once [10:38] a day like how how [10:40] >> 100,000 accelerators. So let me see I [10:42] think that it is definitely going to be [10:44] at the scale multiple times a day and [10:47] perhaps depending on the exact [10:49] configuration multiple times an hour uh [10:51] something is going to fail. And what are [10:53] the most common failure reasons? [10:55] >> This is the problem. Actually, it's a [10:56] very good question. Uh if there were a [10:58] common failure reason, we'd be we'd have [11:01] figured it out and and fixed it. It is a [11:03] long tale of constant discovery. You [11:06] know, whenever we have a new product [11:08] that's being introduced, there's going [11:09] to be something that hits us. Um in many [11:12] cases, frankly, because again, it's at [11:14] the very bleeding edge, it might be [11:15] network related. It might be how we [11:17] connect these things together at super [11:19] high speed. that might be something to [11:21] do with the hardware. Those things we [11:23] work through. Uh but then a lot of the [11:25] issues could be software. So this is the [11:28] other aspect of how we hold ourselves [11:29] accountable. Again, the chip might be [11:31] capable of a certain level of flops. But [11:34] if you have a compiler bug, a runtime [11:37] bug, a model issue, something else, [11:40] operating system issue, it doesn't [11:42] matter. That that's going to impact the [11:44] antenn system performance. So you you [11:46] might have perfect hardware fully [11:48] reliable but then you might have [11:49] software issues uh that hurt you. [11:52] >> Is there a standard reference stack from [11:55] the accelerator companies from Nvidia or [11:57] from the TPU team of you know here is [11:59] the optimal system to build around our [12:01] accelerators and as long as you build to [12:03] that system you're you're good or how [12:06] much of your own kind of data center [12:07] design do you have to do above and [12:09] beyond what the what the semiconductor [12:11] companies give you? Uh right. So there's [12:14] a there's a reference stack and I mean I [12:15] would say that uh Nvidia is a incredible [12:18] uh whole systems uh company that I mean [12:21] they're obviously semiconductor company [12:22] but they're not just a semiconductor [12:24] company. They give you a really really [12:25] strong reference stack but I would say [12:27] that uh many we find that um the way I'd [12:31] put it is most of our uh customers [12:34] leverage that reference stack but many [12:36] also specialize. So in other words, they [12:38] might find that as as natural for their [12:40] particular use case, they have an [12:42] optimization opportunity or something [12:44] different that they need to do and [12:46] they're going to do that. Similarly for [12:47] the TPU side, we have a reference stack. [12:50] But then and most people will leverage [12:52] that. Many will specialize it as well [12:53] though. [12:54] >> Got it. And then I guess in terms of [12:56] overall capacity, you've told your teams [12:58] that Google has to roughly double the [13:00] serving capacity every six months or so. [13:03] Is that right? Well, so to be clear, uh [13:06] this is in terms of effective available [13:09] capacity from the way in the end that we [13:12] look at it is from a serving [13:15] perspective. It's token generation [13:17] capability. So capacity and this is [13:19] where I would say it's a combination of [13:21] software and hardware. The hardware [13:22] might have some quote unquote inherent [13:24] level of flops. [13:26] What I'm saying here is not necessarily [13:28] you have to double the number of flops [13:30] every six months. That's one one path. [13:33] You have to double the capability of [13:35] that hardware to generate tokens every [13:37] six months. And as much or more of that [13:40] is going to come from software as it is [13:42] from uh hardware. So in other words, it [13:45] could be a model optimization that [13:46] delivers that. It could be some runtime [13:48] optimization that uh uh delivers that. [13:51] It's really probably going to be dozens [13:53] or hundreds of individual optimizations [13:55] that are just landing again and again [13:57] and again to make it all possible. But [13:59] yes, the rate the rate of capacity [14:01] improvement is incredible. [14:03] >> I I guess we've had a few years of data [14:05] center buildout now and a few years of [14:07] model progress and software progress. [14:10] What has been the empirical breakdown of [14:12] how much um capacity me as measured by [14:15] intelligence per watt? How much capacity [14:17] increase has come from the silicon [14:19] itself uh versus the models uh versus [14:24] other software and and any other kind of [14:26] big components? Yeah, it's a really good [14:29] question. Um, I don't have the exact [14:30] breakdown, but I would say that in our [14:32] experience, most of the benefits come [14:35] from model side improvements in terms of [14:37] intelligence per watts. And by the way, [14:38] the intelligence per watt is a fantastic [14:39] metric. Uh, really focusing we we also [14:42] for us it's um goodput per watts and we [14:46] can come back to why the watts need to [14:47] be the denominator. Goodput could well [14:49] be a measure of the intelligence [14:50] delivered per watt. In other words, [14:51] goodput is a workload specific metric. [14:53] But I would say most uh of the gains [14:56] often come uh come from the model side [14:59] software system um quite a bit. Why? [15:02] Because they're able to actually make [15:04] sure that the hardware that you have is [15:05] being used effectively. And then the [15:08] hardware, it it is pretty stunning. In [15:10] other words, we're living in a world [15:11] right now where 2x or more [15:14] year-over-year performance improvements [15:16] is absolutely possible. And so the [15:18] hardware does does support it. And it it [15:20] is a if you want it's a free multiplier. [15:24] Not free but um everyone else above it [15:27] can count on it in terms of that being a [15:29] multiplier that lifts everyone else up [15:31] uh year-over-year. [15:33] >> I'd love to talk a bit about the TPU [15:34] program and and codeesign. Uh so Google [15:37] started building custom silicon more [15:38] than a decade ago. It was contrarian [15:41] call at the time. Um how has the TPU [15:43] program evolved? [15:45] >> Quite a lot. I mean when the program [15:47] started 2013 it was really a contrarian [15:50] call and I mean I think that at the time [15:52] it's hard to put yourself back into the [15:53] moment 2013 conventional wisdom you know [15:57] all the smartest wisest people would say [15:59] you don't build a customuilt accelerator [16:01] for a single workload why because [16:04] Moore's law is there doubling of [16:05] performance every whatever it is 18 or [16:07] 24 months is there uh you get to [16:09] leverage standard programming models all [16:12] your C++ code etc java python whatever [16:15] it might be like the bitter lesson of [16:17] chips. [16:17] >> Yes, exactly. It's like the bitter [16:19] lesson of chips is that um [16:21] specialization never wins. But in in [16:23] this particular case, it was we have [16:25] this one application uh or or a few a [16:28] small number of them that would benefit [16:30] tremendously and that would require an [16:32] unimaginable amount of general purpose [16:33] CPU to support. So really in 2013 it it [16:38] was a bet and there were quite a few [16:40] people even within the company I would [16:41] say who were um not sure that it would [16:44] work out. So it was a bet. It turned out [16:47] to be a massively successful bet. So the [16:50] first chip was uh all all about [16:52] inference. [16:54] Um the second chip was hey actually we [16:57] can take the same idea and build a [16:58] training chip. [17:00] From from there it got picked up for uh [17:03] more and more use cases. Around the time [17:05] the second chip came out transformers [17:07] were invented. I mean this this was a [17:09] major major moment and that actually [17:11] completely shifted the TPU program. [17:13] uh we of course learned that recommener [17:15] systems could run really really well on [17:18] the uh TPUs and so ads and related uh [17:21] use cases came along. So I would say [17:23] that change and evolution has been one [17:25] of expansion in scope and impact. In [17:30] other words, it started with a really [17:32] impactful use case of a couple of [17:35] applications for inference serving. This [17:37] was language translation uh primarily [17:39] and voice recognition to then training [17:42] to then transformers recommener systems [17:46] and continuing to generalize as I guess [17:49] the um genai moment took a hold to ever [17:53] larger ever more scalable systems as [17:55] well. [17:55] >> You mentioned moment the transformer [17:57] came through being a a major moment. I'm [18:00] curious to explore [18:02] the TPU was like a specific architecture [18:04] but it wasn't you know specific to the [18:06] transformer specific. Um how do you kind [18:08] of straddle the fine line of you know [18:11] how specific you want your chip to be [18:12] for the workload? [18:13] >> This is a great question and I think [18:14] that it really comes down to the [18:16] applicability. I mean I think that um uh [18:19] of of the chip to a particular workload. [18:21] So in other words in the end we've been [18:24] thinking about this for um every [18:26] generation. Would we further specialize? [18:28] The question that we were facing let's [18:29] say um two years ago or a little bit [18:31] more was do in 2026 should we have two [18:35] chips or one? We could have one chip [18:38] that could do uh inference and training [18:41] simultaneously and do both quite well or [18:44] we could have two chips. One that was [18:46] further specialized for inference and a [18:48] second one that was further specialized [18:50] for training. This analysis and this [18:51] work led to the release of two chips [18:53] this year. 8 I for inference 8 T for [18:56] training and in the end what we realized [18:59] is that it wasn't the case necessarily a [19:01] couple years ago but by 26 we saw [19:05] inference and serving really taking off [19:08] and so having a chip that would be [19:10] significantly faster for serving that we [19:13] thought might be 30 40 50 60% of the [19:16] market in its lifetime started making a [19:19] lot of sense for us relative to if if [19:22] inference if we projected inference to [19:24] be 2% or 5% of the market even if that [19:28] specialized chip is let's say 2x faster [19:32] it might not make sense right because [19:34] you'd actually go for the one general [19:35] purpose chip that's not fully optimized [19:38] but that that's okay because you now [19:40] have the uniformity etc. So really uh it [19:43] does come down to a calculus question of [19:46] okay would you further specialize? How [19:49] big is that workload? How big is that [19:51] workload projected to be in a couple of [19:53] three four years? Is this a um sustained [19:56] growth for a particular workload in [19:59] hardware? Again as you know there is [20:01] this opportunity where the more you [20:02] specialize to a particular workload the [20:04] less flexible it is the faster the more [20:07] power efficient the hardware is going to [20:09] be. So it is this art and it's this [20:12] projection of what are you designing to [20:14] and how persistent is that workload. In [20:17] other words, if it's going to go away [20:18] after a month or two months or three [20:20] months, even if it's big for those three [20:22] months, you got a really narrow window [20:24] to intercept it. So it has to be [20:27] somewhat durable. Uh and you have to be [20:30] able to project exactly what win you can [20:32] get for specializing to it. So the rough [20:35] trade-off is there's a fix a large fixed [20:37] cost to support a new program. [20:40] >> Yes. [20:40] >> And you basically have to think that [20:42] there's going to be enough demand for [20:43] that specific program to justify the [20:46] cost. [20:46] >> Exactly. And uh to some to some extent, [20:49] you know, for example, for our 8i and 8t [20:51] chips, both chips can do the other [20:53] workload. This is also key. If 8i could [20:56] only do inference and it couldn't do [20:58] training at all. In other words, it had [21:00] zero performance for training. And then [21:02] vice versa. If it was amazing at [21:04] training and had zero performance for [21:06] inference, that would have also been a [21:07] key limitation. Why? Because we would [21:09] have to predict ahead of time over a [21:11] six-year period lifetime of the hardware [21:13] exactly how much we need of each. In [21:16] this case, it was nice because both [21:18] chips are better at what they're [21:19] specialized for. But if you needed to, [21:21] if you had leftover capacity in one [21:23] place or the other, both can actually do [21:25] the other uh other's job. depending on [21:27] the specialization you do, you might get [21:29] so specialized that you actually limit [21:31] yourself from being flexible, funible. [21:34] >> And if the rough trade-off then is kind [21:36] of size of program on the other end, [21:38] like it seems to me that so much of the [21:41] modern AI market is transformer-based. [21:43] And so maybe just asked a provocative [21:45] question, why not just burn the [21:46] transformer architecture into the chip? [21:48] >> Yeah. So I think that uh it's [21:50] transformer based but then uh the next [21:53] level of question would be um [21:55] transformers in the end are about vector [21:58] and matrix multiply operations and you [22:00] know a soft max. So there's a range of [22:04] essentially linear algebra primitives. [22:06] We have those and others do as well [22:08] roughly baked into the hardware. It's [22:10] all transformer based but then it's your [22:12] model architecture is okay how many [22:14] layers do you have? How do you go [22:15] through the layers? How do you go across [22:17] the layers? how for each dimension um [22:20] what exact shape of matrices and vectors [22:23] are you applying you you could go [22:25] further and specialize to not just the [22:28] transformer but your model that's the [22:30] next level of specialization I think [22:33] there are a number of companies out [22:34] there that are thinking about that I [22:35] think it's a very very interesting [22:37] direction as well [22:38] >> okay so at this point in time do your [22:40] customers view TPU and GPU as roughly [22:43] fungeible equivalents or is is there a [22:46] certain set of problems that is better [22:47] suited for one or the other. [22:48] >> There's for sure a better set of [22:50] problems that are better suited for one [22:52] or the other. I mean uh GPUs are more [22:54] general purpose than TPUs uh for for [22:56] one. That that is clear. I there we uh [22:58] there are incredible products at uh [23:00] Google and Google Cloud. Uh we we sell a [23:03] lot of GPUs. We use GPUs internally. But [23:06] I think it really then comes down to the [23:08] specifics of your problem. I so I think [23:10] that our customers there there is [23:12] overlap between the two but our [23:14] customers then evaluate their workload [23:16] and evaluate their options. What we like [23:18] to do at uh Google is give our customers [23:21] choice. In other words, we want to have [23:23] the right solution for their needs and [23:25] of course provide the solution that that [23:27] best meets their needs for them. [23:30] >> What is the case for code design from [23:32] the chip to the network to the software [23:34] and then what is the case against [23:36] codeesign? Yeah. So there's huge [23:39] optimization opportunities with [23:42] codeesign. And so what you can imagine [23:44] is that if you have a end layer stack [23:47] and you want to be able to pick and [23:49] choose whatever component you want, let [23:52] let's say you're running across many [23:54] clouds or you're running across many [23:56] pieces of hardware, many pieces of [23:57] software. You could design abstraction [24:00] layers that would basically say I can [24:02] run on any hardware. I can run on any [24:04] software. I can run on any network [24:06] topology um any amount of network that [24:08] you give me [24:10] no problem and my system is going to be [24:12] fully adaptive [24:14] very likely. So now you have a great um [24:17] capability. You can move anywhere like [24:19] new capacity becomes available [24:21] overnight. You're up and running because [24:23] you've designed your system that way to [24:25] actually be able to take advantage of [24:26] anything. You have not hardcoded um or [24:28] specialized at all to anyone's [24:30] particular infrastructure. Downside is [24:32] you probably leave a lot of performance [24:35] on the table if you become fully [24:36] funible. If you become fully flexible to [24:38] anybody's hardware, software, network, [24:40] storage, compute, etc. stack. So the [24:45] case for code designing is um between [24:47] each one of those layers um there's a [24:50] big impedance mismatch if you try to get [24:52] full generality. And so there might be [24:55] 10% 20% 2x across each of these layers. [25:00] You start multiplying those optimization [25:02] opportunities through and all of a [25:04] sudden you're left with a big endto-end [25:06] opportunity in terms of whatever you [25:08] want to say intelligence per watt or [25:10] goodput per watt that you can leverage [25:13] even again all the way down to power [25:15] delivery uh and power availability [25:17] software optimizations etc etc. So pros [25:21] are um you can run anywhere anytime you [25:24] have no lock in. [25:27] Cons are you're leaving um significant [25:30] and in all likelihood significant amount [25:33] of performance on the table. [25:35] >> So my understanding is that if you take [25:36] an open AI and enthropic um OpenAI was [25:39] kind of primarily building on a uh [25:42] homogeneous comput stack and Enthropic [25:44] was building on a roughly more [25:46] heterogeneous comput stack. Do you think [25:48] that's part of the reason that code [25:49] design is part of the reason why they [25:51] converged on what is rumored to be very [25:52] different architectures for their [25:54] models? Um I can't I can't speculate. I [25:57] can't I don't want to speculate in terms [25:58] of what open and anthropic are doing. [26:00] It's is one one possibility. Uh but I [26:02] think that [26:04] without knowing the details of what [26:05] they're doing, [26:06] >> I would imagine that there could be many [26:08] reasons for why they wind up with [26:10] different architectures. [26:11] >> Well, what about for Google then? I'm [26:12] curious what the working relationship [26:14] looks like between your team and [26:17] DeepMind. Kind of who's in the room at [26:19] what stage of model development and how [26:21] are you making decisions together on [26:23] code design? It's one of the most fun uh [26:25] and frankly gratifying parts of uh being [26:26] at uh uh Google is the opportunity to [26:29] work really shoulder-to-shoulder uh with [26:31] the deep mind team in terms of uh [26:33] co-design of our hardware and models. Uh [26:36] there's a third element to it in in [26:38] terms of that we also get to extend that [26:40] with the consumer services and cloud. [26:42] I'll put that aside for a moment. I can [26:43] come back to that. But with respect to [26:45] deep mind, it it really is a deep [26:48] partnership. So let me I mean from the [26:50] past I can give you examples where uh [26:53] they have come up with model [26:55] optimizations [26:57] um let's say to transformers or to [27:00] particular math that they might want to [27:02] do and we might have a chip in progress [27:05] not quite done but then they say oh my [27:07] gosh if we had hardware support for this [27:11] our endto-end training or serving might [27:13] get significantly faster more efficient [27:17] what would it take to actually now [27:18] change our hardware definition that we [27:20] might be an execution on to accommodate. [27:23] And this then leads to our engineers [27:26] from and researchers to getting together [27:29] intensely in the same room uh for a few [27:33] days, a week, two weeks saying, "Okay, [27:36] yes, we can do this. More likely, we [27:40] couldn't quite do what you wanted, but [27:42] we can do this other thing." And then [27:44] maybe you can change your model [27:45] architecture in this other direction [27:47] that gives you 98% that gives us 90% of [27:51] what we were looking for. And yes, we [27:53] can then go back and intercept the [27:55] hardware. Andor we can say, you know [27:56] what, we're going to delay the tape out [27:58] by a week or two weeks because wow, for [28:02] this level of benefit, it's it's totally [28:05] worth it. You know, similarly, when [28:07] we're um projecting our road map out and [28:11] it's time for, you know, we we we have [28:13] many generations of chips that are [28:14] essentially in progress at any point in [28:16] time. We have the chips that are in [28:17] production. That's one. We have the [28:19] chips that we're working on getting into [28:20] production. They're already back from [28:22] the manufacturer and we're debugging [28:23] them, making them work. We have the [28:25] chips that are in implementation that [28:28] are about to tape out and go to the [28:30] manufacturer. We have the chips that are [28:32] in design phase and then we have the [28:35] chips that are in concept phase. So it's [28:36] really this five six stage many many [28:40] year pipeline from in production to in [28:43] your mind etc. The collaboration with [28:45] deep mind for in production is [28:47] significant because we get to work [28:50] together in maximizing delivered [28:52] intelligence or delivered goodput per [28:54] watt and we know exactly what's [28:55] happening in the model and we know [28:57] exactly what's happening in the hardware [28:58] and everything in between. We also get [29:00] to collaborate deeply on though the [29:03] chips that are actually just about to [29:04] tape out. Why? Because we can intercept [29:07] and we can make changes to the chip [29:09] literally the chip architecture in [29:11] flight which would be somewhere between [29:14] hard and impossible to do if we were [29:16] working across company boundaries. not [29:19] not impossible. It would be much harder, [29:20] right, for us to say, "Oh my gosh, we're [29:22] a few weeks or a few months away from [29:24] getting this chip done and now let's get [29:26] in the same room shoulder-to-shoulder [29:28] and figure out if we should disrupt the [29:30] program." It's possible, but but harder [29:33] is what I would say. And then of course [29:34] for the chips that are in design, we [29:36] have many architectures we can evaluate [29:39] together. We can then ask our colleagues [29:41] in deep mind, where do you see model [29:43] architectures going in 2 three years [29:46] time? Here's the paro of things that we [29:50] can do. Here's the paro of where model [29:52] architecture is going. We have uh [29:55] actually deep and significant simulation [29:57] infrastructures that can predict how the [29:58] workloads are going to map to uh [30:00] different hardware architectures. [30:02] Deep deep and fast iteration. It really [30:04] isn't the teams are working separately. [30:07] It's in in uh same building, same rooms [30:12] many times. Deep daily interaction. I [30:15] talk to whether that's Cororey or Demis [30:18] multiple times a week uh etc. So it it [30:21] really is a a super fun aspect of the [30:23] work. [30:23] >> That's awesome. And eventually their [30:25] models will help with with chip design. [30:27] >> We're using Gemini to design uh hardware [30:29] for future Geminis as well. [30:30] >> That's really cool. I want to come back [30:32] to that a little bit later on the [30:34] collaboration itself. I guess is there [30:36] an impedance mismatch still of just I [30:38] figure the cycle times are probably just [30:40] slower when you when you're dealing with [30:43] hardware than than than your deep mind [30:45] folks get to deal with on the software [30:47] side and I figure your planning cycles [30:48] are much further ahead [30:50] >> for sure. [30:50] >> Um and so how much room do you actually [30:52] have to adjust? [30:54] >> Yeah, it's it's a really good question. [30:55] I we are planning hardware two three [30:57] four five years in advance. No, no [30:59] question. In other words, we talked [31:00] about TP8 high and at uh just now that [31:03] we announced, but you can imagine that [31:06] uh 9 10 and maybe some others are are in [31:08] in uh concept execution to to something [31:11] else. And so they and they might be um [31:14] years out if you're working with model [31:16] architecture by default you're not going [31:19] to be thinking years out. [31:20] >> Yeah. But I think this is also the great [31:22] thing about how the company has grown up [31:23] together and you know there was words [31:24] Google research and deep mind and [31:27] transformers being invented all of this [31:29] was happening in in these overlapping [31:31] rooms as well. So in other words, [31:33] there's a whole generation of [31:35] researchers who've grown accustomed to [31:38] being able to influence the hardware and [31:41] knowing that the hardware is operating [31:42] on multi-year cycles and they also know [31:45] that look, if they have a small tweak [31:47] that's going to deliver like 1% or.5% or [31:50] something like that for a chip that's [31:51] about to tape out, they're probably not [31:53] going to come to us because they they [31:54] they know enough to know that actually [31:55] it's not like software where you can [31:57] just do a change list and it's going to [31:59] ship to production in two weeks time. [32:00] like it actually stopping a tape out is [32:03] a big deal, but they also know if [32:06] they've got a big like a really good [32:08] opportunity. [32:11] Yes, we're absolutely going to work [32:12] together to figure out if we can get it [32:13] in there. So, um, as I said, it's [32:16] multiplicative in terms of where the [32:18] benefits come from. The hardware does [32:20] lift all all the tides, right? And so, [32:23] significant portions of the Deep Mind [32:24] team and we're so grateful for it. It's [32:26] an amazing team, but significant [32:28] portions of it are thinking about how do [32:30] I influence the road map because it's [32:32] actually a pipeline. Like the idea I had [32:34] two years ago, like it's in production [32:36] now and it's helping all the workloads [32:39] at Google go faster. Like that's a good [32:41] feeling. [32:42] >> Totally. It seems like an impossible [32:43] task though to predict in five years [32:45] what workloads will be most common and [32:48] what algorithmic breakthroughs will have [32:50] occurred. Uh doesn't seem like an [32:53] impossible task. I uh hear you that it [32:56] would seem like an impossible task, but [32:58] here's the awesome thing and actually [32:59] we're uh working on uh writing this in [33:01] great great detail and it's a lot of fun [33:03] to do it. The stunning thing is that um [33:06] the TPU architecture at a medium level [33:09] of detail, not at a super high level of [33:11] detail, at a medium level of detail [33:13] hasn't really changed since TPU v1. Like [33:15] one way to look at it is the instruction [33:17] set architecture for a CPU. like you've [33:20] got loads and you've got stores and [33:22] you've got ads and you've got subtracts [33:24] and branches and like and yet what has [33:27] the software on top of it done right [33:29] over over that period of time. Same [33:30] thing with TPUs. We have some [33:33] fundamental instructions and fundamental [33:35] primitives. Of course, we've extended [33:36] it. It's not like the instruction set [33:38] hasn't gotten changed at all, but the [33:40] primitives there in terms of [33:42] specializing the numeric [33:46] very large matrix multiply units um a uh [33:50] sparse core that manages uh vector [33:52] operations and scatter gather operations [33:55] etc. There's five or six things that [33:58] really define a load remote load store. [34:00] That's another one actually. We can read [34:01] and write uh remote memory through our [34:03] ICI network. the fundamentals have been [34:06] there and they've extended just to many [34:09] generations of models, many generations [34:11] of even deep neural network algorithms [34:14] and model structures. [34:16] >> Okay, speaking of shifting workloads, it [34:17] seems like one of the biggest changes in [34:19] workloads over the last year or so. I [34:21] think it really started at the beginning [34:22] of this year, this calendar year, was [34:24] kind of the rise of the long horizon [34:26] agent. [34:27] >> Yes. And I would guess that's a very [34:30] different shape of workload than kind of [34:32] the quick turn LLM conversations of of [34:34] years past. Um what does that mean in [34:36] terms of data center needs? [34:38] >> Yeah. So I think one there's two huge [34:41] aspects to this. One is it's no longer [34:43] human to um whatever you want to say [34:46] accelerate interaction. In other words, [34:48] when you are typing at uh prompt on a on [34:50] a web browser or on your phone or [34:52] whatever, uh, of course, in response to [34:54] your prompt, there's going to be a bunch [34:56] of work that happens, but then when the [34:58] response comes back, you've got to read [34:59] it. You've got to think about it, and [35:02] then maybe you have a follow-up that's [35:03] going to be multiple seconds of [35:05] interaction time. Now, in Deep Horizon [35:08] Agents, it's not there's no human in the [35:10] loop that is going to naturally rate [35:12] limits how quickly requests are going to [35:14] go to the model, right? So this what [35:16] went from you know seconds maybe tens of [35:18] seconds in terms of inter interaction [35:20] time is now going into perhaps [35:22] milliseconds right as soon as I get a [35:24] response back I can parse it I can [35:27] reason about it perhaps a bit and I can [35:29] figure out what my next request is going [35:30] to be. So that's big change one. Big [35:33] change two is all that reasoning and all [35:35] that parsing is going to probably happen [35:37] on a CPU. And that CPU is then going to [35:40] have to probably think about, okay, what [35:43] other state do I need to go gather [35:44] before making my next prompt back into [35:46] the model? Like in other words, I' I've [35:48] learned something from this response. [35:50] I'm going to take another step, but I [35:52] actually need to go grab some context. [35:54] maybe from uh DRAM local to me, maybe [35:57] from someone else's DRM on another CPU, [36:00] maybe on SSD or maybe on HDD somewhere [36:03] else. So now uh huge amount of [36:05] orchestration has to take place as well. [36:08] So the design actually is changing [36:10] pretty significantly where the demand [36:13] for accelerated compute is going up but [36:16] the demand for CPU and networking and [36:19] storage the you know traditional data [36:21] center CPU etc is also going through the [36:23] through the roof. Uh [36:24] >> does that mean you're putting more CPUs [36:26] alongside your G your GPU racks then? So [36:28] this is yeah GPU TP racks. This is the [36:31] key question is um going back to this [36:34] question of optimization and uh [36:37] specialization. If we start putting lots [36:39] of CPU racks and it is a question uh [36:41] next to the GPUs and TPUs that means [36:44] that actually we can't fully specialize [36:46] to the density and network requirements [36:48] let's say of a TPU rack relative to a [36:51] CPU rack. TP rack is going to be more [36:53] dense than a CPU rack. It's probably [36:55] going to need more networking than a CPU [36:56] rack. So in other words, now our [36:58] building design is going to change. [37:00] Another option is maintain your [37:03] uniformity. Put your TPUs or your GPUs [37:06] all in one building and the building [37:08] next door, but maybe put your CPUs and [37:12] your maybe your um and maybe by the way [37:14] the hard drives have to be in another [37:16] building on the other side because they [37:17] have yet another set of requirements. [37:21] Now you need networking between these [37:23] buildings at a pretty significant level. [37:25] So once you leave a building the [37:29] networking complexity goes up [37:31] significantly from a reliability [37:32] perspective from a cost perspective [37:34] latency goes up probably acceptably but [37:37] uh still now you might go to hundreds of [37:40] microsconds potentially more with [37:42] queuing between the components. So the [37:44] considerations do change in pretty [37:46] interesting ways. [37:47] >> Your job is hard. [37:48] >> That's fun. [37:49] >> It's fun. Yeah. [37:51] um what's happening on the networking [37:53] side and uh I've heard that Google's [37:56] always been the at the forefront of the [37:58] newest in networking including optical. [38:00] Uh can you say a word on what is like [38:02] this the state of of optical networking? [38:04] You know, we at um Google, this was [38:08] probably 15 16 years ago, were among the [38:11] first to bring um essentially uh what's [38:14] called wave division multipplexing um [38:16] where you could put u multiple signals [38:18] on a single fiber within the data center [38:21] and we actually leveraged that for all [38:22] of our uh communication between racks. [38:25] At the same time, we introduced a [38:27] technology uh along with the wave [38:30] division multiplexing called optical [38:32] circuit switching. And essentially what [38:34] optical circuit switching does is in [38:37] contrast with traditional electrical [38:39] packet switching, what it does is it [38:41] transmits and moves data entirely in the [38:44] optical domain. The great thing about [38:46] that is and so let me describe the [38:48] technology that underpins it in a in a [38:50] packet switch. You would take a packet [38:53] that has a header. You would look at it [38:56] um in the electrical domain. Figure out [38:58] where the packet is headed in its [39:00] header. It might have an IP address that [39:01] says, "Okay, where do I head it?" You [39:03] look up, okay, for that destination, you [39:05] look up in a table, what ports do I [39:07] forward it along? So basically, [39:09] billions, billions and billions, perhaps [39:12] trillions of packets coming through at [39:14] super high speed per second. You're [39:17] forwarding them along. Optical circuit [39:18] switching says I'm not touching these [39:21] bits in the electrical domain. What I'm [39:23] going to do is I'm going to figure out [39:24] for an input port which output port to [39:27] send the light to. Okay. And now I have [39:31] uh there's multiple ways to do this. The [39:33] one that we started with is called MEMS [39:36] uh switches, micro electrical motors [39:39] that basically control mirrors in 3D. So [39:42] we can now programmatically take a box [39:44] that might have you pick 128 ports, 256 [39:47] ports, some number like that. And we can [39:49] configure every input port for a fiber [39:51] that comes into it, map it to an output [39:54] port, and then change the rotation of [39:56] mirrors where the light literally shines [39:58] down on these mirrors and gets reflected [40:00] to the right output port. Initially, the [40:02] reason we did this twofold was to create [40:05] locality between groups of racks. Let's [40:07] say I had a compute cluster and a [40:12] storage cluster and both of them were in [40:15] support of again making it up search. We [40:18] knew that these two clusters would talk [40:20] to each other a lot. So we would [40:21] configure the mirrors to create [40:24] shortcuts purely optically between those [40:26] two clusters, clusters of racks. That [40:29] was reason number one. Reason number two [40:31] was we wanted to be able to expand the [40:32] network and contract the network without [40:35] actually moving any fiber. And without [40:37] working into the details, I could draw [40:39] this on a board. The optical circuit [40:41] switch would allow you to actually [40:42] reconfigure the spine of the network to [40:44] expand it or to shrink it without a [40:46] human being um having to do anything [40:48] other that' be literally a controller [40:50] that would manage that. Now fast forward [40:53] to TPUs. Uh a TPU [40:57] has a Taurus topology that connects all [41:00] the TPUs to one another directly. [41:03] I I talked about throughput and goodput [41:05] earlier. One of the things that we can [41:07] do is if we have a TPU rack that fails, [41:11] we can replace it with another TPU rack [41:13] without moving any fiber. Again, I'd [41:16] have to wave my hands or draw a [41:18] whiteboard. Uh but essentially, we can [41:21] say we have a spare rack available at [41:23] all times and when a rack fails, we [41:26] redirect the light to that new rack [41:28] >> and that can be done in milliseconds. [41:31] >> Why do you have the fiber at all then [41:32] >> as opposed to fully free space? Yeah, [41:35] it's a very good question. Um, [41:37] attenuation loss and the bandwidth would [41:39] drop uh dramatically. F furthermore, [41:41] across the range of a very large uh [41:44] building, aiming everything uh without f [41:47] without the benefit of fiber to connect [41:49] it all together uh in 3D would be would [41:52] be challenging to possibly impossible. [41:55] Um we've talked about it. We've talked [41:57] about it actually and there have been [41:59] some really interesting discussions. uh [42:01] but but yes it's mostly in fiber but [42:03] then when they hit the optical circuit [42:05] switch uh essentially the light then [42:07] literally shines down on these tiny [42:08] chips. So um that's one big direction of [42:12] uh networking. There's but there's [42:13] there's a lot I mean I would say [42:15] networking is exploding in in terms of [42:17] its u uh capabilities and need frankly [42:20] in the data center. [42:21] >> So cool. We could have an entire [42:23] conversation on that. [42:23] >> Yes, it is very very cool. [42:25] >> Even like the going back to the picture [42:26] from ineffable it was all the the cables [42:28] that that cut everybody's eyes. [42:30] >> Exactly. Yeah. And those cables [42:31] eventually some of them wind up at [42:32] optical circuits which in our data [42:34] centers. [42:35] >> Makes sense. Okay. I want to flip to [42:36] talking about power. You've been talking [42:38] about goodput per watt and other units [42:40] per watt. Makes me think per watt means [42:42] power is kind of the the binding [42:43] constraints or the scarce constraints or [42:45] the expensive constraints in some way. [42:47] >> You know, so I I uh I get asked this [42:49] question, what is the biggest constraint [42:50] that we face? And um the reality is [42:53] there is no single biggest constraint [42:54] that we face. They're all constraints. [42:56] They're all super hard. It could be and [42:58] then they shift continuously. But if I [43:00] had to answer fundamentally [43:03] uh I would say that power is the single [43:05] most fundamental constraint that we face [43:07] like everything else seems [43:10] uh like we know how to solve them and [43:12] it's a question of um solving them over [43:14] some period of time. Power yeah I think [43:16] the way that you put it is really nice. [43:18] It's a it's a binding long-term issue [43:21] that we do not have a I mean there you [43:23] know nuclear you know perhaps nuclear is [43:25] going to abundant clean energy which [43:28] would would solve a lot of problems uh [43:30] for for sure when that happens and at [43:33] when it happens at scale uh still still [43:36] unknown [43:37] >> and so how does it work in practice you [43:39] you know you're standing up a new data [43:41] center it needs a gigawatt of power I [43:45] imagine you can't just call PG& and say [43:47] hey please send a gig a lot of power. [43:48] So, h what does provisioning power [43:50] actually look like and are you having to [43:52] actually vertically integrate all the [43:53] way down to like, you know, doing your [43:55] own turbines or how do you solve the [43:57] power bottleneck? [43:58] >> Yeah, it's again it's a it's a big [44:01] question, important question. We do uh [44:03] our our preferred model at Google is [44:05] always to be uh utility connected to be [44:08] good connected. So, it's the equivalent [44:09] of calling your um uh [44:11] >> Oh, so you can't just call PG&E. You [44:13] exactly you call uh ve very politely [44:16] your favorite uh utility wherever it is [44:18] that your data center is and of course [44:19] you give them many many years of notice. [44:22] So in other words, if we're talking [44:23] about a gigawatt scale, it's not [44:25] something that you can say, "Hey, I need [44:26] a gigawatt tomorrow. When when can you [44:28] start billing me?" Uh it's something [44:29] that we co-l plan together. You know for [44:32] uh for us it's something that we also [44:33] take very seriously from the perspective [44:35] of ensuring that when we work with these [44:37] utilities the the costs of putting that [44:40] infrastructure in place the we could get [44:42] into a long conversations long [44:44] conversation just on this topic as well [44:46] that we cover those costs because with [44:48] the way that billing works it actually [44:50] could be that by the act of the utility [44:52] building out capacity for us let's say [44:54] other people's rates could go up in [44:56] theory what we ensure is that actually [44:58] let's say the transmission lines that [45:00] have to be built upgraded ated uh [45:02] additional utility based stations etc [45:04] that we pay for those as well but so it [45:06] is a long planning process uh it uh can [45:09] absolutely be the case that in let's say [45:11] that we need a gigawatt in I'll make up [45:14] a date uh 2028 [45:17] uh and the utility can get us a gigawatt [45:20] in 2029 that can get us let's say 700 [45:23] megawatts in 2028 so now we might be [45:26] left with a question of how do we cover [45:28] those 300 megawws well one answer is [45:30] just Another answer is to say okay well [45:33] would we figure out how we generate some [45:35] subset of that power ourselves and maybe [45:37] that would be with solar cells or with [45:40] uh with batteries as a backup etc. Would [45:43] it be for other sources? So then we [45:45] again work with the utility. So it might [45:47] also be a uh combination where we might [45:50] maintain some power uh generation local [45:53] and have the capability even when the [45:55] utility uh comes online uh fully at [45:58] let's say the gigawatt scale where we [46:00] could actually provide power back to the [46:01] grid. So having that when they need it [46:04] as well, right? So in other words, there [46:06] might be the one way to look at it is [46:07] there might be the two weeks of the year [46:09] maybe it's the hottest two weeks of the [46:11] year where there's huge amount of [46:13] residential demand. If we have some [46:15] local generation, we can then provide [46:17] that back to the grid as well. So it [46:19] really is working over multiple years [46:21] with the utilities uh for us. Why is [46:23] your preference to do that versus to [46:26] kind of go like to work with the [46:28] utilities as opposed to to kind of [46:30] vertically integrate yourselves? [46:32] >> Yeah. So, um the main main reason is [46:35] flexibility and uplift on both sides. [46:37] So, one one way to look at it is um [46:40] statistical multiplexing or if you want [46:42] the law of uh large numbers. If we need [46:46] to have a gigawatt of power, let's say [46:48] we want to have that with 99.99% [46:51] plus reliability, probably means we have [46:53] to build two gigawatts of power, right? [46:56] At that level at 99.99 or 99.999, you [47:00] have to have oneplus 1 redundancy. [47:02] That that gets expensive. And then [47:04] making sure that that's going to be [47:06] ideally a clean energy source probably [47:09] right next to our data center, that can [47:11] also be challenging. Now if we partner [47:13] with the data center again maybe we have [47:14] some some amount of that that we can [47:16] bring ourselves that we can give to the [47:18] grid when they needed less that we can [47:20] take their power. So in other words by [47:22] leveraging that statistical multiplexing [47:23] over a much larger base actually [47:25] everybody wins we win the the grid wins [47:29] residences win etc. We prefer that and [47:32] in rare rare cases we we will uh do it [47:36] uh let's say behind the meter but even [47:38] then we're doing it under the assumption [47:40] that we're going to work with the [47:41] utility where it might be a year later [47:44] we we want to be connected to the grid. [47:46] So again it's an uplift for us it's an [47:48] uplift for the grid. [47:49] >> Makes sense. How do you decide how big [47:51] to make a data center? [47:52] >> Yeah that's that's an art and it's a a [47:54] source of significant debate. You know, [47:56] at some point, you know, we had a [47:57] debate. I remember even 10 plus years [47:59] ago, 15 years ago, big debate at Google. [48:02] Should we just put everything into one [48:03] data center? Like back then, it was [48:05] going to be a gigawatt, which was [48:08] >> simpler times. [48:09] >> Yeah. Different times, like 105 years [48:10] ago, we're going to have a gigawatt data [48:12] center. So, one obvious concern there is [48:15] single point of failure, right? If [48:17] you're looking at things from a 30-year [48:18] perspective, that's a a huge concern. [48:22] You know on the other hand if you're [48:23] talking about uh training workloads [48:24] today bigger is better in other words [48:28] having more because from a networking [48:29] perspective actually you want to have [48:31] things concentrated in as small a [48:33] distance as possible but then again uh [48:37] two issues single point of failure but [48:38] then also now power availability right [48:40] while a gigawatt was huge 10 or 15 years [48:42] ago but imaginable um there's no way [48:45] anywhere in the country or the world [48:47] we're going to get the total [48:48] requirements of Google built in in one [48:50] place. it's just just not going to be [48:52] possible. So now what's the optimum [48:54] size? Uh again we have models for this [48:57] simulators uh etc. But this it it also [49:01] depends on look in some places we're [49:03] going to be at the edge of the network [49:05] or we're going to be in country that [49:07] might be tens of megawatts. We actually [49:10] partner with ISPs that might be Iraq [49:13] like literally it could be Iraq. This is [49:15] a training cluster. Okay maybe that's [49:18] going to be closer to a gigawatt. Um [49:20] some other sites might be hundreds of [49:21] megawatts uh etc. [49:23] >> Interesting. I'd love to understand how [49:24] you think about the portfolio life cycle [49:27] management. Um so I guess my guess would [49:31] be that you know the biggest newest [49:33] clusters are used for training the [49:35] latest frontier model and then you kind [49:37] of recycle the older stuff and run [49:39] inference on it. Is that is that like a [49:41] fair framework? Are you are you also [49:43] standing up inference specific clusters? [49:45] How does that all work? very reasonable [49:47] framework uh that that you're thinking [49:48] about that uh and I think makes a lot of [49:50] sense but I would say that the demand [49:51] for inference is such that we can't just [49:53] rely on um whatever older training [49:56] clusters that are no longer being fully [49:58] used just for training uh as the basis [50:00] for inference and also if you think [50:02] about it we might centralize again uh in [50:05] a particular year into a small number of [50:07] large sites our training clusters and [50:10] there is benefit to keeping the network [50:12] distance between them uh small so now [50:15] let's wherever they're located in the [50:17] world is they're probably going to be [50:19] let's say on the same continent or they [50:21] might be even in the same portion of a [50:22] continent in a particular year. So now [50:25] you might be left with um not enough [50:27] serving capacity on the other continents [50:30] etc. So then okay actually we then have [50:32] to go build the specialized inference uh [50:35] elsewhere across the world. So uh I I [50:37] think your intuition is uh spot on but [50:39] not wholly. In other words, it really [50:41] does have to be okay here are the [50:43] training clusters. Yes, probably they're [50:44] going to be used for serving in some [50:46] number of years. Uh but then we're also [50:48] having to build the serving clusters as [50:50] well. [50:50] >> And your serving clusters are they [50:52] different than the training clusters? [50:53] Like are they smaller? Is there are they [50:55] are they cheaper per per megawatt? uh [50:58] not necessarily cheaper actually because [51:00] uh for serving there is more need to [51:03] colllocate uh uh compute networking and [51:06] storage and so that then we get to the [51:08] inability to specialize. So for training [51:11] you actually have this big density [51:13] uniform deployment etc. Uh whereas for [51:17] serving you're going to have to have the [51:18] mix of storage compute and accelerators. [51:21] Furthermore, for um it's a very [51:24] interesting aspect of this that we could [51:26] go into more detail on as well, but for [51:28] serving, you actually don't want to have [51:29] too much in one place. You want to be [51:31] serving your workloads from uh all over [51:33] the the planet, but now uh we wind up [51:37] having um individual model endpoints and [51:41] that actually can be variations of [51:42] models. So now we have to spread these [51:44] models out across the world also [51:46] accounting for locality. So they will be [51:48] smaller. They'll actually be less [51:51] vertically integrated probably on the [51:54] infant side. [51:55] >> Super interesting. So if you made a data [51:58] center 5 years ago, you know, [51:59] state-of-the-art accelerator 5 years ago [52:02] was very different, [52:03] >> vastly less efficient than the ones that [52:05] are being made today. [52:05] >> Yes. [52:06] >> Do you actually go back and like swap [52:07] out the the chips in those in those old [52:10] data centers? And I know this kind of [52:12] relates to there's been an ongoing [52:13] debate, I think, of like what is the [52:14] actual useful life of a chip? [52:16] >> Yeah. Yeah. So, you know, I've been uh [52:18] uh on on record of saying this and one [52:20] one of my more uh I was surprised by how [52:22] how much pickup this had, but but you [52:24] know, I said that our uh seven and 8 [52:26] year old TPUs are still at 100% [52:28] utilization. [52:29] >> Wow. [52:30] >> Uh and and [52:31] >> I think I saw this. I think that was [52:33] >> Yeah, I didn't mean for it to be a a [52:35] meaningful statement, but it was a [52:36] meaningful statement apparent. Yeah. So, [52:38] our um older TPUs uh GPUs, but our older [52:42] TPUs are seeing significant utilization. [52:44] We do uh replace them in the end. [52:46] there's a question of okay have they uh [52:49] uh depreciation lifetime is [52:51] approximately 6 years uh so once they're [52:53] fully depreciated uh and given the power [52:56] efficiency of newer generations etc it [52:57] does make sense to um replace them and [53:00] upgrade them it's not that we I do [53:02] replace the chips but it's really the [53:03] the the systems so in other words we [53:05] think in terms of pods so a TPU let's [53:08] say um 8 pod might be 9600 chips and it [53:12] might be uh 140 something 152 to racks [53:16] etc. So then we would say, okay, we're [53:18] going to pull out that pod and we're [53:20] going to replace it with a whatever it [53:22] is, TPU 12 or 13 or 14 pod. It might not [53:26] be a perfect fit and then we have to, in [53:29] other words, the new pod's footprint [53:31] might not be a perfect fit for the old [53:33] pod's vacancy. We we then just have to [53:36] account for that. We have to figure out [53:37] how we would retrofit. We can't plan [53:39] that because we don't know what that [53:40] many generations of TPUs are going to [53:42] be. Uh so then it's again a bit of an [53:44] art to figure out how we would and it's [53:46] actually uh hard work figure out how we [53:48] would decom and then replace as quickly [53:50] as possible with with new TPS. [53:53] >> You've written publicly about uh open [53:55] standards. Um can you say a word on [53:57] that? [53:58] >> Yeah. So I think that's the we we talked [54:00] about this um interoperability question [54:03] and so for us while we support vertical [54:06] integration and we make it possible to [54:08] you know extract just as much [54:09] performance as you want for us it's [54:12] really important to um not force lock in [54:15] and not force a sort of walled garden in [54:18] terms of how the system works end to [54:20] end. So you know for mean I'll give one [54:23] example at Google we've developed a um [54:26] model development framework called Jax [54:29] uh we like it a lot we think it's really [54:31] really good and we use it extensively [54:33] internally many of our customers like [54:36] PyTorch right and so one one thing we [54:38] could say is hey if you want to run on [54:40] TPUs um you got to use Jax it's the best [54:44] I'm being facicious a little bit but [54:47] maybe it is maybe it isn't we think it's [54:49] the best and that's only choice because [54:52] it's it's so great or we could say look [54:55] if you like Jax we like Jax if you like [54:57] Jax you can use it but if you like [54:59] PyTorch we have torch TPU where your [55:03] unmodified models etc can can run we've [55:06] and you know we've seen this uh [55:07] throughout history uh actually with many [55:10] many examples I've used the example of [55:12] um IP why did the internet protocol win [55:15] there were actually many competing [55:16] protocols to IP this is ancient history [55:19] in the 70s and early8s piece. Why did IP [55:22] win was because it was uh open standard [55:25] and it was this narrow waist to the [55:27] hourglass where anything could run on [55:29] top in terms of software and anything [55:32] could run on underneath it in terms of [55:34] hardware. So open standard interoperable [55:37] anyone who brought IP could plug into a [55:40] router port and become part of the [55:42] internet like that. It was beautiful and [55:44] that's what allowed the internet to [55:46] explode in its uh growth and reach [55:49] across the world. So we really do [55:50] believe in those uh open standards in [55:52] terms of plug-in points. Now if you want [55:54] to plug in something highly specialized, [55:57] you can right if if you think that [55:59] you've got a better thing to plug into [56:01] our our framework, you absolutely can. [56:03] But we want to really support open [56:05] standards ideally open source around it [56:08] as well. [56:08] >> It's too big and too important of a [56:10] buildout to be you know any one vendor's [56:12] closed proprietary st. Yeah, we really [56:14] we really believe that and it's got to [56:16] be one of choice and that's also why for [56:17] example we fully support and have uh [56:20] TPUs, GPUs, other accelerators etc. [56:23] >> Yeah. How has day-to-day life for for [56:25] your team changed with AI? Where is it [56:28] changing your your function the most? [56:31] >> So I think the the whatever you want to [56:34] say the easiest answer is on the [56:36] software engineering side where I mean [56:38] it's been documented uh externally quite [56:41] a bit. Uh so I think that we are at [56:43] Google and my team using it to uh great [56:46] effect in in terms of our software [56:48] development uh uh capabilities but also [56:52] uh frankly uh test rollouts even uh [56:56] helping with design etc. But there have [56:58] been uh on the hardware side which has [57:00] maybe gotten less uh external coverage [57:04] also significant change. In other words, [57:05] my my hardware engineers, I was just [57:07] looking at the numbers earlier today, in [57:09] fact, are using AI using whatever the [57:12] the token counts if if you want, not the [57:14] best metric, but it's still a metric. [57:16] Our hardware engineers are using AI as [57:19] much as the software engineers are. And [57:21] productivity has gone up. Time to you [57:23] know, time from design kickoff to tape [57:26] out is shrinking. Time for bringup is [57:29] shrinking. So another where's [57:31] productivity is going up uh [57:33] significantly but then other maybe less [57:36] expected changes how we do data center [57:39] design uh has uh changed significantly. [57:42] In other words how we and how we [57:44] evaluate you you mentioned hey do we do [57:46] a gigawatt building or 100 megawatt um [57:49] campus or 200 megawatt campus. In the [57:52] past, this would be very detailed, very [57:55] sort of um spreadsheet driven, [57:57] human-driven processes, and it still is [58:00] to some extent, but there's a lot of AI [58:02] now involved that really streamlines the [58:04] process in terms of planning and uh [58:08] development as well. [58:09] >> Interesting. So, [58:10] >> like reasoning models or [58:12] >> um I wouldn't say quite yet reasoning [58:14] models. It's not it's not replacing [58:15] human judgment, but it's um making it [58:18] much easier to bring the necessary [58:19] information together like uh in in in [58:22] one place and uh basically uh put the [58:25] right information in front of the humans [58:27] making the decisions. [58:29] >> Okay, I'm going to bring us home with [58:30] two kind of out there kind of fun [58:32] questions. [58:33] >> Yeah. [58:33] >> Um question number one, uh orbital data [58:37] centers. Um I've seen that you know [58:39] Google is thinking about this quite [58:41] seriously and you want me to comment on [58:43] that but I mean does the fact that [58:44] people are seriously running the numbers [58:46] on orbital computes mean that there are [58:49] kind of serious binding constraints on [58:50] Earth and and what do you think of of [58:52] orbital data centers? [58:53] >> Yeah, it's an exciting direction. We are [58:55] actually pursuing it and we've uh [58:57] referred to it uh with no tongue and [58:59] cheek as a as a moonshot as one of our [59:01] big um big efforts that uh we're excited [59:03] about investing in. uh goes back to the [59:06] earlier part of the conversation where [59:07] you raised I think correctly that in [59:08] terms of fundamental constraints um [59:11] energy and energy production is a key [59:14] key challenge and so really um bottom [59:17] line is from a fundamentals perspective [59:20] um in space um you have something like [59:24] 40% more power available because of lack [59:27] of attenuation in the atmosphere etc. In [59:29] other words it's just more solar [59:30] capacity what 1.4x 4x. So that's one [59:33] part of it. But but in a um [59:35] sun-synchronous orbit, you have 98 to [59:39] 100% coverage of sunlight on your solar [59:42] cells relative to 28 30 maybe 35%. [59:49] Um on land, right? So in other words, [59:51] just this enormous amount of power, [59:53] obviously the sun, you you take the 1.4x [59:55] 4x and you take the 3 to 4x uh in terms [60:00] of number of hours per day, you remove [60:03] batteries more or less from the equation [60:06] and now you have the potential for [60:08] something that can really and and of [60:10] course carbon-f free lots of benefits, [60:13] lots of challenges. [60:14] >> Yep. [60:14] >> Right. Lots and lots of challenges. So [60:16] in other words, okay, now what about [60:18] cooling? Uh you know, naively you might [60:20] think cooling in space is easier. It's [60:22] actually harder. Uh reliability. We [60:24] talked about how these things fail [60:26] sometimes. So repairs uh is becomes [60:28] harder not not impossible but it becomes [60:30] harder in space robot up with the with [60:32] the cluster [60:33] >> and exactly so perhaps there'll be uh [60:35] this model uh is a is a promising uh [60:37] direction. Uh redundancy will probably [60:39] be your friend. Uh your uh pre question [60:42] on free space optics is now going to [60:44] become reality. We're probably not going [60:46] to be stringing fiber uh between these [60:48] components. So it will actually be free [60:50] space and we're going to have the lasers [60:52] pointing at the um receivers and [60:54] calibrating in in real time. These are [60:57] there are no showstoppers here. No [60:58] fundamental showstoppers. [61:00] >> Last question. I've had this image of [61:02] the ineffable uh big supercomputer that [61:05] you built them [61:07] >> um in my head this whole time. And so [61:10] here's a question. [61:12] >> 10 years from now, what will the most [61:14] frontier supercomputer look like? [61:17] >> Oh my gosh. Yeah. 10 years is right at [61:19] that edge where it's um uh I would say [61:23] in all sincerity probably some people [61:26] are thinking about what the 2036 [61:29] computer looks like. We might have a few [61:31] thinking that far ahead but wow I um the [61:36] cone of uncertainty there is is uh way [61:38] way too wide. Like in other words if you [61:40] look at the trends um the level of [61:42] integration is going to be tremendous. [61:45] So, you know, we saw the beautiful fiber [61:46] and the um VR200 uh racks that we put [61:50] together for ineffable. [61:52] My my guess is it's going to be more [61:54] integrated, less fiber to I won't say no [61:57] fiber in 2036, but I think that probably [62:00] from a rack perspective um the rack will [62:02] look um much more tightly integrated and [62:05] the amount of fiber there'll be maybe a [62:07] small fiber bundle coming out of the [62:08] rack. And then uh so in other words, one [62:11] one view you might have from a modular [62:13] manufacturing perspective is these racks [62:15] likely in 2036 are going to be [62:18] manufactured centrally. And so whether [62:20] they have 72 or 144 or 288 or probably [62:25] more you know 576 52 pick your favorite [62:28] multiple of GPUs since we talking about [62:32] the an ineffable case but then be the [62:34] same for the TPUs deeply integrated into [62:37] a rack would a rack be multiple [62:39] megawatts you know we don't know exactly [62:41] how but 2036 we get to we get to imagine [62:43] things could be multiple megawatts in a [62:45] single rack and now you bring water and [62:48] you bring power and you ing fiber, [62:50] right? You you you wheel that uh rack [62:52] in, you plug those three things in and [62:55] you're off uh off to the off to the [62:57] races uh as well. [62:59] >> Okay. So, it's not going to be like a [63:00] big alien orb then [63:02] >> up in space by 2036. I wouldn't say that [63:05] it's not possible for 2036 for it to be [63:08] then directly launched into space uh and [63:10] then maybe um taken by a robotic arm at [63:14] some space station and then plugged into [63:16] the right module. Yeah, maybe that for [63:18] 2036. [63:19] >> Fun [63:19] >> fun fun fun to think about. [63:20] >> Um I really enjoyed this conversation [63:23] where this is [63:24] >> the most intense biggest capback spilled [63:28] out in history, but I also think it's [63:30] it's a technology [63:32] >> revolution [63:33] >> revolution that is a thing of beauty and [63:35] I think [63:36] >> you have such a grasp over you know the [63:39] beauty of the technology and and the [63:41] binding constraints and and how to [63:42] balance this this all. Uh Google is in [63:45] good hands. Thank you for taking the [63:47] time to share what you're doing. [63:48] >> Ton of fun. Ton of We get to live. We [63:50] get to live through this and we get to [63:51] define it. So So thank you very much. [63:52] This is a great conversation. [63:53] >> Thank you. [63:54] >> Yeah. Thanks.