Tech Giants Turn to AI Agents to Manage Hyper-Scale Infrastructure

科技巨头转向AI智能体管理超大规模基础设施 微软、Meta和谷歌的工程师发现，用专门的智能体工作流和本地代码库记忆取代人工故障排查，对于防止系统级联故障并维持超大规模迭代速度至关重要。 该专家小组探讨了科技巨头和自动驾驶系统在扩展AI智能体与物理基础设施方面面临的运维挑战。核心议题包括利用本地代码库记忆和多智能体编排层来应对智能体的上下文限制，以及减少SRE繁重琐事的相关策略。与会嘉宾还将软件可靠性与严格的自动驾驶汽车安全框架进行了对比，同时强调了支撑超大规模AI集群所需的数据流动、遥测数据保留及网络SLO要求。 企业系统在超大规模下面临严峻的“上下文上限”，除非通过恰当的上下文工程和结构化语义进行引导，否则会导致AI智能体低效消耗Token。 (0:01) 为防止超大单体代码库出现上下文饱和，本地记忆和领域特定技能被编码到代码库中，使智能体能够迅速上手。 (0:07) 谷歌部署了针对特定故障领域的专用智能体，而不是针对所有基础设施运维问题采用单一庞大的上下文窗口。 (0:07) AI智能体加快了故障停机期间SRE事件的排查解决，并将非关键后台Bug的处理时间从数天大幅缩短至数小时。 (0:47) 由于持续运行非确定性模型的成本极高，微软将智能体的学习成果提炼为标准编译代码，用于常规遥测数据处理。 (0:47)

Engineering leaders from Aurora, Google, Meta, and Microsoft report that operating hyper-scale AI infrastructure requires rethinking software reliability, context engineering, and physical deployment as compute scales toward gigawatt-level clusters. Context Engineering and Hierarchical Routing at Scale (0:01) As software environments expand to hyper-scale enterprise infrastructure, AI agents quickly hit a context ceiling where traversing massive dependency graphs burns tokens and triggers erratic behavior. To mitigate context saturation, Meta checks local memories and domain-specific knowledge directly into code repositories while maintaining agent-legible dependency maps. Similarly, Microsoft found that providing structured context—such as human onboarding manuals and operational guides—matters more than prompt engineering, enabling agents to parse database queries and formulate diagnostic routines. Deterministic Coding and SRE Toil Reduction (0:47) In operations and incident management, Google deploys targeted, domain-specific agents coordinated by an orchestrator to isolate hardware, software, and networking bugs rather than routing all telemetry through a single monolithic agent context. In Azure Site Reliability Engineering, AI agents accelerate root-cause analysis during major outages and have reduced low-priority bug resolution queues from seven days to seven hours. Because continuous non-deterministic model execution is cost-prohibitive, Microsoft teams use agentic workflows to explore and discover failure patterns, then convert those learnings into deterministic compiled code to handle recurring telemetry processing. Safety-Critical Rigor Versus Cloud Product SLOs (18:35) Risk tolerances vary fundamentally between physical autonomous operations and enterprise software. Aurora adheres to a formal Safety Case Framework where hundreds of explicit safety claims must be verified by empirical evidence before autonomous 80,000-pound trucks can operate on public roads at 65 mph, noting that standard 99.9% uptime targets would lead to catastrophic failures. Conversely, product and cloud teams treat service interruptions as manageable outages, focusing on strengthening automated testing gates to prevent rapid agentic code generation from deploying bugs into production faster. Network Critical Paths and Physical Infrastructure (24:17) With AI advancements constrained by data motion across tens of thousands of co-processors, the network sits directly on the computational critical path. To maintain service-level objectives against line-speed visibility limitations, modern AI architectures implement multi-path message splitting across wide switch fabrics and provide diagnostic baselines so clients can differentiate infrastructure regressions from application errors. On physical data center floors, technicians installing millions of cables now coordinate directly with agents possessing full network topology views to confirm link health without waiting for remote engineering oversight. While AI agents are now viewed as an irreversible operational necessity to manage complex systems and accelerating power footprints, infrastructure providers still face unresolved hurdles in balancing the high cost of non-deterministic exploration with hardware supply constraints and strict verification standards.

来自 Aurora、Google、Meta 和 Microsoft 的技术专家指出，AI Agent 在超大规模企业基础设施中的落地正面临上下文饱和、非确定性运行成本以及硬件网络瓶颈等多重制约；工程团队必须依托上下文工程、分层专业化路由以及确定性代码提炼来规避级联风险，并在吉瓦级算力扩张与严苛物理安全标准之间寻求平衡。 突破上下文瓶颈与分层编排架构 (0:01) 在超大规模企业环境中，单一系统变更会触发跨多层抽象的级联效应，导致 AI Agent 消耗大量 Token 盲目遍历基础设施却难以定位故障。对此，Microsoft 在网络领域的实践表明，上下文工程的重要性远超提示词工程（Prompt Engineering）；工程团队通过向 Agent 注入员工入职文档和排障指南，并让其解析指南内的数据库查询语义，使其能够自主重构超越单一专家水平的高效查询，并通过迭代反馈闭环持续纠正错误。 SRE 运维降本与探索逻辑的代码化提炼 (0:47) 在 Azure 的 SRE 运维中，大语言模型通过结构化基础设施代码加速排障，将传统低优先级积压缺陷的解决周期从 7 天大幅压缩至 7 小时。然而，持续运行非确定性 LLM 的经济成本过高，Microsoft 团队采取的策略是利用 Agent 进行开放式故障探索并发现模式，随后将这些经验沉淀并蒸馏为标准的编译代码，以确定性逻辑低成本处理日常遥测与数据降维。 针对基础设施中难以避免的复合型“黑天鹅”级联故障，小组认为引入 Agent 属于不可逆的单向决策。防御级联扩散的关键在于将传统工程管理规范（如清晰指标、测试集和退出标准）深度嵌入 Agent 提示词，驱动其生成具备自检能力的代码。Jenny 同时指出，虽然 Agent 加速了根因排查，但在快速迭代发布中，若自动化门禁不足，Agent 快速生成的代码亦会引入新缺陷，因此亟须利用 Agent 自身强化防御性的代码准入系统。 算力扩张重塑成本结构与安全框架差异 (15:49) 在前沿大模型训练领域，成本层级已形成“计算 > 存储 > 网络 > 可观测性”的绝对序列，行业逻辑从追求节省 2% 至 5% 或 1 至 2 兆瓦能耗，全面转向在 12 个月内快速部署并利用数个吉瓦（Gigawatt）规模的电力基建。在极速扩张下，“记录全部数据”成为通用准则，因减少调试时间所挽回的巨额训练停机损失远超遥测存储成本。与此同时，行业也依赖 AI Agent 来过滤和解析呈毫秒级涌入的庞大观测数据。 物理系统的安全要求与普通软件的可用性指标存在本质分歧。Aurora 专家阐述，一辆以 65 英里/小时行驶的 80,000 磅重卡无法容忍系统崩溃，行业通用的 99.9% 正常运行时间 SLO 对自动驾驶而言意味着每年数小时的灾难性事故。为此 Aurora 采用“安全案例框架”（Safety Case Framework），将数百项明确安全主张与数千项实测证据逐一映射，只有软硬件与路线验证完全闭环才准许部署；并且由于车载合规认证周期严苛，其计算硬件往往滞后前沿消费级芯片数年，算法必须在受限的物理算力预算内完成极致优化。 数据流动瓶颈与大规模集群网络验证 (24:17) Dave 强调，当前 AI 的核心演进瓶颈本质上是“数据流动问题”。由于海量并行计算强依赖数以万计的协处理器协同工作，网络已直接嵌入计算的关键路径中。现代 AI 网络采用多层交换机容错架构与专门的传输协议，将消息拆解切分并在多条可用路径上并发路由，以此向计算上层提供可靠的 SLO 保证。在数据中心物理现场，一线施工人员（Smart Hands）已开始直接与具备全网拓扑视角的 AI Agent 协同作业，由 Agent 实时校验数百万根线缆的物理连接与链路健康，逐步替代远端网络工程师的人工介入。 Sad 进一步指出，在面临供应链、资本开支与电力极度紧张的背景下，集群网络已从单端点互联跃升至十万卡级 GPU 的同步并发通信。为确保交付可用性，云服务提供商在交付客户前，必须在数千至万卡级集群上端到端执行合成基准训练任务，比对理论运行时长模型以排查深层软硬件故障；同时还需向客户开放底层遥测与诊断基线，协助用户准确界定性能回退究竟源自基础设施故障还是应用层缺陷。 总体而言，引入 AI Agent 虽已成为巨头在技术竞争中无法回避的必选项，但大模型推理的非确定性、高昂的计算成本以及自动生成代码带来的潜在级联缺陷，依然制约着其自主闭环能力。无论是在自动驾驶的绝对安全边界下，还是在十万卡级集群的数据通信瓶颈中，通过严苛的基准测试、确定性代码沉淀与精细的上下文工程对 Agent 进行边界约束，依然是确保基础设施稳定性的核心工程底线。

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 7 月 2 日发布。页面描述显示，这是一场在 @Scale 行业大会上举行的现场圆桌论坛，汇集了来自 Aurora、Google、Meta 和微软等前沿科技公司的工程专家。 ## TL;DR 本次圆桌讨论深入探讨了 AI 智能体在超大规模云基础设施与自动驾驶运维中的前沿落地与严峻挑战。来自微软、Meta、Google 和 Azure SRE 的专家分享了如何利用故障手册冷启动智能体、在 Monorepo 沉淀领域技能、编排垂类专家智能体，以及将动态非确定性探索沉淀为低成本确定性代码的实践。随后，Aurora 代表对比了传统互联网高可用性与自动驾驶重卡对“绝对功能安全”的零容错哲学，指出工程团队需针对物理毁灭进行防御性建模。最后，论坛探讨了十万卡智算集群面临的供电与物理交付瓶颈，强调通过自动化门禁、智能体协同布线以及基准测试来应对级联故障与责任边界界定。 ## Atmosphere & style 整场讨论展现出硬核务实、充满工程敬畏感的技术研讨氛围。现场嘉宾不仅毫无保留地抛出真实的运维账本和故障痛点，还以风趣而尖锐的行话幽默调侃“凌晨两点修网络”以及“代码生成狂热”背后的现实风险。在探讨自动驾驶关乎人命的物理重载极限与数十万卡集群的吉瓦级供电扩张时，对话流露出极强的危机感与冷静克制，为听众呈现了一场真实前沿技术与物理世界残酷约束相互碰撞的思想盛宴。 ## Bulletin - [00:01](https://www.youtube.com/watch?v=HrYag7rM65s&t=1s) 超大规模基础设施的复杂度正迅速击穿大模型的上下文窗口，微软、Meta 与 Google 正通过注入内部排障指南、将技能与记忆沉淀入 Monorepo，以及多智能体专业化编排等上下文工程手段打破该瓶颈。 - [09:05](https://www.youtube.com/watch?v=HrYag7rM65s&t=545s) 微软与前沿团队确立了由非确定性向确定性代码蒸馏的生命周期，先利用高成本的 AI 智能体探索动态未知场景，随后提炼沉淀为高吞吐、低成本的编译代码执行常规数据缩减。 - [14:41](https://www.youtube.com/watch?v=HrYag7rM65s&t=881s) 大模型训练开销以算力为主导，推动可观测性策略从抠细节转向“全量打日志”以缩短数万卡集群的排障时间，同时数据中心正迎接数个吉瓦级电力的调度挑战。 - [17:26](https://www.youtube.com/watch?v=HrYag7rM65s&t=1046s) 自动驾驶卡车将绝对物理安全置于常规可用性之上，Aurora 引入航空与核电领域的“安全案例框架”，严格执行针对飞石、断线和爆胎等极端物理失效的降级停靠设计。 - [20:37](https://www.youtube.com/watch?v=HrYag7rM65s&t=1237s) 随着 AI 辅助编码将发布周期缩短至周级，缺陷外溢风险显著增加，工程团队必须以 AI 制 AI 并全面重构质量门禁，筑牢代码准入与运行时防御体系。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=HrYag7rM65s&t=1s) 在超大规模云基础设施中，微小变更即可触发跨抽象层的级联反应，单体通用智能体易盲目遍历耗尽 Token，必须依赖上下文工程而非单纯的提示词来落地。 - [01:31](https://www.youtube.com/watch?v=HrYag7rM65s&t=91s) 微软将员工入职指南和内部排障手册灌入智能体上下文，驱动推理智能体理解底层数据库语义并自动生成新型复合查询，挖掘资深专家未曾总结的故障线索。 - [03:07](https://www.youtube.com/watch?v=HrYag7rM65s&t=187s) Meta 将团队专属的技能与记忆沉淀于 Monorepo 代码仓，并构建对智能体友好的跨系统依赖索引，避免代码库全量检索时的上下文压缩与信息丢失。 - [05:02](https://www.youtube.com/watch?v=HrYag7rM65s&t=302s) Google 针对纵深极大的全栈基础设施采取专家智能体分工机制，由顶层编排器按异常特征动态调度硬件、网络和上层软件专属智能体协同排障。 - [06:31](https://www.youtube.com/watch?v=HrYag7rM65s&t=391s) Azure SRE 实践表明，智能体不仅在重大故障 Bridge 会议中提供关键排障线索，更将长尾低优先级 Bug 的平均处理周期从 7 天大幅压缩至 7 小时。 - [09:05](https://www.youtube.com/watch?v=HrYag7rM65s&t=545s) LLM 推理单次成本高昂，微软主张将智能体作为模糊场景的先遣探索队，再将其提炼固化为可编译的确定性硬编码逻辑，实现经济高效的大规模生产执行。 - [10:47](https://www.youtube.com/watch?v=HrYag7rM65s&t=647s) 超大规模 AI 硬件故障模式随生命周期动态变化，基础设施系统需要兼备模式库快速防御与深层未知异常捕捉的双重自愈能力以保障负载可靠。 - [12:31](https://www.youtube.com/watch?v=HrYag7rM65s&t=751s) 自动驾驶必须经受车规级审查，车载芯片性能落后于前沿算力数年，Aurora 依靠极其紧绷的算力预算推演算法，倒逼低功耗与低成本的软硬件设计。 - [14:41](https://www.youtube.com/watch?v=HrYag7rM65s&t=881s) 万卡训练集群中算力成本占据绝对主导，全量记录日志挽回哪怕几十分钟的集群宕机即可覆盖日志存储开销，基础设施策略全面转向吉瓦级算力调度。 - [17:26](https://www.youtube.com/watch?v=HrYag7rM65s&t=1046s) 区别于云服务容忍故障换取迭代，重卡自动驾驶将“安全”而非“可靠性”奉为铁律，在 80,000 磅高速运行场景下以物理毁灭为假设构建平稳降级刹停底线。 - [19:18](https://www.youtube.com/watch?v=HrYag7rM65s&t=1158s) Aurora 借鉴航空与核电领域的“安全案例框架”，通过成千上万个严格测试用例严密对应数百项安全声明，证明物理硬件、传感器与路线运行的绝对安全性。 - [20:37](https://www.youtube.com/watch?v=HrYag7rM65s&t=1237s) AI 辅助编码将发布节奏压缩至周级别，倒逼团队利用 SRE 智能体反向加固发布准入质量门禁，防止低频缺陷演变为生产环境的高频级联故障。 - [22:06](https://www.youtube.com/watch?v=HrYag7rM65s&t=1326s) 面对系统数十年的历史包袱与黑天鹅级联风险，业界正将明确指标、准出标准和防御性编码等经典工程管理规范深度注入智能体 Prompt 之中。 - [24:17](https://www.youtube.com/watch?v=HrYag7rM65s&t=1457s) 分布式 AI 训练的瓶颈本质是数据流动问题，基础设施正发展高容错多路径传输网络保障 SLO，并借助 AI 智能体指导机房现场技师进行数百万级光纤布线。 - [27:16](https://www.youtube.com/watch?v=HrYag7rM65s&t=1636s) 超大规模集群交付由单机升级为十万卡级别，必须经历从物理光纤、网络互联、合成基准测试到全栈负载周期的全维度偏执型实测验证。 - [28:57](https://www.youtube.com/watch?v=HrYag7rM65s&t=1737s) 排障中想要为底层网络“证伪”极度困难，系统必须建立健全遥测与一致性健康基准（Common Baseline），以精准区分基础设施性能回退与上层应用软件 Bug。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=HrYag7rM65s&t=1s) 智能体运维的“上下文天花板”与扩展困境 主持人向在座来自微软、Meta、Google 等公司的嘉宾抛出了一个核心议题：在基础设施运维中，智能体（AI Agent）面临的“上下文天花板（context ceiling）”究竟是怎样的形态？主持人对比了初创公司与超大规模云厂商（Hyperscalers）的巨大差异：在初创公司，系统拓扑规模较小，依赖图谱一目了然，甚至完全可以塞进大模型的上下文窗口，工程师彼此坐在旁边，彼此系统的工作流极易掌握；然而一旦扩展到超大规模云基础设施，任何一处微小的改动都会在极其复杂的抽象层之间引发级联反应，没有任何单个工程师或智能体能够全面映射。 面对这种复杂度，通用智能体容易“撞墙”——它们可能会消耗海量 Token 盲目遍历系统，做无用功，而经验丰富的人类老手却能在数秒内看穿根因走捷径。主持人因此提问：行业应如何思考重建智能体所需上下文的机制？何时应当让人类专家介入，又何时放手让智能体“磨砺爪牙”去试错？ ```mermaid flowchart TD accTitle: Context Ceiling in Agentic Operations accDescr: Comparison of agent operations between startup scale and hyperscaler scale. subgraph Startup["初创公司模式 (Startup Scale)"] S1["小规模系统拓扑"] --> S2["依赖图谱清晰"] S2 --> S3["完全装入上下文窗口"] end subgraph Hyperscale["超大规模集群模式 (Hyperscale)"] H1["单点微小变动"] --> H2["跨多抽象层级联反应"] H2 --> H3["上下文窗口迅速耗尽 / 盲目遍历"] H3 --> H4["需要上下文工程与领域记忆"] end ``` --- ### [01:31](https://www.youtube.com/watch?v=HrYag7rM65s&t=91s) 微软实践：用人类排障手册与查询语句反哺上下文 微软的代表指出，微软已在内部（尤其是网络运维领域）广泛使用 AI 智能体来自动化日常运维。他强调，智能体落地的关键在于“上下文工程（Context Engineering）”而非仅靠“提示词（Prompting）”——投喂什么内容给上下文，决定了系统的上限。 微软走通的一条绝佳路径，是直接利用最初为了新入职人类工程师编写的内部排障指南（Troubleshooting Guides）。通过将这些运维文档灌入智能体的上下文，推理智能体能够顺藤摸瓜分析以往资深工程师编写的各类数据库查询（Database Queries），从而精准提炼出底层数据表的业务语义。更有甚者，智能体还能基于这些语义重组、生成全新的复合查询，挖掘出哪怕顶尖人类专家此前都未能总结出来的排障信息。 - **快速冷启动**：直接复用存量的人类员工入职指导文档与排错手册，实现智能体快速冷启动，使其独立接管突发事件而无需人类每一步并排陪跑。 - **渐进式学习机制**：智能体会像初级工程师（Junior Engineer）一样犯错，团队需要建立反馈学习闭环，在下一轮事件发生前，通过修正上下文、调整提示词或注入补充指南规范智能体行为。 --- ### [03:07](https://www.youtube.com/watch?v=HrYag7rM65s&t=187s) Meta 方案：Monorepo 中的记忆与技能沉淀 来自 Meta 的 Mark 分享了在超大型单体仓库（Monorepo）下应用智能体的经验。随着过去半年 AI 工具使用量的爆发式增长，团队发现初期智能体会迅速吃满上下文窗口；一旦上下文被迫压缩，智能体就会开始丢失重点、进行混乱的操作。 为了解决此问题，Meta 内部各业务线团队开始将各自项目专属的“记忆（Memories）”与“技能（Skills）”直接提交并沉淀在 Monorepo 的代码仓内。智能体在遍历代码库时，能够直接挂载该目录下的特定经验，迅速拉齐到一个中等经验人类工程师的技术认知水平。此外，针对大范围跨系统调试中仍然存在的上下文超限问题，Meta 正在探索把相互调用的依赖图谱转换为“对智能体友好（Agent-legible）”的目录索引结构。这样智能体在定位跨系统问题时，无需再耗费极长时间去全文检索整个代码库，只需快速查阅索引目录，即可精准跳转到对应子系统直接拉取核心记忆。 --- ### [05:02](https://www.youtube.com/watch?v=HrYag7rM65s&t=302s) Google 实践：模块化专业智能体编排与全链路排查 来自 Google 的代表从客户支持与系统分层的角度分享了应对策略。以 Meta 这样部署在 Google 云上的超大规模客户为例，大客户往往希望在出现严重故障时直接连线核心工程团队，而非流转于普通的客户经理或客服支持层，但“工程师也需要睡觉”。 为此，Google 正在重度投入构建高度专业的智能体集群。由于大型 AI 基础设施的技术栈纵深极大——从最底层的复杂专用硬件、光网络，到控制面乃至顶层的 AI 应用程序——将所有系统的全量日志硬塞进单一超大智能体显然不切实际。Google 采取了专业化分工加统一编排的架构： - **细分领域专家智能体**：分别构建专精于底层硬件故障、网络故障以及上层软件故障的独立智能体，各自掌握垂类上下文与日志权限。 - **调度编排器（Orchestrator / Router）**：顶层编排器扮演路由角色，根据异常特征动态尝试调度不同的专家智能体，逐步排查，直至形成能够准确答复的闭环结论。 - **服务对外输出**：这套多智能体协同排障系统目前已被 Google 内部技术支持团队与研发工程师深度使用，且未来计划直接作为能力开放给外部企业客户自用。 --- ### [06:31](https://www.youtube.com/watch?v=HrYag7rM65s&t=391s) Azure SRE 智能体实践：重构 MTTD/MTTR 的成功标准 针对事故管理中智能体的角色，主持人向 Azure SRE 智能体团队负责人 Jenny 提问：行业传统习惯用“平均解决时间（MTTR）”或“平均缓解时间（MTTM）”来衡量自动化工具的成效，但在生产故障复杂度和智能体开发速度日益剧增的环境下，这种传统指标是否依然适用？智能体在消除复杂性的同时，是否也带来了新的不确定性？ Jenny 对此提出了不同视角。她认为“并不是故障本身变得更复杂，而是系统随时间推移逐渐变得极其庞大与盘根错节”，层层依赖才导致人类直觉在排障时容易卡壳。然而，现代企业生产环境的所有发布绝非在图形界面中随意操作，而是全部以代码形式落盘、经过严格的 Release Gates 灰度发布流程。智能体在解析代码和拓扑图方面具有天然的人类无法比拟的速度优势。 - **重大 P0/P1 事故中的角色**：在十多名核心专家齐聚在线排障会议（Bridge）的关键事故中，智能体并非完全自主接管，而是扮演超级参谋，以前所未有的速度提供极其关键的排错线索（Clues），显著缩短资深团队的协同诊断时间。 - **长尾非严重故障的颠覆性提速**：更具实质性降本增效的是那些优先级没那么高、但持续困扰系统与业务的长尾隐患（Bug）。以往这类排期靠后的问题在人类工单流转中往往需要“耗时整整 7 天（7 days）”，而现在通过 Azure SRE 智能体，直接被压缩到了“7 个小时（7 hours）”内解决，带来了数量级的生产力飞跃。 ### [09:05](https://www.youtube.com/watch?v=HrYag7rM65s&t=545s) 非确定性到确定性：利用 AI Agent 进行探索与代码精简 讨论切入了一个关于大语言模型（LLM）落地机制的核心问题：大模型本质上依赖非确定性（non-deterministic）的方法解决问题，那么在什么节点上，这种动态探索出的经验应该被沉淀为确定性的代码逻辑？而这些确定性知识在何时又会过时失效，从而倒逼系统重启整个学习循环？ 来自微软的嘉宾指出，这不仅是一个系统学习机制的问题，更直接关系到“运行效率与经济成本”。AI Agent 和大模型非常适合被引入到高度动态、缺乏成熟规则的模糊场景中去摸索路径；但若按单次执行结果的成本来算，大模型比人类多年来编写的确定性逻辑与硬编码规则昂贵得多。因此，微软内部鼓励团队建立一种闭环：先让 AI Agent 去承担最前沿的探索性与推理工作，摸清数据与业务逻辑，随后再让这些 Agent 结合传统机器学习技术反哺工程，编写出标准的、可编译的确定性代码，以此将底层的海量数据处理和常规分析自动化。嘉宾特别强调，必须坚持“将 AI Agent 用于推理或处理新奇事物，但始终设法从中提炼蒸馏，并沉淀为简单的编译后代码”，因为这对于控制大规模运行的成本效率至关重要。 ```mermaid flowchart LR accTitle: AI推理沉淀为确定性代码的生命周期 accDescr: 展示从AI Agent的动态探索到提炼生成编译代码，并在知识陈旧时回流的循环过程 subgraph Exploration["探索与推理"] A["模糊/动态业务场景"] --> B["AI Agent 探索与推理"] end subgraph Distillation["提炼与工程化"] B --> C["Agent/ML 自动编写代码"] C --> D["确定性编译代码"] D --> E["低成本高吞吐生产执行"] end E -->|"规则过时/遇到新场景"| B ``` - 核心矛盾：非确定性 LLM 具备强大的柔性推理能力，但单次计算成本远高于传统的确定性代码分支。 - 架构策略：使用 Agent 作为先遣队进行动态分析，随后将其蒸馏沉淀为常规的编译代码执行数据缩减（data reduction），兼顾灵活性与推理预算。 --- ### [10:47](https://www.youtube.com/watch?v=HrYag7rM65s&t=647s) 基础设施故障排查与持续学习：全栈复杂性下的异常自愈 另一位来自大规模云与 AI 基础设施领域的嘉宾给出了截然不同的实践视角：在底层架构运维中，AI 与智能体最核心的用武之地是应对系统故障。在超大规模集群下，AI 加速卡、高速网络、固件、底层软件与复杂配置交织在一起，其极端复杂性导致故障模式根本无法一劳永逸地预判。刚上线第一天（Day 0）遭遇的毛病，与运行 6 个月后因散热、热循环（thermal issue）或硬件老化引发的问题截然不同。 面对如此庞杂的软硬件全栈，团队必须借助持续学习：实时消费集群遥测数据，识别新的失效模式，并在不中断训练或推理作业的前提下进行热修复与规避。一旦摸清规律，这些对策就会被固化为自动化系统的确定性规则。然而，快进到三至六个月后，随着负载和规模进一步膨胀，新的未知故障又会冒头。因此，团队必须打造兼具两套能力的系统：既能调取历史模式库快速执行既定防御动作，又拥有极高的敏锐度去捕捉从未见过的深层异常（spot anomalies），而系统最高的铁律始终是“以一切手段将训练或推理负载恢复至可用与可靠状态”。即便半年前见过的故障形态，在更大规模上重现时，也往往会产生完全不同的连锁反应。 --- ### [12:31](https://www.youtube.com/watch?v=HrYag7rM65s&t=751s) 软硬件研发周期的脱节：Aurora 在自动驾驶卡车上的平衡术 主持人引出了一项所有硬件与系统工程团队都在承受的极限挑战：在产品和软件层面，行业要求团队以极快速度迭代、随时掉头（change on a dime）；然而在底层硬件端，物理研发周期漫长且具有极强的刚性，两者节奏存在根本性错配。 自动驾驶卡车公司 Aurora 的代表结合自身处境坦陈了这一痛点。Aurora 夹在两类完全不同的合作伙伴之间： 1. **传统重卡制造商**：必须在物理卡车底盘上协同布置各类高精度传感器与车载算力单元。 2. **核心芯片供应商**：包括 NVIDIA、Intel、Arm 等半导体巨头，各家有着截然不同的产品演进节奏和风险承受意愿。 其中最严苛的现实阻力在于**车规级认证（automotive-grade approval）**。任何搭载上车的硬件模块都必须通过监管机构漫长严苛的安全合规审查，这意味着车载实际可用的计算芯片永远比消费级或数据中心的前沿水平“落后数年”。Aurora 的工程团队不得不长期进行跨周期的算力推演：一方面要前瞻未来数年后可用的车载芯片能力，另一方面在当下必须恪守极为紧绷的车载算力预算，严格限制算法开销。“在一天结束时，如果你能用更少的资源做同样的事，对所有人都是最好的结果”，这种算力约束反过来倒逼团队尽可能选择成本更低、功耗更可控的车载部件。 --- ### [14:41](https://www.youtube.com/watch?v=HrYag7rM65s&t=881s) 可观测性成本重构：从节省数兆瓦到拥抱吉瓦级算力扩张 话题延伸至可观测性（Observability）的数据洪流与成本平衡。自动驾驶汽车每毫秒都在疯狂吞吐传感器数据，以此构筑人类对系统安全性的底层信任；而在数据中心基础设施中，工程师同样面临困惑：为了看清全栈状态，我们究竟需要将多少观测指标与日志灌入系统？ 来自前沿大模型实验室的嘉宾幽默地感叹“谢天谢地我们现在有了智能体”，并当场算了一笔大模型训练的真实账本：在顶尖前沿实验室里，**算力成本占据绝对统治地位**，紧随其后的是存储成本，再往后是网络，而可观测性开销在总预算中排在最末端。正因如此，在大模型训练中的策略极其干脆——“把所有东西统统打上日志（log it all）”。因为在数万卡规模的集群上，哪怕微小的日志能帮工程师将单次训练故障的定位时间缩短几十分钟，挽回的算力损失就足以覆盖海量日志本身的存储开支。 这一逻辑也折射出过去两年 AI 范式转移对传统成本哲学的彻底颠覆： - **传统数据中心思维**：工程师费尽心机去压榨 2% 到 5% 的能效，或是精打细算如何在既有电力预算中抠出一两兆瓦（MW）。 - **AI 军备竞赛新现实**：团队在接下来的 12 个月内面临着数个吉瓦（GW）级电力的大规模接入与调度。 现在的最高优先级是“集中全力驾驭即将并网的庞大算力洪流”，而不是盯着老旧系统计较边际碎屑。思考决策的最高有效位（highest order bits）已经彻底改写。 --- ### [17:26](https://www.youtube.com/watch?v=HrYag7rM65s&t=1046s) 速度焦虑与生存危机：自动驾驶中的绝对安全 vs 系统 SLA 面对研发速度带来的压迫感，现场嘉宾笑称“我们还有别的选择吗？”对于绝大多数科技巨头与前沿实验室而言，跟不上 AI 进化节奏不是丢失几个百分点市场份额的问题，而是关乎公司生死的“存在性风险（existential risk）”。所有人必须押上全副身家，在每一个环节极限榨取 AI Agent 的生产力以维持竞速优势。 但主持人随即抛出一个尖锐的反差：在互联网与云服务领域，失去速度是存在性风险，可以通过容忍故障换取迭代；可是在自动驾驶重卡领域，“故障根本不是一个选项（failure is not an option）”，系统绝不允许一年出现两个小时的停机。在动辄涉及数百万美元合同、更背负着人类生命的场景下，底层系统依赖错综复杂，客户保障与服务等级协议（SLA/SLO）究竟该如何定义？ Aurora 的代表对此给出了极具分量的回答：在 Aurora 内部，工程师几乎从不使用“可靠性（reliability）”这个字眼，唯一被奉为准则的词汇是“安全（safety）”。因为传统的可靠性关注系统能否持续无误运行，而安全关注的是**系统遭遇灾难性故障时如何收场**。在真实物理世界中，没有任何代码能保证硬件绝对不坏： - 传感器可能被路面弹起的飞石击得粉碎； - 通信线缆可能在剧烈颠簸中瞬间断开； - 重型卡车的轮胎可能在高速行驶中突发爆胎。 系统的第一准则不是假设故障不会发生，而是把所有物理失效场景巨细靡遗地建模：当这台重达 80,000 磅（约合 36.3 吨）的钢铁庞然大物正以 65 英里/小时（约 105 公里/小时）的高速在公路上飞驰时，哪怕核心系统当场死机崩溃，自动驾驶系统也必须拥有一套绝对可靠的备用方案，将其平稳、安全地带停在路边。安全的设计核心，永远建立在对不可避免的物理毁灭与软硬件失效的极限建模之上。 - 物理现实：80,000 磅重卡在 65 mph 巡航速度下具有极大的动能，任何计算宕机或传感器失灵都必须保证降级刹停。 - 研发哲学：以“功能安全（Safety）”取代单纯的“高可用性（Reliability）”，全面针对断线、飞石碎屏、爆胎等边缘极端工况进行防御性降级推演。 ### [19:18](https://www.youtube.com/watch?v=HrYag7rM65s&t=1158s) 自动驾驶卡车的安全案例框架：从航空与核电汲取的零容错哲学 Aurora 在自动驾驶重卡的安全保障上采用了一套极其严苛的“安全案例框架”（Safety Case Framework）。这套框架本质上是一份高度形式化、结构化的声明清单，详尽规定了卡车在各类现实工况下应当表现出的确定性行为。与之对应的另一端则是成千上万个细致入微的测试用例与证据链映射——例如，系统必须通过多组联合测试证明车辆能够足够早地辨识交通信号灯并留出充分的制动响应时间。只有当数以千计的单点测试用例逐一闭环，并严密对应到数百项顶层安全声明时，整个“安全案例”才算正式闭环。这一闭环标志着底盘硬件、车载计算软件、传感器套件以及特定运营路线的完整组合具备了驶入公开道路的资质。 谈及研发节奏时，发言人自嘲团队的进展“并不快，甚至希望能够更快一些”，但他们必须保持极度的审慎。对于常规互联网服务而言，99.9%（三个九）的可用性或许已令人满意，但若将这个标准置换到重型卡车领域，便意味着“一年之中会有几个小时卡车处于撞车状态，这显然是不可接受的”。正因关乎人命与产业存亡，自动驾驶工程团队全面吸收了医疗器械、核电站以及现代民用航空业的高可靠性工程方法论，以此来向自身及公众切实证明系统的绝对安全性。 - **安全闭环要素**：物理车辆硬件、控制软件、传感器套件及特定运行路线（ODD）必须联合完成证据链验证。 - **跨行业安全标准**：借鉴核能、航天航空与医疗领域的工程验证方法，拒绝将消费级软件的可用性容错标准套用于物理重载场景。 --- ### [20:37](https://www.youtube.com/watch?v=HrYag7rM65s&t=1237s) SRE 运维与 AI 研发：用智能体筑牢防线并强化门禁 在云原生与常规互联网产品中，服务短暂宕机固然是糟糕的一天，但通常不至于构成“生存危机”。然而在全面拥抱 AI 的时代，如何向客户做出可靠性承诺成为了所有工程管理者必须面对的考题。从 SRE（网站可靠性工程）与业务负责人的视角来看，对比基准非常明确：在没有 AI 智能体协助的传统时代，一旦线上突发异常（尤其是凌晨两点），值班人员必须设法唤醒对口领域的资深工程师，等待众人接入排障会议，耗费大量时间抽丝剥茧排查根因。而引入运维智能体后，这一故障定界与根因定位的过程被大幅压缩。 ```mermaid flowchart TD accTitle: AI时代研发与交付防御体系 accDescr: 展示从智能体代码生成到高频发布门禁及SRE智能体防御的全流程 subgraph Dev["开发与构建阶段"] A["工程师 + AI 智能体辅助编程"] --> B["超高频代码提交与发布"] B --> C{"强化型质量门禁 (Gates)<br/>指标校验 / 准出标准"} end subgraph Defense["运行时与防御体系"] C -->|"门禁拦截隐患"| D["平稳上线部署"] C -->|"门禁疏漏导致 Bug 外溢"| E["生产环境级联错误风险"] E --> F["SRE 监控智能体"] F -->|"快速定位根因"| G["加固工程防御体系<br/>(Shore up defense)"] end ``` 然而，AI 智能体在提升排障效率的同时，也给研发侧引入了全新的风险敞口。当工程团队大规模使用 AI 智能体辅助编码时，若发布质量门禁（Quality Gates）没有同步升级，缺陷就会快速渗透至代码库并进入生产系统。以往两个月才发布一次版本时，即便存在遗漏的缺陷，错误暴露也是低频且分散的；但在 AI 赋能下，团队能够轻松实现每周发布，缺陷在系统中的暴露与交织频率呈指数级上升。因此，当前业界的重心正经历一场深刻转向：利用 AI 智能体反向“筑牢防御系统”（shore up our defense system），建立更严格的代码准入测试与验证门禁。 - **排障与响应**：运维智能体显著缩短了跨时区、夜间紧急事故（如凌晨两点 PagerDuty 报警）的平均恢复时间（MTTR）。 - **迭代频率与风险共振**：发布节奏由数月缩短至周级别，倒逼团队重构门禁系统以防范 AI 生成代码所带来的缺陷级联。 --- ### [22:06](https://www.youtube.com/watch?v=HrYag7rM65s&t=1326s) 级联故障与黑天鹅：将工程管理规范编码进 AI 智能体 大型科技公司现存的底层系统往往历经数十年演进叠加，日常发生的重大故障绝非单一组件失效，而几乎全是由多重隐患并发引发的“黑天鹅事件”或级联雪崩（Cascading Failures）。面对这种复杂性，Mark 指出，全行业借助 AI 智能体加速软硬件研发与系统部署已是一道“单向门”（one-way door）——这是一场不可逆的产业变革，没有人能够退回手工时代。 既然变革不可逆转，应对级联风险的手段便演变为“以 AI 制 AI”。尽管自嘲听上去极具“AI 狂热味”（AI pilled），但最现实的解法恰恰是部署更多专门的智能体来实时监测异常、斩断故障蔓延的链路。深入本质会发现，驯服 AI 的关键并非神话算法，而是回归经典工程管理的通用最佳实践： 1. **明确指标定义**：清晰界定业务核心指标、系统校验指标（Check Metrics）以及准出退出标准（Exit Criteria）。 2. **提示词工程化**：将传统管理规范与架构约束提炼并注入给智能体的 Prompt 模板中。 3. **架构内嵌防御**：引导智能体在生成业务逻辑的同时，主动将防御性编码、健康检查与异常熔断机制深度固化到系统之中。 在真正达成通用超级智能（ASI）让工程师全员“彻底放假”之前，这种将资深工程规范注入智能体生成的闭环，是当下阻断系统级联崩溃最有效的技术屏障。 --- ### [24:17](https://www.youtube.com/watch?v=HrYag7rM65s&t=1457s) 智算时代的网络底座：数据流动、多路径传输与 AI 协同布线 当整个行业将目光聚焦于 GPU 等算力芯片时，大规模 AI 模型分布式训练的核心瓶颈往往卡在网络基础设施上。网络工程师习惯将自身置于“基础设施金字塔”的最底层，而当前的 AI 技术跃进本质上是一个海量“数据流动问题”（data motion problem）。若要将成千上万颗异构协处理器紧密协同运作，海量参数与梯度必须在高吞吐、极低延迟的网络总线上无损流转，网络因而完全处在所有分布式计算的关键路径上。 与此同时，AI 集群的网络可观测性依然面临经典难题：通信双方发生拥塞或丢包时，最直接的归咎往往是“网络故障”，但要在极高线速（Line Speed）下无死角捕获全网链路状态在物理上极具挑战。当前数据中心正通过两大路径破解这一死局： ```mermaid flowchart LR accTitle: AI时代的网络双向演进架构 accDescr: 展现Networking for AI与AI for Networking两个技术路径 subgraph NetForAI["Networking for AI (底层韧性)"] direction TB N1["大带宽交换网络架构"] --> N2["极简健康指标检测"] N2 --> N3["多路径传输协议<br/>(动态拆包重组/保障SLO)"] end subgraph AIForNet["AI for Networking (上层运维)"] direction TB A1["AI 智能体全局拓扑感知"] --> A2["现场技术人员 (Smart Hands)"] A2 --> A3["数百万光纤线缆协同物理排布"] end NetForAI <-->|"基础设施协同支撑"| AIForNet ``` 在**“面向 AI 的网络体系”（Networking for AI）**方面： - 构建高容错的多层交换矩阵，依托极简的网络健康度量指标促使网络在出现单点故障时实现纳秒级的自适应收敛与动态自愈。 - 研发专门的端到端传输协议，充分调度网络中存在的所有可用物理路径；即便底层个别交换机或链路性能劣化，协议层也可将数据包拆解分流至健康链路并对外提供严格的通信 SLO 保障。 在**“面向网络的 AI 赋能”（AI for Networking）**方面： - 针对超大规模超算集群中数以百万计的物理光纤排布工程，以往需要高端网络工程师在后台监控拓扑、现场运维技师在机房布线的模式被重塑。 - 现场技师（现场俗称“灵巧之手”Smart Hands）如今直接配备 AI 智能体。AI 承接了顶层网络拓扑的全局视野，指导现场人员完成点对点的精准插拔与拓扑自检，极大地缓解了数据中心快速扩建期的人力技能瓶颈。 --- ### [27:16](https://www.youtube.com/watch?v=HrYag7rM65s&t=1636s) 十万卡集群的交付挑战：严苛物理验证与全栈可靠性 Saad 从超大规模基础设施供应的角度总结了当前数据中心面临的多重现实撕裂。算力芯片更新换代的速度令人窒息，几乎每年都有全新的 AI 加速卡问世，驱动着所有企业争先恐后抢占最新算力；然而，现实环境却受到极度紧缩的供应链、资本开支（CapEx）以及电力容量等多重严酷约束。尽管行业身处交付时限与硬件供给的双重高压之下，客户对于系统可用性与 SLO 的门槛却从未降低半步。 以往网络验证只需确保端到端通信无误，但如今云厂商交付的单位已演化为由数万乃至数十万颗 GPU/加速器构成的超巨型集群。将这样一头“算力巨兽”完整交付给上层客户前，基础设施团队必须在物理与逻辑全栈实施近乎偏执的验证： | 验证层级 | 核心工程任务与挑战 | 验证手段与容错机制 | |:--- |:--- |:--- | | **物理设施层** | 百万级光纤与铜缆连接，极易发生虚接、微弯损耗或端口误码 | 现场部署阶段实施全链路拓扑物理自检 | | **网络互联层** | 异构集群网络吞吐不均，个别交换机劣化拖慢全网速度 | 定位亚健康链路并精准分派数据中心运维现场更换 | | **集群计算层** | 协同万卡进行并行通信与梯度同步 | 注入万卡规模基准分布式训练任务（Synthetic Workload） | | **交付履约层** | 客户需要确定性的性能与集群可用时间承诺 | 校验实际训练耗时是否严格符合底层理论建模周期 | 这种端到端能力验证必须从机房物理层、网络拓扑层、硬件计算层贯穿至最终负载层。唯有在整套底座通过高压实测并精准剔除所有隐形硬件缺陷后，整机算力池才能真正签署 SLO 并安全移交至外部客户手中。 ### [28:57](https://www.youtube.com/watch?v=HrYag7rM65s&t=1737s) 基础设施诊断、基准校验与排障挑战 在确保 AI 训练任务的稳定运行方面，仅仅验证基础设施能够成功跑通一个训练任务只是最初始的第一步。更关键且复杂的环节在于，必须为客户提供自我诊断（Self-diagnose）的能力。这需要通过设备端或系统级的遥测数据（Telemetry）来实现，甚至在必要时允许将部分算力容量下线隔离，让用户能够重新运行平台构建的同套诊断套件，从而排查他们遇到的具体问题。 构建多层诊断能力的核心目的，是为了明确区分问题到底出在哪里——究竟是基础设施本身发生了性能回退（Regression），还是属于应用程序或模型代码本身的问题。正如讨论中风趣提及的，做网络与底层系统的人往往一生都在面对各种甩锅说法：“嘿，肯定是网络的问题吧！”（"Hey, it's the network, right?"）。然而在工程实践中，想要“证伪”（Prove a negative）——即证明网络或底层毫无问题——是极其困难的。因此，系统必须建立层层分明的监控与基准测试（Baseline），以快速判断究竟是网络层出现了异常指标，还是能够顺利回滚/对照到一个已知的健康基准；只要通用的基准测试可以正常运行，就能够有力证明“底层栈运转正常，问题应当出在更上层的软件或配置中”。 ```mermaid flowchart TD accTitle: 基础设施与应用故障排查流程 accDescr: 展示从训练故障出现到区分基础设施退化与应用特定问题的诊断逻辑 A["捕获训练故障 / 异常"] --> B["获取系统与设备遥测数据 (Telemetry)"] B --> C{"是否需要下线排查？"} C -- 是 --> D["隔离部分算力容量 (Offline Capacity)"] C -- 否 --> E["在线运行标准诊断套件"] D --> E E --> F{"对照已知基准 (Common Baseline)"} F -- 基准测试失败 --> G["确认为基础设施回退 / 网络异常"] F -- 基准测试通过 --> H["排除底层故障，定位为应用特定问题"] ``` - **两步走排障逻辑**：第一步是验证基础设施能成功运行训练任务；第二步是让客户能利用遥测和诊断套件自我定位瓶颈。 - **证伪的困境**：遇到问题时人们总习惯归咎于底层网络，但排查团队需要依靠明确的健康基准（Common Baseline）来打破这种推测，厘清责任边界。 ### [29:41](https://www.youtube.com/watch?v=HrYag7rM65s&t=1781s) 圆桌讨论总结与致谢 在对全栈复杂性与故障排障的深入探讨后，主持人对各位嘉宾分享的深厚技术积累与洞见表示赞叹，感叹面对如此庞大且精密的系统工程令人心怀敬畏（"Humbled by all of the complexity, humbled by your incredible expertise"）。整场圆桌讨论内容丰富且涉及面广，主持人希望这场思想碰撞能为现场观众提供极具价值的思考线索，以便大家在接下来的 Happy Hour 交流环节中继续深入探讨。最后，主持人号召全场为来自各大公司的专家嘉宾们送上热烈的掌声，圆桌论坛在掌声中正式落下帷幕。 ## Notable quotes > [09:05](https://www.youtube.com/watch?v=HrYag7rM65s&t=545s) 将 AI Agent 用于推理或处理新奇事物，但始终设法从中提炼蒸馏，并沉淀为简单的编译后代码 — 微软代表 > [12:31](https://www.youtube.com/watch?v=HrYag7rM65s&t=751s) 在一天结束时，如果你能用更少的资源做同样的事，对所有人都是最好的结果 — Aurora 代表 > [14:41](https://www.youtube.com/watch?v=HrYag7rM65s&t=881s) 谢天谢地我们现在有了智能体 — 嘉宾 > [14:41](https://www.youtube.com/watch?v=HrYag7rM65s&t=881s) 把所有东西统统打上日志（log it all） — 嘉宾 > [14:41](https://www.youtube.com/watch?v=HrYag7rM65s&t=881s) 集中全力驾驭即将并网的庞大算力洪流 — 嘉宾 > [17:26](https://www.youtube.com/watch?v=HrYag7rM65s&t=1046s) 故障根本不是一个选项（failure is not an option） — 主持人 > [19:18](https://www.youtube.com/watch?v=HrYag7rM65s&t=1158s) 一年之中会有几个小时卡车处于撞车状态，这显然是不可接受的 — Aurora 代表 > [22:06](https://www.youtube.com/watch?v=HrYag7rM65s&t=1326s) 以 AI 制 AI — Mark > [28:57](https://www.youtube.com/watch?v=HrYag7rM65s&t=1737s) 嘿，肯定是网络的问题吧！（Hey, it's the network, right?） — 嘉宾 > [29:41](https://www.youtube.com/watch?v=HrYag7rM65s&t=1781s) Humbled by all of the complexity, humbled by your incredible expertise — 主持人 ## Mentioned Aurora, Google, Meta, Microsoft, Mark, Jenny, Saad, Azure SRE, NVIDIA, Intel, Arm, PagerDuty, @Scale ## Video description Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=HrYag7rM65s)

[9:05] 微软代表: 将 AI Agent 用于推理或处理新奇事物，但始终设法从中提炼蒸馏，并沉淀为简单的编译后代码 [12:31] Aurora 代表: 在一天结束时，如果你能用更少的资源做同样的事，对所有人都是最好的结果 [14:41] 嘉宾: 谢天谢地我们现在有了智能体 [14:41] 嘉宾: 把所有东西统统打上日志（log it all） [14:41] 嘉宾: 集中全力驾驭即将并网的庞大算力洪流 [17:26] 主持人: 故障根本不是一个选项（failure is not an option） [19:18] Aurora 代表: 一年之中会有几个小时卡车处于撞车状态，这显然是不可接受的 [22:06] Mark: 以 AI 制 AI [28:57] 嘉宾: 嘿，肯定是网络的问题吧！（Hey, it's the network, right?） [29:41] 主持人: Humbled by all of the complexity, humbled by your incredible expertise

[0:01] Managing the Agentic Context Ceiling in Enterprise Infrastructure [7:00] Monorepo Scale, Memory Check-ins, and Hierarchical Routing [47:00] Reducing SRE Toil and Managing Non-Deterministic Workflows [13:00] Hardware Constraints and the Observability Data Influx [15:49] Rethinking Cost Models and Speed in the AI Era [18:35] Autonomous Safety Frameworks vs. Product SLOs [22:06] Cascading Failures and Engineering Best Practices [24:17] AI Infrastructure, Data Motion, and Network-Layer SLOs [27:16] Managing Supply Constraints, Baseline Testing, and Telemetry

[0:01] Thank you all so much for being here. Uh [0:03] we've got a lot to talk about. I wish [0:04] that we had five hours for this, but um [0:06] I kind of want to start out with a broad [0:09] question around the context ceiling for [0:12] agentic operations. What that looks like [0:14] for different vantage points within our [0:17] infrastructure. So we can think about [0:19] this like scaling to from startup to [0:22] hypers scale the point where the agent [0:24] is going to hit the wall in operations. [0:26] So at a startup, your system typology [0:29] can fit within the agents context [0:31] window. The dependency graph is fairly [0:33] knowable. You might actually just sit [0:35] next to the person or know that person's [0:37] agent that's sort of working on it at [0:38] that point. Um but when you're operating [0:40] at the scale of any hyperscaler, as you [0:42] all know, any single change is going to [0:45] cascade through layers of abstraction [0:47] that likely that no one person can or [0:49] agent can fully map. So the agent might [0:52] not know what to check for inhouse, [0:54] what's actually affected. It might be [0:56] burning tokens, just traversing [0:57] everything that it can do when there [1:00] might be an engineer out there who can [1:02] short circuit this and know exactly what [1:03] to do in seconds. So we're sort of at [1:05] this like funny middle ground in the [1:07] evolution. You all have experience with [1:09] this at varying degrees of complexity [1:11] and scale. I want to think about how you [1:13] reason about how we build that context [1:16] back in how we think this through uh [1:19] with respect to um uh when we pull in [1:23] the humans when we sort of let the [1:25] agents cut their teeth a bit like how [1:27] are we thinking about this? Anyone's [1:28] welcome to take it away. [1:31] >> Well, sure. So, uh we're using AI [1:34] agentics extensively to automate [1:36] operations at Microsoft particularly in [1:38] like our networking space. That's why I [1:40] have most famili familiarity. Um, and [1:43] many speakers here have said it. It's [1:45] more about the context engineering and [1:47] what goes into context than it is about [1:48] what goes into the prompts. And uh, [1:51] honestly, some of the best documents [1:53] that we've had to start our journey have [1:55] been essentially the troubleshooting [1:56] guides that were originally written for [1:58] onboarding our human engineers. And by [2:00] feeding those into the context, uh, the [2:03] AI agents themselves are actually able [2:05] to look at all of the database queries [2:07] that our human engineers wrote to use to [2:10] try and figure out what was going on. [2:12] They've actually been able to extract [2:14] essentially the semantics of what's in [2:15] those tables from the queries and then [2:18] by using reasoning agents to actually [2:20] sort of reformulate and put those [2:22] queries back together to surface [2:24] information that even we didn't have a [2:26] human expert who could have written that [2:28] query. And so um again by carefully [2:32] picking what we put into the context and [2:34] honestly using documentation that we [2:36] already had that was written for sort of [2:37] onboarding human engineers we're able to [2:40] very quickly bootstrap our agents into [2:42] being able to then respond to uh [2:44] incidents that would come in without [2:46] then a human writing sidelong. Now, of [2:48] course, there has to be a learning cycle [2:49] around that because the agents will make [2:51] mistakes in the same way a junior [2:53] engineer might make mistakes. And so, we [2:55] have to have ways that we can then [2:56] correct either to context the prompt or [2:59] uh provide extra guidelines uh that then [3:01] the system will take into account the [3:03] next time it deals with an incident. [3:04] >> Yeah, it's great. Mark, does that look [3:06] like that resonated with you? [3:07] >> Um, definitely. At at Meta, we have a [3:10] massive monor repo, right? And um I [3:13] think as AI usage has grown especially [3:17] in the last 6 months what you see is [3:19] people start and um you know the copious [3:23] massive that the context window gets [3:24] chewed up right away and agents start [3:26] making a lot of mistakes because it [3:28] basically compacts that context and then [3:29] it just starts doing random stuff. What [3:32] we've seen across much of the monor repo [3:34] is um teams have started to encode [3:37] memories and skills specific to their [3:39] project areas and check them into the [3:40] repo. And what that means is that as [3:43] your agents traverse the repo, it's [3:45] basically easily able to quickly like [3:48] ramp up almost to the level of, you [3:50] know, uh a a semi-experienced, [3:52] moderately experienced human engineer in [3:54] that space and pick that up very [3:56] quickly. Um even with that though, [3:59] right, like we're always trying to uh [4:01] upscale, right, and like kind of like uh [4:04] uh distill um kind of like higher level [4:06] abstractions on top of this because at [4:09] any system of scale when you start [4:11] traversing this entire system, your [4:12] context window still gets chewed up as [4:14] you're doing like bigger and bigger [4:15] debugging runs. And so um I think it's a [4:19] continuous journey right now where we [4:21] like keep making the system better and [4:22] finding the seams. Uh I think the right [4:25] now some interesting things that we're [4:27] looking at are um trying to encode uh [4:30] basically like the the um the dependency [4:32] graph of like which systems are are [4:34] related to each other that call into [4:36] each other into something that's agent [4:37] legible. Uh and that can really help [4:39] basically be like a directory where if [4:41] an agent is being driven to debug a [4:43] system, it can look at this like [4:45] directory and say which other places do [4:47] I need to look and kind of like quickly [4:49] do that. we can always grab the codebase [4:50] or whatever, but you know that takes [4:51] forever compared to just like doing a [4:53] quick lookup and then starting to read [4:55] the memories directly from those [4:56] locations. [4:57] >> Ah, fantastic. It's great. Anyone else [5:00] want to weigh in on this one? [5:02] something uh at least we've been doing [5:04] at Google more and more with with agents [5:06] and where some of this may be relevant [5:08] is you know we we of course you know [5:09] Meta is a big customer of ours and uh [5:12] we're always trying to make sure we [5:13] support our customers in the best [5:15] possible ways and of course Meta like [5:18] other hypers scale companies would want [5:20] us to just talk directly to an engineer [5:23] versus going through support and you [5:25] know accounts team and stuff like that [5:26] right and that's of course becomes a big [5:28] challenge because you know we need our [5:29] engineers to sleep as well So what we've [5:32] been been investing a lot is just kind [5:33] of building some of these agents that [5:35] have are you know sophisticated enough [5:37] that have they have access to the [5:39] entirety of various logs across various [5:41] systems and a lot of the AI scale [5:43] systems are very complex they go from [5:46] the bottom of the stack where they're [5:47] talking to very complex hardware systems [5:49] through networking all the way to the [5:50] control plane of course in the in the in [5:53] the AI applications as well. So we [5:55] basically are building agents specific [5:58] there have specialtities if they look at [6:00] hardware failures they look at software [6:02] failures networking failures and then we [6:04] build kind of an orchestrator that can [6:06] go and so the the entire context does [6:08] not have to be pulled in one large [6:10] agent. It could be there are specific [6:13] agents that are designed for specific [6:14] use cases and then sort of a router that [6:16] goes and says okay you know and it and [6:18] the orchestrator tries different agents [6:20] until it gets to a point where it can [6:22] actually answer the questions. So it's [6:24] something our support team uses a lot, [6:25] our engineers are using a lot and we're [6:27] actually looking at seeing how we can [6:28] make it available even to our customers. [6:31] >> Fantastic. And that's actually something [6:32] I want to come back to too is how we [6:34] think about the role of not only [6:36] incident management but just in general [6:37] the guarantees that we're making to [6:39] customers at scale. Uh you know [6:41] particularly looking at at Sad and Dave [6:44] on on this one perhaps. Um, but I I [6:46] actually want to turn to to Jenny [6:47] because obviously so much of the the [6:48] conversation today has revolved around [6:51] uh involving agents in incident [6:53] management if not letting them steer. [6:54] And so your team is running uh Azure's [6:58] SR agent. Um, one of the things I've [7:00] been really interested in is like a [7:03] normal metric of success that we might [7:05] see with a tool like that is uh a [7:07] reduction in in meantime to resolution, [7:09] reduction in meantime to mitigate. Um, [7:11] and I'm thinking about that and sure it [7:13] makes sense as long as we're sort of [7:15] seeing the usual suspects of incidents [7:16] over time. But what at least I've been [7:19] experiencing is that these incidents are [7:21] getting more and more complex almost [7:23] daily and agents are the the the pace at [7:27] which agents are developing is [7:28] outstripping uh our our ability to have [7:31] a full context on what they're doing. [7:33] And so I could see where the where where [7:35] an agent in that regard might might help [7:37] abstract away some of that complexity, [7:39] but it might also introduce some too. So [7:41] curious of how we think about success [7:43] when the incidents themselves are [7:45] getting more complex. [7:47] >> So I I look at it a little differently. [7:49] I wouldn't say incidents are getting [7:51] complex. I would say over time we our [7:54] systems become complex and hence the [7:56] there are so many dependencies between [7:58] each other [7:59] >> that then it becomes harder to debug [8:01] like that's the human intuition right [8:04] >> but with agents what we have seen is if [8:07] it has access to the source code and [8:08] your deployment topology like almost all [8:10] enterprises are just not making changes [8:12] in the UX when it comes to production [8:14] deployment it's actually a code written [8:16] somewhere gone through release gates and [8:19] agents can reason through code extremely [8:21] Well, so what some things might be [8:24] difficult for a human to comprement, we [8:27] have realized agents are able to do that [8:28] faster. Now in when you talk about MTR [8:33] and all those things, if it's a critical [8:34] enough incident, there are already 10 [8:36] people on an incident bridge. That's [8:37] what happens in almost all big companies [8:39] and they're talking to each other and [8:40] helping each other out. In that context, [8:43] an agent is usually giving you really [8:45] useful cue clues way quicker than you [8:48] would have otherwise got it. and hence [8:50] it helps it faster. But in the case [8:53] where many of us have issues which are [8:55] not that critical but it's still bugging [8:57] someone or somewhere those are the [8:59] issues which in the past would take 7 [9:01] days to finish and now an agent is able [9:03] to do it in 7 hours and that's a massive [9:05] reduction. So it reduces toil overall in [9:09] our experience. [9:09] >> Oh that's great. Fantastic. Um, I want [9:13] to kind of tease one thing out of there, [9:14] too. And it's this point at which All [9:16] right, so if LLMs are all operating on, [9:18] non-deterministic [9:20] approaches to solving problems, at what [9:23] point does that kind of flip into [9:26] knowledge that we would encode in a [9:29] deterministic sense? At what point does [9:32] that knowledge, that deterministic [9:34] knowledge becomes stale and we kind of [9:36] have to go back and the cycle starts all [9:38] over again? How do we even how are you [9:40] thinking about this in your respective [9:42] spaces? [9:45] >> Well, I think it's not just a a learning [9:48] question. It's also an efficiency [9:49] question. So, for example, AI, agentics, [9:52] large language models can be uh great to [9:56] sort of introduce into a very fluid [9:57] situation where you're just trying to [9:58] map out what things are going. But per [10:02] result they are much more expensive than [10:04] having you know coded decision logic or [10:07] any of the other kinds of techniques [10:08] that we've used for many years. And so, [10:11] you know, we're encouraging our teams [10:12] inside Microsoft to sort of do the [10:14] exploratory work using agents, but then [10:17] use those same agents and other machine [10:20] learning techniques to then write code [10:23] that'll automate in sort of the standard [10:25] way uh as much of the uh data reduction [10:29] underlying analysis possible. So we're [10:32] going to use the AI agents for the [10:33] reasoning or the novel things, but we're [10:36] always trying to distill out of that and [10:38] down into just simple compiled code uh [10:40] as many as much of the processing as [10:42] possible. And that ends up being very [10:44] important for cost efficiency. [10:45] >> Yeah, [10:47] >> I'll answer a little differently maybe. [10:49] I I think some of the areas where we've [10:53] found ourselves using agents and AI more [10:56] and more which is relevant to our [10:58] customers is in the in the how we handle [11:00] failures, right? because our you know AI [11:02] AI infrastructure all the accelerators [11:04] the networking everything it and scale [11:07] is just extremely complex right and the [11:09] failure modes aren't the sort of thing [11:10] you would see are different what you [11:12] would see on day zero versus you would [11:13] see in 6 months here and so on and so [11:16] forth right uh cuz it it involves [11:18] hardware it involves configuration it [11:20] involves a lot of software it involves [11:21] firmware so it's the entire stack and uh [11:24] when you're building to a reference [11:25] architecture there are failure modes [11:27] that show up at small scale there are [11:28] failure modes that show up 6 months from [11:30] now cuz there is some thermal issue that [11:32] could be there. So when we train our [11:35] models, a lot of this happens through [11:37] just learning, looking at the data and [11:38] understanding that hey, here's a failure [11:40] mode that we ran into and then figuring [11:42] out how do we remediate that in a way [11:44] that it doesn't impact the training job [11:46] or the inference job and then over time [11:48] those become sort of rules that you [11:50] embed in a system. But you can fast [11:52] forward 3 months, 6 months and a new [11:54] failure mode pops up, right? So it's [11:55] this continuous learning that has to [11:57] happen all the time. So you have to [11:59] build systems that can detect the look [12:01] at the patterns from the past and code [12:03] them determine how you react to those [12:05] and also then spot anomalies with the [12:08] highest you know uh rule being that hey [12:10] you must always find a way to bring back [12:12] the training workload the inference [12:14] workload to its reliability and [12:15] availability. So it's very important to [12:17] kind of build some systems that are [12:18] sophisticated enough to spot the [12:20] anomalies that are happening uh as as [12:22] you go to large scale versus also [12:24] learning that even though you may have [12:26] seen a failure mode 6 months ago, it may [12:29] have different implications right down [12:30] the road. [12:31] >> Yeah. So everybody here is tasked with [12:35] an an impossible charge to have to go [12:38] faster, better change on a dime. And [12:43] we're often doing this with at least on [12:45] the the hardware side a development [12:47] cycle that has a fundamental mismatch [12:48] with that of software and product. [12:52] Right? So I'm actually really curious [12:54] from different vantage points maybe even [12:56] on the risk side for all we could talk [12:58] about this in in in your industry um of [13:01] how we negotiate with our partners that [13:04] are on the opposite side on the software [13:06] on the product side when we're talking [13:08] about kind of real world uh hardware and [13:11] infrastructure constraints. [13:13] Um so I work [clears throat and snorts] [13:15] at Aurora. Aurora is a self-driving [13:17] trucking company for PE folks who don't [13:19] know. Um so for us we have we have two [13:23] partners right? One is the truck itself. [13:25] We have to work with the truck [13:26] manufacturers to get us sensors on the [13:29] truck and get a compute on the truck. [13:32] And on the other side, we have to work [13:34] with the same gang of people that all [13:36] the other people work with the Nvidias [13:39] and the Intels and the ARMs and you know [13:41] that class of people and they are also [13:44] on different um schedules and how risk [13:48] averse they may or may not be. Um one of [13:51] the biggest challenges we face is uh we [13:55] have to slowly migrate towards [13:57] uh parts that are approved for [14:02] um use in vehicles and automobiles. So [14:05] that always is a few years behind the [14:08] curve because they need to be approved [14:10] by the various regulatory agencies out [14:13] there. So it's always a challenge where [14:16] we are projecting in the future saying [14:19] hey we will have this compute then so [14:22] for now let's make sure that whatever we [14:24] need to do fits within the compute [14:27] budgets that we have today. Um and then [14:31] hopefully that translates into a cheaper [14:33] compute in the future [14:34] >> or we can pick a cheaper part because at [14:36] the end of the day if we can if you can [14:38] do the same with less that's better for [14:40] everybody. [14:41] >> Yeah. Let me pick up this thread of of [14:44] um efficiency and cost when it comes to [14:46] observability. [14:48] So one of the things that like you know [14:51] thinking about my experience from [14:52] autonomous vehicle and I'm sure [14:53] everybody in the audience has or [14:55] watching has some experience of you know [14:58] that that that first set of nerves that [15:00] you get going oh my god [15:01] >> right and then you kind of stop and [15:03] realize and you put your engineering hat [15:04] on and go there's there's so there's [15:07] such an incredible influx of data of [15:09] sensors that are running every single [15:11] millisecond that are feeding it and and [15:14] allowing us to at least have that uh [15:16] that that reassurance and that [15:18] confidence in its safety and I'm sort of [15:20] seeing infrastructures as well grapple [15:23] with how much additional data we need [15:27] coming into uh our observability systems [15:30] to really have a full handle on what's [15:32] going on. But of course that's that's a [15:35] that's a cost decision that is an [15:37] engineering decision that's a it's a [15:39] total resources decision. So I'm I'm [15:40] curious about how we're thinking about [15:42] that across space. [15:45] Well, thanks good thank goodness we have [15:47] agents, right? [15:48] >> Yeah. [laughter] [15:49] >> Um I guess uh in in a lot of the the I [15:53] guess the the frontier labs and we're [15:55] training the the latest models, right? [15:57] It's like um when you think of costs [15:59] definitely by far the compute costs [16:01] dominate and then after that you're [16:03] looking at usually oftentimes like [16:05] storage costs are right right behind it [16:07] and then after that you might have some [16:08] networking costs and then finally [16:10] somewhere down there you have some [16:11] observability. So I guess in the term in [16:14] terms of general scale what we had done [16:16] um when training models is just like log [16:19] it all right because at the end of the [16:21] day um anything that you can get that [16:24] can help you debug a potential run [16:26] failure uh ends up being worth it if you [16:28] can shorten that debug cycle time and [16:31] get your large scale training back up [16:32] and running. And so I think actually a [16:35] lot of um uh what's happened this last [16:38] couple years with the AI transformation [16:40] is that it's really flipped a lot of our [16:42] like cost and efficiency models on its [16:45] head [16:46] >> right where we used to say like how can [16:48] I save like 2% 5% how can I like you [16:51] know save a megawatt or two on on this [16:54] power budget right now you're saying [16:56] well I need to roll several gigawatts of [17:00] power in the next 12 months so maybe I [17:02] need to focus my energy on how to best [17:04] harness that power that's coming that [17:06] all that compute that's coming online [17:08] versus trying to focus on saving like [17:10] that last little bit on my old systems, [17:12] right? And so you're kind of like seeing [17:14] this massive acceleration and like [17:16] rethink of of what really matters. What [17:18] are the highest order bits that come in [17:20] when you start to make your decision-m [17:22] process? Um I mean on the topic of speed [17:26] in general, it's like what choice do we [17:29] have? [laughter] [17:30] >> Yeah. you know, I mean, at this point, [17:31] it's it's uh not competing in the AI [17:35] space. It's existential risk for most of [17:38] the companies that we're part of, right? [17:40] I mean, you have to get in there, [17:42] compete, use whatever you can, and that [17:44] basically today means [17:46] >> AI agents to their fullest extent in [17:48] every single way you can to try to get [17:50] there. Yeah. So if if the existential [17:53] risk is losing the competitive edge uh [17:57] because of a slower velocity right [17:58] that's that's one risk to call out [18:01] all in your space failure is not an [18:03] option you cannot have two hours of [18:06] downtime a year in your space [18:10] >> no [18:10] >> how do we think about risks and then I'm [18:13] curious too Jenny this would also go [18:15] into your space how do we how do we even [18:18] like think about SLOs's when there's [18:21] like complexities and dependencies are [18:24] several layers deep. What does that [18:26] actually look like and what does a [18:27] customer guarantee look like, especially [18:29] when you've got multi-million dollar [18:31] contracts on the line or people's lives? [18:35] >> Um, I'll go first. Um, at Aurora, we [18:37] take a very serious and formal approach [18:40] to safety. In fact, I have rarely heard [18:44] the term reliability at the company. [18:46] It's always safety is the term we use [18:49] because uh because even when something [18:53] fails, we need to have a plan of how do [18:57] we stop this 80,000 pound truck [19:00] going at 65 mph to a safe stop, right? [19:04] Even when something crashes, when [19:06] something fails, uh you know, a cable [19:08] breaks, a stone hits a sensor, what do [19:10] you do? A tire blows, what do you do? [19:12] So, we model all of these things out. Um [19:16] so primarily what we do is what we call [19:18] the Aurora safety case framework which [19:21] is a very formal list of claims we make [19:24] about what shall the truck do in various [19:27] situations. And then on the flip side, [19:31] we build up our test cases and map them [19:34] to specific pieces of evidence that hey, [19:37] these four tests put together provide [19:39] evidence that we can observe traffic [19:41] lights early enough that we can react to [19:43] them. And imagine thousands of such [19:46] simple examples put together. And then [19:48] we map each one to each of the hundreds [19:51] of safety case claims that we have made. [19:54] And when we close all of these, that's [19:56] when we say we have closed the safety [19:58] case. And this combination of physical [20:01] hardware software and the sensor packet [20:04] and the route are ready to be used in [20:07] public. [20:08] >> So we are not fast. I wish we were [20:10] faster. We take our time because 99.9% [20:16] uptime means what I you guys will know [20:18] probably a few hours a year of the you [20:22] know trucks crashing that's not [20:24] [laughter] okay. So we we learned a lot [20:27] from the the medical the nuclear power [20:29] the airline industry and we use those [20:32] methodologies [20:33] >> to to prove to ourselves that we are [20:36] safe. [20:37] >> Yeah. Now it's a bad day if a product [20:40] goes down. [20:41] >> Yeah. [20:41] >> But it's not you know it's not an [20:44] existential threat. So I'm I'm curious [20:46] to know Jenny like what how how do you [20:48] give guarantees to the customer in the [20:50] age of AI on the product lens? Yeah, I [20:52] think all leaders have a responsibility [20:55] of an accountability for a business. And [20:58] in in the SR space, our comparison is [21:01] simple. If let's say agents didn't exist [21:05] and there's an issue, how long does it [21:06] take to resolve it, [21:08] >> right? You your your job is and if it's [21:10] 2 a.m. in the night, call the right [21:12] people, hope they wake up, come and [21:14] attend the call and help you get to the [21:16] root cause. [21:17] >> And so far in our experience, agents [21:19] have only made this faster. um in the SR [21:22] space. Now where risk gets introduced in [21:24] this space is when you're coding with [21:26] agents and your gates are not good [21:29] enough [21:29] >> that you got bugs getting into the [21:31] system which ruined everything right [21:33] which causes more errors than in the [21:35] past [21:36] >> and that's where now there is a renewed [21:39] interest in building better gates [21:42] because in the past if a release took [21:44] happened once in two months your errors [21:46] were also once in two months like spread [21:48] out right but now if releases are [21:50] happening every week your error rate can [21:52] go up even with existing bugs. So this [21:54] is now where agents are being applied to [21:57] shore up our defense system, I would [21:59] call it, but it's an incremental [22:01] journey. We don't know what next year [22:02] will look like, but this year we are in [22:04] shoring up the defense. [22:05] >> Yeah. Yeah. [22:06] >> That's great. And Mark, I know we've [22:07] talked about, you know, how do we [22:10] prevent cascading failures uh as I'm [22:12] sure everyone has. So kind of curious to [22:14] know your your take on this. You've seen [22:16] this through through multiple vantage [22:17] points. And then I I want to wrap up [22:20] because time is flying. I want to wrap [22:21] up with with with sod and Dave kind of [22:24] asking about the interplay between [22:26] compute infra and networking obviously [22:29] getting so intertwined where like for [22:31] for the whole machinery to work we need [22:33] all of these things to be working [22:34] together so curious about kind of the [22:35] the interplay that you've seen there but [22:37] over to you Mark [22:39] >> I mean I think for all of us probably [22:42] right I mean the the these systems have [22:44] been built over decades now and so most [22:47] of the time that we see major failures [22:50] they are almost always black swan [22:52] events, cascade failures, three things [22:54] have gone wrong and things and things [22:56] like of that nature. Um I think that for [23:00] us right something that is now uh um [23:04] kind of like a one-way door that we've [23:06] already gone through is we everybody's [23:08] using AI agents to accelerate the [23:10] development and the deployment of all [23:11] these systems. So that's like you know [23:13] an irreversible change in the industry, [23:15] right? So now it's what can we do to try [23:18] to um use that same technology. I know I [23:21] sound very AI pil here. Use more agents [23:24] to you know quickly detect and find [23:27] those failure uh failures as they happen [23:30] to prevent them from cascading into each [23:32] other. And actually a lot of that I [23:34] think and in my experience um comes [23:36] almost from uh best practices from [23:39] regular engineering management. How do [23:41] you set the right metrics? How do you [23:43] set the right check metrics? how do you [23:44] set the right, you know, like uh kind of [23:47] um uh tests and and other like exit [23:49] criteria and things. And if you use [23:52] those to build your prompts for your [23:54] agents, they can then basically operate [23:57] and build code that encodes all of those [24:00] requirements into the way they operate [24:01] and then they can basically check each [24:03] of the systems as they go and try to [24:05] prevent that cascading failure. That's [24:07] probably the best defense we're going to [24:08] have in the in the short term here until [24:10] we truly achieve uh ASI. We all get to [24:14] take a vacation. [24:17] >> Uh Dave and Sad, everyone's talking [24:19] about GPU compute. [24:20] >> Yep. [24:21] >> One of the big constraints obviously in [24:22] large scale training is it can be the [24:24] network, right? Or is often the network. [24:27] >> Well, of course the network is the most [24:29] important part of the system. [24:31] >> I'm not going to disagree with you. I [24:32] got a bunch of friends in. [24:34] >> Well, yeah. And I mean, networking often [24:36] thinks of itself as sitting towards the [24:38] bottom of the infrastructure pyramid. [24:40] And um I would say it is true that a lot [24:44] of progress in AI is essentially a data [24:46] motion problem. And so we're able to [24:48] gang together all of these various [24:50] co-processors, move huge amounts of data [24:52] through the network. And so the network [24:54] is on path for many of these [24:55] computations. The network still has the [24:57] same problems with um uh uh ability to [25:02] introspect on is it working well or not [25:04] that we've always had with the internet, [25:06] right? two endpoints are trying to [25:08] communicate and they're like, well, if [25:09] the data didn't get through, it must be [25:11] the network's fault. Um, and it's really [25:13] hard to collect data of all the paths on [25:16] the network because it has to operate at [25:17] line speed. Um, so those fundamental [25:20] challenges are still with us. Um, [25:23] >> you know, we of course are trying to use [25:24] AI to help both in our AI infrastructure [25:27] clusters to build that uh [25:29] infrastructure. So, networking for AI. [25:32] Um it comes in all flavors just like [25:34] we've done network resiliency and [25:36] reliability for many years. We build [25:38] broad layers of network switches so we [25:40] can tolerate failures. We try and have [25:42] very simple health metrics so that the [25:44] network can itself um adapt to failures [25:47] and automatically correct itself as [25:49] necessary. We write transport protocols [25:52] on top of that. So take advantage of all [25:54] the paths that are in the network and [25:56] all the paths that are working because [25:58] again as we build these massive parallel [26:00] networks, there's always some path or [26:02] some switch that's not working as well [26:04] as the others. And so we're figuring out [26:06] how the network layer can offer an SLO [26:08] to the layers above it that you give us [26:11] messages, we will get them through even [26:13] if we have to tear them apart and send [26:15] them on different paths. Yeah. [26:16] >> Um and then ultimately we're also using [26:20] uh AI for doing networking better and [26:22] like we're talking about using uh AI [26:25] agents to help us uh assesses the [26:27] network healthy or not troubleshoot [26:29] localized failures. Even things like [26:31] buildout operations where we have to [26:33] install literally millions of cables. We [26:36] have data center technicians on the [26:38] floor interacting with AI agents because [26:41] that data center technician has a very [26:43] good view of hey I'm got a wire going [26:44] from here to here whereas previously [26:47] we'd have a network engineer who had the [26:49] big picture of how well the network [26:52] itself was functioning. [26:53] >> Now we can have AI agents uh replace [26:56] that human engineer. So we can have [26:58] people on the floor talking with an AI [27:01] agent and that way use the human smart [27:04] hands guided now by an AI agent [27:07] providing the overall view of how the [27:08] network's doing in terms of overall [27:10] health. [27:10] >> That's great. [27:11] >> Yeah, I'm I'm anxiously awaiting a data [27:13] center tour at some point but that would [27:15] be great. Sad bring us home. [27:16] >> Yeah. So I think uh some couple of [27:18] things you asked earlier too about hey [27:20] the speed of innovation you know with [27:22] their new accelerators coming out every [27:24] year and you know everybody's jumping on [27:27] the latest one we're in a supply [27:29] constraint environment capex constraint [27:30] environment power constraint environment [27:32] so there's always a desired hey I should [27:34] go to the latest and greatest cuz I want [27:36] to grab it and I want to make use of it [27:38] and so they they kind of put these [27:40] constraints time to market everything [27:42] but yet at the same time the bar for [27:44] reliability is never going to go down. [27:46] Every customer expects an SLO and [27:49] expects every cloud service provider to [27:51] meet that SLO. And some of the [27:53] challenges Dave was talking about, you [27:54] know, if if we used to have a constraint [27:56] from a networking point of view that you [27:57] had to be able to talk from one endpoint [27:59] to another endpoint. Now you're talking [28:00] about a cluster. You're talking about, [28:02] you know, potentially hundreds of [28:03] thousands of GPUs or accelerators [28:05] talking to one another over variety of [28:07] networks. And when you hand it over to a [28:10] customer, they expect all of them to [28:11] work, which means they need to be wired [28:13] correctly. You need to verify all those [28:14] wires actually work and and as you are [28:17] building those data centers out and as [28:19] you're doing these wiring you have to [28:22] implement checks every point that hey [28:24] can I go run you know a training job [28:26] across these you know 10,000 you know [28:28] accelerators and do they work do they [28:30] finish in the expected uh duration that [28:33] we we've modeled right and as things [28:35] fail figuring out okay what link was [28:37] broken that I need to go send to the [28:39] operations team and fix right so so [28:42] there are variety of is you have to sort [28:44] of build from the very bottom of the [28:46] stack, you know, at the data center [28:48] level, at the at the networking level, [28:50] at the compute level to and at the end [28:52] workload level that you need to be able [28:53] to verify that hey, before we hand this [28:55] capacity over to the customer, we have [28:57] verified that hey this we able to run a [29:00] training job successfully. So this is [29:01] kind of a step one. But then you need to [29:03] allow the customers to a self diagnose [29:06] through telemetry on or device or [29:07] otherwise take some capacity offline and [29:09] go back and see hey if they were [29:12] experiencing any problem are they unable [29:14] to go back and run the same diagnostics [29:16] you had built [29:17] >> determine if the infrastructure has [29:20] regressed because it does or if that's [29:22] something that's application specific [29:24] it's it's you know it's very easy to say [29:25] and Dave's lived his life with this hey [29:27] it's the network right but it's really [29:29] really hard to prove a negative so so [29:31] that's why we kind of a lot of these [29:33] layers to just kind of know if it's an [29:34] anomaly in the network or if you can go [29:36] back to a common baseline and know hey [29:38] this worked therefore the rest of the [29:39] stack should be working as well. [29:41] >> Fantastic. Humbled by all of the [29:43] complexity, humbled by your incredible [29:45] expertise. Thank you for sharing it. [29:47] It's been a fun and sprawling [29:49] conversation. Hopefully, we've given the [29:50] audience uh very good threads to keep [29:52] pulling at uh once we get to happy hour. [29:54] So, thank you. Uh it's here for our [29:56] panelists. Big round of applause. Thank [29:58] you so much. [applause]