CoreWeave, OpenAI, Meta Mandate AI Agents to Manage GPU Fabrics

CoreWeave、OpenAI与Meta要求采用AI智能体管理GPU网络架构 超大规模服务商正针对异构硬件推进横向扩展架构的标准化，因为当庞大的超算网络满负荷运行时，传统的网络容错机制往往会失效。 来自主流AI基础设施提供商的工程师们深入探讨了将数万张GPU连接至单一训练网络架构中的实际运维现状。专家小组讨论了由于高度同步的AI工作负载几乎无法容忍故障，传输设计不得不从脆弱的高性能方案（被比作“法拉利”的RoCE）转向更具弹性的模型（被比作“面包车”）。专家们还分析了核心运维挑战，包括超高速增长环境下的遥测缺失、多轨Clos拓扑的兴起、数据中心间的横向扩展需求，以及通过引入内部AI智能体来实现网络运维自动化。 AI 超级计算机网络需要连接数万至百万级 GPU，由于同步训练对丢包和灰度故障几乎零容忍，业界正致力于在 RoCE/RDMA 的极致性能与类似 TCP 的高容错弹性之间取得平衡。 (0:05) 针对 AI 扩展织网缺乏现成管理与可观测性软件的痛点，基础设施团队正自研遥测系统，并引入 Agentic AI 工具与 ChatOps 界面来加速网络排障并实施自动化安全防护。 (9:00) AI 织网优化需要贯穿交换机 ASIC、网卡固件与拥塞控制的系统级协同设计，Meta 与 OpenAI 等公司正积极联合硬件厂商推动 Multi-Rail Clos（MRC）拓扑与数据包喷洒（Packet Spraying）的行业标准化。 (15:00) 模型体量的快速膨胀正促使训练任务跨越单数据中心边界，在带来更高 RTT 与电力限制的同时，也借助多级 KV Cache 调度和光路交换（OCS）等技术加速了 Scale-Up 与 Scale-Out 架构的融合。 (19:15)

Engineers from CoreWeave, Meta, OpenAI, and Oracle argue that scale-out networking has shifted from a background utility to the primary determinant of AI supercomputer efficiency and return on investment. As cluster sizes grow toward millions of GPUs, network fabrics must handle near-zero fault tolerance, blurred workload boundaries, and multi-datacenter latency constraints. Fragile Transports and the Loss of Fault Tolerance (0:05) Traditional enterprise networks rely on TCP, which Andrew Dickinson of Oracle and Jitendra Padhye of OpenAI describe as forgiving like a 'family minivan.' In contrast, scale-out AI clusters depend on low-latency protocols like RoCE (RDMA over Converged Ethernet), which operate more like a high-performance race car—highly sensitive to silent data corruption, switch reboots, and misconfigurations. Because distributed AI training runs can collapse from minor packet drops, engineers are challenged to co-design transport and application layers to restore resilient, fault-tolerant execution without sacrificing performance. Tooling Gaps, Observability, and Operational Agents Jim Julson of CoreWeave and Joseph Provine of Meta note that commercial off-the-shelf management tooling cannot keep pace with AI scale-out fabrics. Operators face a persistent observability gap where standard network metrics fail to identify gray failures and performance degradations that slow GPU step times before triggering hard crashes. To manage complex environments, CoreWeave connects internal sources of truth, such as NetBox, to agentic AI tools via webhooks, allowing technicians to query configurations via natural language, with plans to automate safely guarded actions like draining and isolating impaired spine switches. Architectural Convergence Across Physical and Logical Boundaries Network architectures are undergoing structural shifts as workload boundaries dissolve. Provine notes that traditional distinctions between front-end and back-end network traffic are breaking down, driven by dynamic inference workloads and multi-tier key-value (KV) cache routing that function as distributed memory fabrics. Simultaneously, Dickinson emphasizes that scaling requires holistic system-level co-design across switch ASICs, NIC firmware, and congestion protocols, as single-component upgrades like faster NICs cannot resolve throughput bottlenecks alone. Multi-Datacenter Scaling and Standardization (19:15) As model sizes continue their aggressive expansion, Padhye predicts distributed training will inevitably stretch beyond single datacenters, confronting power constraints, discrete infrastructure islands, and significantly higher round-trip times (RTTs). Concurrently, scale-up domains are expanding beyond individual racks with emerging interconnects like Optical Circuit Switching (OCS). Managing this growth across heterogeneous hardware vendors requires unifying architectures, such as standardizing Multi-Rail Clos (MRC) topologies and packet spraying based on fundamental workload physics rather than proprietary stacks. The panel emphasizes that overcoming future scale-out bottlenecks requires balancing cutting-edge automation with core engineering fundamentals. While AI agents assist internal operations, mastering foundational queuing theory, multi-vendor collaboration, and unified transport standards remains critical as training fabrics expand across datacenter boundaries.

在AI超算时代，网络已从单纯为计算让路的基础设施转变为决定昂贵GPU集群利用率与投资回报率的核心瓶颈。来自CoreWeave、Meta、OpenAI与Oracle的技术专家指出，由于分布式大模型训练对丢包与延迟波动的容忍度已降至冰点，业界必须摆脱单点加速思维，通过全栈协同设计、多轨拓扑标准化、深层可观测性以及AI驱动的自愈运维，在追求极致吞吐的同时重建网络的容错韧性。 容错容限坍塌：从TCP旅行车到RDMA法拉利 (0:05) Oracle的Andrew Dickinson指出，丢包、拥塞与硬件故障等经典网络问题并未改变，发生根本颠覆的是上层工作负载对故障的容忍度：传统企业级TCP流量能够平滑掩盖网络缺陷，而动辄跨越整个数据中心建筑、紧密同步的AI训练作业却要求近乎零丢包与极高的通信效率。OpenAI的Jitendra Padhye将这种转变生动比喻为从TCP“家庭旅行车”（只要加油便能在恶劣路况中稳定运行）向RoCE/RDMA“法拉利”（速度极致但需要昂贵燃油与完美路况，稍有失误便全盘烧毁）的演进。 Padhye强调，在包含海量节点的物理基础设施中，静默数据损坏（CRC错误）、交换机重启和人工配置失误是不可避免的客观现实。不同于传统公有云以多租户隔离和渐进部署为核心的设计思路，AI训练属于团队自持全栈软硬件的单租户环境，首要目标是“不惜一切代价保证计算步进（keep the job stepping）”。因此，工程团队必须联合设计传输层与应用层，在不牺牲极限性能的前提下赋予网络类似旅行车的容错韧性，防止底层局部微小故障直接导致价值数百万美元的训练作业崩溃。 全系统协同设计与多轨拓扑标准化 (0:15) Dickinson认为，破解AI横向扩展瓶颈无法通过孤立的组件升级来实现，网络从根本上是一项系统级挑战。如果底层的网络控制平面、全局网络拓扑以及集群级拥塞管理机制无法有效协同，单纯为端点服务器配备更高速率的网络接口卡（NIC）并不能带来切实的算力利用率提升。超算网络的设计重心已经从单纯堆砌原始带宽，转向如何使整个交换矩阵在满负荷运转时依然保持极高的有效利用率与极低的长尾延迟。 针对拓扑架构的演进，Meta积极推动了多轨Clos（Multi-Rail Clos, MRC）拓扑与数据包喷洒（packet spraying）技术的跨硬件厂商标准化落地。Padhye指出，尽管在跨厂商推进过程中面临法律、管理与各家专用硬件限制等阻力，但AI工作负载严苛的物理定律与极限性能需求，自然迫使竞争对手与硬件供应商在核心设计原则上形成共识。结合最新一代的高性能网络芯片与交换设备，现代超算得以摆脱早期遗留硬件的束缚，加速了多轨统一标准的建立。 可观测性鸿沟与AI智能体运维落地 (0:09) 超算规模的急剧扩张引发了严重的工具链脱节。CoreWeave的Jim Julson透露，市场上根本不存在能直接适配超大规模AI横向扩展矩阵的现成软件和管理工具，迫使基础设施团队在采购现有工具以解燃眉之急的同时，必须持续自研内部平台。目前CoreWeave面临的最大运营挑战，是如何将包含遥测流、日志、指标与警报在内的可观测性堆栈，扩展至超大规模物理集群的体量。Meta的Joseph Provine对此表示赞同，并指出在耗资不菲的分布式训练中，未被常规指标捕获的“灰色故障”会导致GPU步进时间严重降级甚至作业停顿，带来巨大的算力浪费。 为化解运维瓶颈并打破团队间的信息孤岛，CoreWeave开始在内部引入针对性的AI智能体（Agent）与ChatOps架构。通过Webhook将智能体与NetBox等内部数据源打通，现场运维与算力团队能够直接使用自然语言查询IP分配、子网归属与设备维护状态，以辅助硬件更换操作。Julson披露，CoreWeave的自愈路线图旨在设立严格安全护栏的前提下实现完全自动化的故障隔离，例如在包含572台交换机的高密度网络矩阵中，无需人工介入即可自主识别、排空流量并下线出现受损迹象的脊层交换机（Spine Switch）。 跨数据中心扩展与网络边界消融 (19:15) 面对大模型参数规模的持续爆炸式增长，Padhye指出此前关于训练集群规模将见顶停滞的预测已被打破。由于单体数据中心内的二层或三层网络容量与电力供给正逼近极限，未来的超大规模训练任务将不得不跨越多个地理隔离的数据中心进行扩展。这种多数据中心分布式扩展带来了显著增大的往返时延（RTT）、跨供应商异构基础设施互联难题以及电力分配瓶颈，也加速推动了光路交换（OCS）等动态互联技术的引入。 与此同时，网络内部的传统物理分界正经历全方位消融。Provine指出，多GPU scale-up域（如NVLink）正在打破单个服务器机架的物理限制，向外延伸并与scale-out网络交织；在流量层面，推理任务中多级KV Cache的跨节点传输使其表现得如同分布式共享内存，导致东西向与南北向网络边界、前端服务网络与后端训练网络的界限趋于融合。Julson补充道，由于不同模型架构在多云或跨服务商互联时网络行为迥异，业界不存在通用的“万能方案”，高度依赖定制化工程梳理与跨组织协作。 本场讨论表明，AI横向扩展网络已进入以系统工程协同与物理极限管理为主导的深水区。尽管端到端协同与AI辅助运维已初见成效，但跨数据中心的大时延开销、无损性能与故障弹性之间的权衡，以及缺乏通用现成工具的现实，仍是悬而未决的严峻考验。与会专家在职业建议中一致强调，无论上层生成式AI工具如何提升个人产出效率，排队论、网络底层传输协议及扎实的故障排查基本功，依然是驾驭这一复杂体系不可替代的基石。

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布。根据页面描述，这是一场由 Rajiv Krishnamurthy 主持，汇聚了来自 CoreWeave、Meta、OpenAI 和 Oracle 顶尖工程师的高端圆桌座谈，重点探讨在十万卡至百万卡级 AI 超算集群中构建紧耦合横向扩展（Scale-Out）网络所面临的架构拓扑、拥塞控制及软硬件协同设计等前沿工程挑战。 ## TL;DR 四位身处一线的大型 AI 基础设施架构师深入剖析了 AI 训练对底层网络容错率的极端苛求，指出传统 TCP 虽如皮实的“家用面包车”，但当下面临的 RoCE 则如同娇贵敏感的“法拉利”，亟需通过多路径喷洒与统一传输协议（如 MRC）进行自愈与加固。讨论涵盖了极端规模下“边买现成方案边造自研工具”的工程拉锯、多层嵌套架构导致的“灰色故障”排查痛点，以及借助 ChatOps 智能代理和 Webhook 推进运维自愈的实践。此外，专家们还展望了跨数据中心大 RTT 传输、Scale-Up 与 Scale-Out 域融合及光路交换（OCS）等技术趋势，并为新一代技术人给出了拥抱 Agentic AI、扎根排队论等网络经典理论的职业发展建议。 ## Atmosphere & style 整场座谈洋溢着硬核工程实战派特有的松弛与坦诚，老练的技术专家们一边笑谈自嘲为“首席修包官”或靠“一杯接一杯的苏格兰威士忌”顶住高压，一边毫无保留地揭示万卡集群幕后的惊心动魄。台上既有关于“千万别当经理”与“留在 IC 岗位”的幽默互怼，也有对经典网络底层理论历久弥新的深沉敬畏，交流节奏紧凑欢快且充满一线视角的思想火花。 ## Bulletin - [01:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=75s) 随着 AI 超算集群迈向十万至百万卡 GPU 规模，网络角色已从传统公有云中“不添乱即可”的后台支持，转变为直接决定昂贵 GPU 利用率的核心基础设施。 - [05:27](https://www.youtube.com/watch?v=YRUdzADM2H4&t=327s) 网络底层的硬件故障与丢包并未减少，根本转变在于 AI 同步计算对抖动的容忍度极低，当前架构的核心目标是将娇贵高效的 RoCE“法拉利”改造得如同 TCP“家用面包车”般具备自愈与容错韧性。 - [15:20](https://www.youtube.com/watch?v=YRUdzADM2H4&t=920s) 超大规模 AI 网络的复杂性已演变为全系统工程，业界正通过确立统一的多路径拥塞控制（MRC）技术愿景拉齐芯片厂商，并结合全栈流式遥测来压榨极限吞吐与降低排障耗时。 - [19:26](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1166s) 单数据中心电力与空间物理见顶，正倒逼网络向跨中心、跨云商大 RTT 互连扩展；同时 KV Cache 的跨级调度模糊了东西向与南北向边界，推动 Scale-Up 与 Scale-Out 拓扑走向融合。 ## Key takeaways - [05:27](https://www.youtube.com/watch?v=YRUdzADM2H4&t=327s) AI 训练网络对缺陷的容错率极低，任何细微抖动都会被同步屏障无限放大，网络设计必须在接近 100% 满载推挤流量的同时，向应用层屏蔽底层的物理链路与硬件异常。 - [08:31](https://www.youtube.com/watch?v=YRUdzADM2H4&t=511s) 应对异构硬件算力存在两条路线：一种是构建内部同构的“计算隔离岛”并在边界做流量校准，另一种是推动跨代际、跨芯片厂商在拥塞控制与通信协议行为上实现完全一致。 - [09:38](https://www.youtube.com/watch?v=YRUdzADM2H4&t=578s) 商用现成产品在极端 AI 训练规模下全面失效，基础设施团队不得不陷入“边采购应急边在飞行中换引擎自研”的死循环，可观测性工具链的扩展速度严重滞后于硬件膨胀。 - [11:11](https://www.youtube.com/watch?v=YRUdzADM2H4&t=671s) 深厚的技术分层使网络可见性成为核心瓶颈，难以探测的“灰色故障”（Gray Failures）会静默拖垮全局分布式训练同步，造成巨大的算力闲置与检查点回滚代价。 - [13:01](https://www.youtube.com/watch?v=YRUdzADM2H4&t=781s) 团队正在利用 Agentic AI 重构运维体系，通过 ChatOps 交互查询权威配置库，并在严格安全护栏下自动化隔离异常交换机，实现大规模网络拓扑的免打扰自愈。 - [15:20](https://www.youtube.com/watch?v=YRUdzADM2H4&t=920s) AI 超算网络是涵盖物理走线施工、网卡硬件卸载、流式遥测与拥塞协议的系统级工程，盲目提升单端口带宽无法解决突发流量积压与端到端传输效率问题。 - [17:22](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1042s) 推动竞争芯片厂商在 MRC（多路径拥塞控制）上达成共识的核心，在于先建立无懈可击的顶层技术愿景，并借力新一代交换芯片流片升级的窗口期化解硬件微码硬伤。 - [19:26](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1166s) 两层或三层拓扑在物理规模上逐步见顶，网络架构正向跨数据中心甚至跨供应商互连演进，同时 Scale-Up 域正由单机架向多机架外延，倒逼 Scale-Out 网络结构做出深度重构。 - [21:10](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1270s) KV Cache 在多级缓存中的流动打破了传统的南北向与东西向流量边界，推理与训练系统的边界日趋模糊，要求网络架构兼具极低延迟、超高吞吐与高动态弹性。 - [23:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1395s) 跨基础设施提供商的异构互联不存在通用的标准化模版，大模型在不同物理拓扑与地理跨度下的流量特征迥异，工程团队必须针对具体合作场景开展定制化架构设计。 - [29:00](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1740s) 深度掌握 Agentic AI 工具已成为工程师的基线要求，但排队论等经典计算机网络理论依然是解决前沿 AI 网络瓶颈的根本基石，真实的故障排查（Troubleshooting）是沉淀核心竞争力的关键路径。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=YRUdzADM2H4&t=5s) 小组座谈开场：横向扩展网络与十万卡级 GPU 超算互联 主持人 Rajiv Krishnamurthy 在午餐后组织了本场座谈。他谈到刚刚在餐厅看到大家热烈讨论的场景令人振奋，本次会议的核心目的正是让经验丰富的行业老兵与刚毕业的研究生齐聚一堂、交流切磋。Rajiv 引用了当天早些时候 Omar 在主题演讲中提出的观点：网络存在多个不同维度的挑战——既有机架内部（intra-rack）的互联，也有连接广域数据中心之间的网络。 本场座谈聚焦于核心命题：将海量机架与 GPU 紧密耦合成一个统一集群究竟需要什么？如今行业中频繁出现“10万张 GPU”乃至“百万张 GPU 超级计算机”的宏大叙事，但在网络工程视角下，这到底意味着什么？为了解答这一难题，现场邀请了来自 CoreWeave、Meta、OpenAI 和 Oracle 的四位核心网络架构师与实践者。 - 讨论范畴明确聚焦于机架间与 GPU 之间的大规模横向扩展网络（Scale-Out Fabric），而非单一数据中心边界之外的广域网或单纯的单机架内总线。 - 核心现实背景是行业正在向 100,000 卡至 1,000,000 卡 GPU 的超级计算网络规模迈进。 --- ### [01:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=75s) 专家介绍与前线视角：从“首席修包官”到极高风险下的算力博弈 来自 CoreWeave 的 Jim Julson 率先发言，他幽默地将自己的职位称为“首席修包官”（chief packet polisher）。谈及最令他兴奋的事，Jim 感慨这是一个棘手但令人振奋的问题：当前所面临的挑战是他职业生涯中规模最大的，从人员组织、底层系统到工程流程都需要全面横向扩展。这些前所未有的复杂问题没有任何现成的开箱即用方案，必须亲自迎难而上。他坦言自己在这段经历中意识到自己非常适合这种高压环境，“这是我职业生涯中第一次面对赌注如此之高的局面（the stakes so high）”，而 AI/ML 领域最不缺的就是大难题。 紧接着是来自 Meta 网络基础设施团队的 Joseph Provine，他主要负责网卡（NIC）与传输层协议。Joseph 指出，由于工作负载的演进，网络流量特征一直在发生剧烈变化。如果截取当前的切片来看，现状极为复杂：不仅有传统同步训练带来的突发通信，还有由 KV Cache 调度迁移直接把网络变成“分布式内存织机”（memory fabric）的新需求；此外，智能体（Agents）架构不断兴起，Agent 与 LLM 交互、实时派生新的子 Agent 并调用各种工具，形成复杂的调用闭环。这种计算模式的重构彻底改变了底层流量形态。 OpenAI 的 Jitendra Padhye（Jitu）谈及自己的职责时直截了当地表示：“正如我今天上午在演讲中所说，我在 OpenAI 的工作就是不惜一切代价保证训练任务持续推进（keep the job stepping at all costs）。”他对比了自己此前在微软公有云环境的工作经历：在多租户公有云中，工程师在发布变更时必须慎之又慎，因为大量客户依赖其稳定性；而在单一客户的大规模 AI 训练织机中，极致的性能与运行速度压倒一切。他感慨能同时见证两种模式非常幸运，眼下正是从事网络技术研发“最美好的时代”。 Oracle Cloud Infrastructure（OCI）的软件架构师 Andrew Dickinson 表示，他自 2007 年云网络萌芽之初便投身于此，目前兼顾虚拟网络团队、全球物理网络控制平面以及头部 AI 核心客户的战略合作。Andrew 观察到，AI 正在颠覆网络在计算架构中的传统定位：“传统上，网络的作用仅仅是配合计算，大家希望网络躲在幕后别添乱（make the network not the problem）”；但如今在 AI 负载中，网络已成为不可或缺的底层基石，昂贵 GPU 能否高效运转，完全取决于网络能否即时、可靠地交付流量。 ```mermaid flowchart TD accTitle: AI 工作负载驱动的网络演进 accDescr: 展示从传统多租户云网络到现代 AI 超算网络的特征与角色变迁 subgraph Traditional["传统公有云网络架构"] T1["工作负载: 多租户通用计算 / Web 服务"] --> T2["网络角色: 辅助支持，尽量隐藏自身故障"] T2 --> T3["协议特征: 依赖 TCP 掩盖丢包，高容忍度"] end subgraph ModernAI["现代超算 AI 训练织机"] M1["工作负载: 同步训练 / KV Cache 迁移 / 智能体闭环"] --> M2["网络角色: 决定昂贵 GPU 利用率的核心织机"] M2 --> M3["协议特征: 追求近 100% 满载，极低容忍度"] end Traditional -.->|"算力密度激增 & 角色重塑"| ModernAI ``` --- ### [05:27](https://www.youtube.com/watch?v=YRUdzADM2H4&t=327s) 网络老问题与极低容错率：“家用面包车”与“法拉利”的隐喻 在探讨当前网络的核心瓶颈时，针对客户视角（OpenAI、Meta）与服务提供商视角（OCI、CoreWeave）的分歧，Andrew Dickinson 强调：网络面临的基本问题其实从未改变，依然是丢包、拥塞、利用率不均以及硬件故障。真正发生根本性转变的是**应用对网络缺陷的容忍度（tolerance requirements）**。传统工作负载在不完美的网络中往往运行得相当平稳，TCP 协议能极好地掩盖链路抖动和丢包之痛；然而当整栋建筑的 GPU 构成一台紧密耦合的超大型同步计算机时，任何细微的抖动都会被无限放大。现在的核心挑战在于，行业必须研发出全新方案，在几乎 100% 满载极限推挤网络的同时避免拥塞和丢包，而这是传统网络协议从未做到过的。 Jitu 对此表示高度赞同，并生动地重温了微软早期推广 RoCE（RDMA over Converged Ethernet）时的一张经典幻灯片对比： > “TCP 就像一辆家用面包车（family minivan），只要你给它加满油，它就能一直晃晃悠悠开下去；而 RoCE 则是昂贵娇贵的法拉利（Ferrari）——它能跑出极限超高速度，但你必须喂它高标号燃料，驾驶员必须具备极高技巧，而且路面最好一尘不染、绝无杂质，因为一旦发生一次撞车，整台车就会立刻被烈火吞噬。” Jitu 阐述道，当前架构演进的目标正是让这台“法拉利”具备“家用面包车”般的皮实与韧性。网络底层设备的本质不会奇迹般变好：交换机路由器不会突然变得绝对可靠，链路 CRC 校验错误不会凭空消失，硬件老化、软件缺陷、配置发布失误以及人为操作事故都是不可避免的客观事实。在 AI 训练环境中，唯一的幸运之处在于它是单任务/单应用环境，工程团队与硬件厂商合作拥有“从上至下完整的技术栈所有权”。因此，团队可以在网络各层进行深度定制与软硬件协同设计，把底层织机在所难免的物理故障对上层训练的影响降到最低，全力保障昂贵的 GPU 集群不会因为网络抖动而闲置停摆。 - **传统 TCP**：如同家用面包车，容错能力强，能够自适应遮蔽网络底层的抖动与丢包。 - **早期 RoCE**：如同赛道法拉利，追求极限吞吐与极低延迟，但对丢包极度敏感，遇到网络轻微拥塞或硬件故障极易引发连锁崩溃。 - **当前设计目标**：在维持极高吞吐与满载能力的前提下，向应用层屏蔽底层的 CRC 校验错误、交换机故障与链路异常。 --- ### [08:31](https://www.youtube.com/watch?v=YRUdzADM2H4&t=511s) 异构环境的应对路线：隔离计算岛与统一传输协议 针对主持人提出的跨硬件厂商异构算力环境挑战，Jitu 提出了两条截然不同的落地路径： 第一种路径是**构建“计算隔离岛”（Islands）**。每个计算岛内部保持高度的同构性（例如全量采用同一种特定型号的加速卡与配套网络架构），使内部通信保持极致效率；随后在岛与岛之间的通信边界上进行精细校准与流量控制，借此规避底层硬件物理特性的不一致。 第二种路径则是**真正的全异构环境（True Heterogeneity）**。如果同一个复杂系统中混部了来自不同供应商的加速器与网卡硬件，网络层面就必须依赖**标准化的通信协议**。上层 AI 应用的逻辑是统一的，但如果要让不同厂商的硬件协同工作，各方不仅要在字节级别保持兼容（byte compatible），更必须保证各种网络拥塞控制与传输优化策略在不同代际、不同品牌的硬件上具备完全一致的行为表现。唯有解决跨代际与跨厂商的协议一致性，超算集群才能健康持续地扩展。座谈在此引出 CoreWeave 等新型基础设施提供商在面临爆发式规模增长时，所承受的底层系统扩张重压。 ### [09:38](https://www.youtube.com/watch?v=YRUdzADM2H4&t=578s) “边买边造”的死循环与工具链规模脱节 Jim Julson 顺着前面的讨论指出，在当前超级计算机这种极端规模下，想在市场上找到现成可用（off-the-shelf）的软硬件产品根本是不切实际的。一旦意识到这一点，团队就必须亲自动手去构建满足需求的自研工具；然而现实的矛盾在于，业务问题在当下这一秒就已经爆发了。因此团队陷入了一种永无止境的循环——“一边买现成产品顶上，一边在飞行的飞机上换引擎去自研构建（buy the thing to do the thing, but be building the thing while you're doing the thing）”。在整个技术栈中，买还是造（build vs. buy）是一场永恒拉锯的双人舞。 以可观测性（Observability）为例，这个术语涵盖了流式遥测（Streaming Telemetry）、日志、指标、报警以及事件处理等极广的范畴。整个生态虽然有五六款核心产品支撑，并且具备一定的通用性与可编程性，允许工程师围绕它们编写代码并扩展功能，但即使做到这一步，面对 AI 集群的吞吐与规模依然显得捉襟见肘。目前摆在团队面前最紧迫的头号任务，就是让工具链的承载能力尽快追赶上底层基础设施的扩展步伐。 ```mermaid flowchart LR accTitle: Build vs Buy Dilemma in AI Infrastructure accDescr: 展现规模化扩张中采购方案与自研工具之间的迭代循环机制 subgraph Problem["现实矛盾"] A["业务规模激增"] --> B["现成产品无法满足"] B --> C["立即出现系统故障"] end subgraph Cycle["应对循环"] C --> D["采购应急方案顶住"] D --> E["边运行边自研替代"] E --> F["自研工具追赶底层规模"] end F -.->|规模进一步放大| A ``` - **核心矛盾**：商用现成产品（COTS）在极端 AI 训练网络规模下直接失效。 - **可观测性现状**：依赖 5 到 6 款主流核心监控与遥测底座，但即使自研二次开发依然无法完全消化海量事件。 - **核心目标**：全力补齐工具栈短盘，使其吞吐与解析能力匹配超大规模基础设施。 --- ### [11:11](https://www.youtube.com/watch?v=YRUdzADM2H4&t=671s) 多层架构下的能见度瓶颈与“灰色故障”之痛 作为基础设施的使用方与客户，Joseph Provine 探讨了供应商与服务商带来的价值与挑战。服务商的多样化生态与高度灵活性，确实帮助团队以极快的速度完成业务上线与容量扩张。然而，真正的噩梦依然落在了可见性（Visibility）上。当系统输入产生了一个异常的不良反应时，究竟是底层物理网络、交换机固件、集合通信库还是调度器出了问题？面对层层嵌套的复杂架构，想要快速定位根本原因极其困难。 在 AI 训练场景中，这种黑盒问题的代价被呈指数级放大。如果因为网络故障导致正在运行的大规模模型训练任务崩溃，团队就不得不从上一个检查点（Checkpoint）重新启动，损失数以万计的 GPU 卡时；或者因为网络性能隐性劣化，导致最终用户根本无法享受到千卡、万卡集群应有的线性加速收益。可见性原本就是网络领域的百年老题，但在当前庞大且紧耦合的 AI 工作负载挤压下，这个伤口被彻底撕裂。台上有人随即语带讽刺地自嘲打趣道：“我们的训练团队可‘太喜欢’那些我们根本探测不到的灰色故障（Gray Failures）了！” - **供应商优势**：敏捷接入多种异构系统与软硬件服务，保障计算容量快速铺开。 - **排障成本剧增**：系统分层过于深厚，无法快速厘清性能衰减或扰动的源头归属。 - **灰色故障（Gray Failures）**：部分硬件或链路性能隐性下降而未彻底宕机，静默拖垮全局分布式训练同步。 --- ### [13:01](https://www.youtube.com/watch?v=YRUdzADM2H4&t=781s) ChatOps 与智能代理驱动的运维自愈演进 当主持人询问是否有团队已经落地 AI 技术来简化日常网络运营时，Jim Julson 坦言，网络工程团队的核心工作并非整天围着写大语言模型代码打转。但在公司长期高速扩张、每月新入职数百人的背景下，内部形成了严重的“信息孤岛”——许多人对某一个狭窄的垂直子领域了如指掌，跨领域的知识传播与共享却极为闭塞。为了打破这一壁垒，网络团队将所有上下游角色均视作“客户”，涵盖内部的计算团队、存储团队、数据中心现场工程师（DCT），以及最上层的终端用户。 为了给内部团队赋能并减少人工打扰，他们构建了基于智能代理（Agentic Workloads）的 ChatOps 运维机器人。现场技术人员或数据中心团队无需反复提工单询问网络拓扑配置，只需在聊天界面中输入带有简单参数的基础指令，Agent 就会通过 Webhook 自动向真实信息源（如 NetBox 等 Source of Truth 数据库）检索站点信息、网络地址或排查设备维护状态。展望未来，团队正在为这些代理加装严格的安全边界与安全护栏（Guardrails），计划逐步放行基础变更与自愈操作：例如在由 572 台 Spine 核心交换机组成的巨大 Fabric 中，即便某一台 Spine 出现物理异常被系统自动拉出集群隔离，“我们也没必要大惊小怪，这根本不该需要人工介入”。 ```mermaid flowchart TD accTitle: ChatOps and Agentic Operations Architecture accDescr: 展示内部团队如何通过智能代理访问网络真实数据源并执行自愈维护 subgraph Users["内部用户角色"] U1["数据中心技术员 (DCT)"] U2["计算/存储系统团队"] end subgraph Automation["智能运维中间层"] AG["Agentic ChatOps 智能代理"] GR["安全防护与围栏 (Guardrails)"] end subgraph Backend["真实信息源与硬件设施"] NB["NetBox 拓扑配置库"] HW["572 台 Spine 架构 Fabric"] end U1 -->|自然语言/轻量指令| AG U2 -->|自然语言/轻量指令| AG AG -->|Webhook 校验查询| NB AG --> GR GR -->|自动化隔离/维保出流| HW ``` - **信息传递壁垒**：业务规模月均增员数百人，垂直领域专家分工过细，运维信息急需自助化通道。 - **已落地场景**：基于 Webhook 与 Agent 的交互式 ChatOps，自主检索设备位置、地址段及是否处于维护模式（Maintenance Mode）。 - **自动化演进路线**：引入护栏机制，允许系统在成百上千台 Spine 交换机发生单点链路故障时，自动下线故障单板，实现现场免打扰自愈。 --- ### [15:20](https://www.youtube.com/watch?v=YRUdzADM2H4&t=920s) 解决 AI 网络困境的系统级全局视角 当讨论转向具体解决方案时，Andrew Dickinson 指出，超算级网络面临的绝非单纯的硬件缺陷或单独的软件故障，这本质上是一个庞大的“全系统工程问题（Systems Problem）”。AI 集合通信工作负载的物理特征极其严苛，仅仅盲目换上一块单口速率更高、物理带宽更强的网卡毫无用处；如果你无法保证突发流量以极高效率灌入网卡、穿过 Fabric 拓扑并到达对端，带宽再大也只是空转。 Andrew Dickinson 强调，必须将硬件卸载加速、高效拥塞协议与全栈监控结合起来。诸如 RDMA、RoCE（RDMA over Converged Ethernet）以及多路径拥塞控制方案（MRC, Multi-path/Multi-route Congestion control）等技术，正是克服以往网络吞吐天花板、充分压榨链路利用率的核心抓手。与此同时，强健的流式遥测数据流搭配细致的标准作业程序（SOP）同样至关重要，只有两者联动才能最大程度缩短故障恢复时间（MTTR）并精准排序排障优先级。整个解决方案的演进必须打破软硬件与网络设计的边界，一路下沉渗透到数据中心现场的走线施工、布线安装、日常驻场巡检与全局生命周期维护。 | 核心维度 | 关键考量与落地技术 | 目标收益 | |:--- |:--- |:--- | | **数据传输协议** | RDMA、RoCE、数据包喷洒（Packet Spraying）/ MRC | 突破网络利用率瓶颈，消弭突发流量积压与丢包 | | **可观测性与运维** | 高频流式遥测（Streaming Telemetry）+ 结构化 SOP | 快速感知隐性劣质链路，智能规划抢修与容灾优先级 | | **系统级全流程协同** | 涵盖物理层线缆安装、交换机固件、调度层与现场工程 | 构建端到端高可用体系，杜绝局部单点性能塌陷 | --- ### [17:22](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1042s) 推动行业跨厂商协同：统一技术愿景的力量 针对上午会议中提到的“MRC 方案已成功拉齐业界几乎所有主流硬件厂商并实现落地”这一成果，主持人追问其背后的协调细节——面对不同芯片厂商各异的硬件限制、私有架构与能力参差，究竟是如何将各方拉到一张谈判桌上的？发言嘉宾（Jitendra Padhye）透露，破局的核心在于“先拿出一个无懈可击且足够坚实的顶层技术愿景（Solid Technical Vision）”。在极大规模 AI 网络中，采用类似数据包喷洒（Packet Spraying）或 MRC 这类动态多路径拥塞控制方案早已成为行业共识与必然之选。 只要整体方向具有充分的说服力，各家厂商自然会有强烈的参与动力。在这个过程中，虽然免不了遭遇典型的组织行政阻力与法务合同拉锯，但由于底层共享同一个明确的技术愿景，所有参与者最终都围绕着几项最核心的物理与网络原则收敛了方案。至于不同硬件之间的性能天花板与指令集差异，团队在工程落地中也找到了一系列妥协与变通方案（Workarounds）；加之本次推进恰好赶上了最新一代交换网络硬件的流片迭代窗口期，使得硬件层面的阻碍相对以往小了许多，最终促成了全行业标准规范的落地。 - **协同核心动力**：清晰且不可逆的顶层技术愿景，驱动竞争厂商在多路径喷洒与拥塞控制上走向收敛。 - **跨越行政摩擦**：通过成立开放联合规范，化解跨企业的法务条款拉扯与管理沟通障碍。 - **代际硬件红利**：依托最新一代网络芯片的升级契机，规避了旧架构存在的底层微码与流水线硬件硬伤。 ### [19:26](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1166s) 跨数据中心扩展与 Scale-Up / Scale-Out 网络的交织 随着前沿大模型的训练任务不断膨胀，网络架构面临物理极限的严织考验。Jitendra Padhye 指出，业界曾一度认为可能不再需要进行超大规模的训练任务，但现实推翻了这一假设——训练规模目前依然处于强劲的上升曲线上。随着两层或三层拓扑在物理规模上逐渐见顶，单个数据中心内部在电力、空间与冷却等维度的约束显现，迫使网络架构必须跨越单体数据中心的物理边界。一旦走向多数据中心互连，网络就必须应对更宽广的往返时延（RTT）以及更遥远的地理跨度。未来的必然演进路径甚至需要将由不同基础设施提供商所维护的“计算孤岛”（islands）互连起来。 而在纯后端网络内部，纵向扩展（Scale-Up）与横向扩展（Scale-Out）网络的界限正在加速靠拢与融合。Scale-Up 域的物理边界持续扩张——例如英伟达（NVIDIA）最初出货的紧耦合域通常局限在一个机架内，未来势必向更广的多机架甚至更深维度蔓延。这种演进要求工程师在设计 Scale-Out 网络时，必须将扩大的 Scale-Up 域特性深度纳入考量。此外，像光路交换（OCS, Optical Circuit Switching）等新兴技术也将在未来的互连体系中占据关键一席之地。 ```mermaid flowchart TD accTitle: AI训练网络演进挑战示意图 accDescr: 展示从单数据中心物理限制到跨中心、多供应商互连以及内部网络融合的演进结构。 subgraph Limitations["物理规模瓶颈"] P1["双层 / 三层网络拓扑见顶"] P2["单数据中心电力与空间约束"] end subgraph Expansion["跨地域与多方扩展"] E1["跨数据中心大 RTT 传输"] E2["跨不同云商/服务商孤岛互连"] end subgraph FabricConvergence["后端互连形态演进"] F1["Scale-Up 域持续外延<br>(单机架 -> 多机架)"] F2["Scale-Out 围绕大域重构"] F3["新型技术引入<br>(如光路交换 OCS)"] end Limitations --> Expansion Expansion --> FabricConvergence ``` - **训练任务规模持续攀升**：大模型训练并没有放缓，持续处于算力需求曲线上行期。 - **跨数据中心与跨云商互连**：当单机房无法提供充沛电力与容量时，必须向远距离互连与跨供应商集群协同寻找出路。 - **Scale-Up 域外溢**：单机架紧耦合网络正向更宏大的域演变，迫使 Scale-Out 拓扑做出协同架构变革。 --- ### [21:10](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1270s) 前后端流量融合与灵活的高性能网络架构 主持人 Rajiv Krishnamurthy 针对 Joseph Provine 此前提及的 KV Cache（键值缓存）切入，探讨前端网络（Front-end Network）与后端网络（Back-end Network）是否会逐步走向收敛。Joseph 肯定地指出，当下网络流量特征已经不再是传统意义上非黑即白、界限分明的“东西向”或“南北向”，两者的流量模式正在深度交织融合。面对这种变革，网络无论是自研还是与外部供应商合作，都必须依托具备高度弹性和极低延迟的高性能系统来化解挑战。 随着 KV Cache 在多级缓存（Multi-tier Cache）架构中的流动，推理与训练系统的边界愈发模糊。Joseph 提到现场发布的各类最新动态（如网络软硬件加速方案），强调必须通过加速硬件与灵活架构来满足极具动态性的工作负载需求。过去云计算时代沉淀下来的经典经验，正与当下 AI 工作负载的具体诉求深度融合；要在多变的设计空间中立于不败之地，就必须拥有既能快速演进又能兼顾极致吞吐与超低时延的互连方案。 --- ### [23:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1395s) 跨服务商异构互连：“一杯威士忌接着一杯”的现实应对 面对多供应商、多服务商共存的宏观异构现实——例如 Meta 的专属基础设施与 CoreWeave 的基础设施互连，或是把 Oracle 及其他公有云与专业 AI 云无缝打通——Jim Julson 风趣地调侃其应对策略就是“一杯接一杯地喝苏格兰威士忌”（"One glass of scotch at a time."）。在现场的笑声中，Jim 强调当前网络行业最迷人、最独特的特点，恰恰就在于“没有任何一套方案可以包打天下”（no one-size-fits-all）。与 Meta 合作的打法，跟与 OpenAI 合作的技术路径可能截然不同。 尽管传统网络工程与系统架构的核心原则一直是提炼通用模型、制定标准化框架以实现复用（"rinse and repeat"），但由于大语言模型在跨园区、跨区域乃至跨地理大区（Geo）的基础设施上表现出的网络流量行为差异巨大，团队必须针对每一个具体的合作形态与场景开展定制化考量。这一过程看似回归到了最质朴的“需求收集、全局审视、针对性解决”闭环，但也让每次工程实践都充满了解未知问题的兴奋感。Rajiv 感慨归根结底这一切还是取决于“人”，Jim 斩钉截铁地附和道：“永远取决于人，一直都是这样。” --- ### [25:21](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1521s) 职业建议：用 AI 杠杆武装自己，以及“千万别当经理”的现场交锋 在小组讨论的收尾环节，嘉宾们面向现场的学生和从业人员分享了职业发展的真知灼见。Andrew Dickinson 首先给出了一个具体的实操建议：充分利用像 ChatGPT 这样的工具作为个人能力放大器。新入门者完全可以建立一个名为“Networking 101”的项目，指令 AI 充当严师，为自己量身定制一套从零成为世界级网络专家的系统化学习路径，涵盖权威书籍、前沿博客与实操练习。比起老一代工程师只能翻旧书、四处托人打听知识的年代，AI 已经成为极具颠覆性的自学加速器，但与此同时，也绝不能忽视与真人交流与建立连接的重要性。 紧接着发言的嘉宾对此深表认同，并坦言自己前一天刚用 ChatGPT 彻底攻克了 BGP EVPN 相关的困惑——由于 Jim Julson 经常出难题，他必须借助 AI 迅速摸透其运行机制。针对业内盛行的“AI 正在抢走工程师饭碗”的焦虑，他坚定地驳斥了这一论调，认为 AI 实际上极大地加深了技术人员的能力边界，以往需要多人耗费数小时的大型项目，如今个人凭借 AI 辅助即可独立完成。最后，他话锋一转，幽默地给出现场最具冲击力的一条建议：“尽量不要去当经理（Manager），留在 IC（Individual Contributor，独立贡献者）岗位上你能学到多得多的东西！管理 AI Agent 可比管人轻松多了！” 这一“开火”瞬间引爆全场，Rajiv 笑着惊呼“有人开枪了！”（"Shots fired."）。作为当场管理岗代表的 Jim Julson 则笑着反击说“咱们 Slack 上见”，并随即正色道，作为一名兼顾深度一线技术的技术型经理（Technical Manager），他不仅没有远离实操，反而更能从大局引导方向。在呼应 Andrew 建议的同时，Jim 进一步提出更深层的主张：仅仅使用 AI 是远远不够的，从业者必须找到自己真正热爱并具有辨识度的细分领域，将个人热情与 AI 的赋能深度融合，打造出与众不同的独特优势。 - **ChatGPT 101 启动法**：建立专属提示词工程，让大语言模型为你定制阶段性书单、博客推荐与动手实验。 - **技术突破真实案例**：现场工程专家现身说法，面对高难度的 BGP EVPN 等前沿协议，直接借助 AI 快速完成知识盲区攻坚。 - **IC 与技术经理之争**：IC 路线能最大程度沉淀纯粹技术复利且无需背负人事协调负担；而技术型经理同样能保持硬核的技术敏锐度。 ### [29:00](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1740s) 拥抱 Agentic AI 与跳出思维定势 对于刚踏入职场的新人而言，跳出既定框架进行思考（thinking outside the box）将变得前所未有地关键。有嘉宾提到自己在团队管理中的硬性标准：“我实际上要求我所有的工程师都必须使用 Agentic AI。这不是一个可选项，而是一个硬性要求（hard requirement）。”在当下 Claude Code、Gemini、Codex 等 AI 编程助手已经能替人类完成大量常规编码与繁琐事务的时代，工程师的核心竞争力不再是机械性的代码编写，而是能否深度借助并拥抱这些工具，并在群体中脱颖而出，证明自己不可替代的独特价值。这种能够将前沿 AI 工具与创新发散思维相结合的能力，是当前技术人最为紧迫且关键的素质。 ### [29:38](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1778s) 扎实计算机网络基本功与排错能力的价值 主持人随后邀请 Joseph Provine 为本场圆桌讨论做收尾发言。Joseph 强调，在追逐新技术浪潮的同时，最关键的一点依然是“深耕你的基本功（lean into your fundamentals）”。他打趣道，自己在网络领域摸爬滚打多年、头发都已经花白，正因如此他深切体会到经典理论的恒久生命力——哪怕是现在，随手拿出一本关于排队论（queuing theory）的教科书，里面的核心概念与数学模型依然可以直接映射并解决当今超大规模 AI 集群网络面临的现实挑战。对于刚毕业加入职场的新人来说，在学校里学到的基础理论绝非无用功，即使面临的具体工程问题在变，其底层解空间（solution space）与基本法则始终未曾改变。 除了理论基础，Joseph 还特别叮嘱新人们在实际工作中一定要沉下心来学习排错（troubleshooting）。在他看来，技术生涯中所有真正豁然开朗的“顿悟时刻（aha moments）”，几乎全是在深度排错的过程中诞生的；哪怕最后排查出来发现只是自己犯的一个愚蠢低级错误，也同样是一次极具价值的学习经历。席间有人幽默地插话补充“即便你现在已经是管理者也是如此”，引发全场会心一笑。最后，主持人 Rajiv Krishnamurthy 进行了简短总结，向所有分享干货的嘉宾致谢，并表示希望台下听众能同台上的工程师们一样，在思想交流与新知探索中收获乐趣，从而为本场关于 AI 超级计算机 Scale-Out 网络的圆桌论坛画上了圆满句号。 - **必须掌握 Agentic AI**：团队要求全员深度使用 Agentic AI 工具，将其作为基线能力，核心竞争力转向在此基础上的创新思维与差异化表现。 - **经典理论历久弥新**：如排队论等扎实的网络基础概念依然在解决当今最前沿的 AI 网络拥塞与性能瓶颈中发挥核心作用。 - **故障排查即核心资产**：真正的技术成长与顿悟来源于真实场景中的 Troubleshooting，即使是低级错误也是不可替代的经验积累。 ## Notable quotes > [01:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=75s) the stakes so high — Jim Julson > [01:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=75s) keep the job stepping at all costs — Jitendra Padhye > [01:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=75s) make the network not the problem — Andrew Dickinson > [05:27](https://www.youtube.com/watch?v=YRUdzADM2H4&t=327s) TCP 就像一辆家用面包车（family minivan），只要你给它加满油，它就能一直晃晃悠悠开下去；而 RoCE 则是昂贵娇贵的法拉利（Ferrari）——它能跑出极限超高速度，但你必须喂它高标号燃料，驾驶员必须具备极高技巧，而且路面最好一尘不染、绝无杂质，因为一旦发生一次撞车，整台车就会立刻被烈火吞噬。 — Jitendra Padhye > [09:38](https://www.youtube.com/watch?v=YRUdzADM2H4&t=578s) buy the thing to do the thing, but be building the thing while you're doing the thing — Jim Julson > [11:11](https://www.youtube.com/watch?v=YRUdzADM2H4&t=671s) 我们的训练团队可‘太喜欢’那些我们根本探测不到的灰色故障（Gray Failures）了！ > [23:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1395s) One glass of scotch at a time. — Jim Julson > [23:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1395s) no one-size-fits-all — Jim Julson > [23:15](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1395s) 永远取决于人，一直都是这样。 — Jim Julson > [25:21](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1521s) 尽量不要去当经理（Manager），留在 IC（Individual Contributor，独立贡献者）岗位上你能学到多得多的东西！管理 AI Agent 可比管人轻松多了！ > [25:21](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1521s) Shots fired. — Rajiv Krishnamurthy > [29:00](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1740s) 我实际上要求我所有的工程师都必须使用 Agentic AI。这不是一个可选项，而是一个硬性要求（hard requirement）。 > [29:38](https://www.youtube.com/watch?v=YRUdzADM2H4&t=1778s) lean into your fundamentals — Joseph Provine ## Mentioned Rajiv Krishnamurthy, Jim Julson, Joseph Provine, Jitendra Padhye, Andrew Dickinson, CoreWeave, Meta, OpenAI, Oracle, Oracle Cloud Infrastructure (OCI), Microsoft, NVIDIA, TCP, RoCE, RDMA, KV Cache, NetBox, MRC, Packet Spraying, BGP EVPN, OCS, ChatGPT, Claude Code, Gemini, Codex, Slack ## Video description Live Panel with Coreweave, Meta, OpenAI, Oracle Moderated by Rajiv Krishnamurthy Speakers: Joseph Provine, Andrew Dickinson, Jim Julson, Jitendra Padhye Training frontier AI models requires connecting tens of thousands of accelerators into a single, tightly coupled fabric — pushing scale-out network design into uncharted territory. This panel brings together engineers building some of the world's largest AI training networks to discuss the architectures, transport protocols, and operational lessons behind today's supercomputer-scale fabrics. Topics will include topology choices, congestion control at extreme scale, and the co-design between network infrastructure and distributed training frameworks. Expect a practitioner-level conversation on what it takes to keep massive AI clusters connected, fast, and reliable. Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=YRUdzADM2H4)

[1:15] Jim Julson: the stakes so high [1:15] Jitendra Padhye: keep the job stepping at all costs [1:15] Andrew Dickinson: make the network not the problem [5:27] Jitendra Padhye: TCP 就像一辆家用面包车（family minivan），只要你给它加满油，它就能一直晃晃悠悠开下去；而 RoCE 则是昂贵娇贵的法拉利（Ferrari）——它能跑出极限超高速度，但你必须喂它高标号燃料，驾驶员必须具备极高技巧，而且路面最好一尘不染、绝无杂质，因为一旦发生一次撞车，整台车就会立刻被烈火吞噬。 [9:38] Jim Julson: buy the thing to do the thing, but be building the thing while you're doing the thing [11:11] 我们的训练团队可‘太喜欢’那些我们根本探测不到的灰色故障（Gray Failures）了！ [23:15] Jim Julson: One glass of scotch at a time. [23:15] Jim Julson: no one-size-fits-all [23:15] Jim Julson: 永远取决于人，一直都是这样。 [25:21] 尽量不要去当经理（Manager），留在 IC（Individual Contributor，独立贡献者）岗位上你能学到多得多的东西！管理 AI Agent 可比管人轻松多了！ [25:21] Rajiv Krishnamurthy: Shots fired. [29:00] 我实际上要求我所有的工程师都必须使用 Agentic AI。这不是一个可选项，而是一个硬性要求（hard requirement）。 [29:38] Joseph Provine: lean into your fundamentals

[0:05] Jim Julson: chief packet polisher [0:05] Jitendra Padhye: whatever it takes to keep the job stepping at all costs [0:05] Andrew Dickinson: get the network out of the way [5:00] Jitendra Padhye: As long as you put gas in it, it keeps running. [5:00] Jitendra Padhye: one crash and the whole thing is going to go up in flames [25:20] Jitendra Padhye: managing AI agents is far easier than managing people

[0:05] Okay, I hope everybody's having fun. Um, [0:07] it's great to see all of you folks and [0:10] we were just having uh lunch at the U [0:14] dining hall and you know it was really [0:16] great to see you know all the [0:18] conversations that folks were having and [0:20] the whole point of u this conference is [0:23] to basically get all of us together [0:26] share our experiences and learn from [0:27] each other. So it's really heartening to [0:29] see you know these kinds of [0:30] conversations we have fresh grad [0:32] students as well as experienced [0:33] professionals and um the you know it's [0:37] an amazing experience I hope all of you [0:39] are enjoying it this panel is [0:42] essentially for us to understand a [0:43] little bit more about what's happening [0:45] in scale out networking and as Omar [0:48] alluded to in his keynote earlier in the [0:50] morning you know there are different [0:52] dimensions for networking there's the [0:55] networking that happens within the rack [0:57] then there's the networking that happens [0:58] to connect all of our wide data centers. [1:00] Our focus in this topic and this panel [1:03] is going to be around what does it take [1:04] to connect all of these racks and GPUs [1:07] together to build these numbers. You [1:08] hear these numbers, right? 100,000 GPUs, [1:10] a million uh GPU supercomput, what does [1:13] it mean from a networking perspective? [1:15] Now, to get us started, we have a pretty [1:18] distinguished panel. I'm really excited [1:20] to hear from all of them. And we'll [1:22] start with a round of intros because we [1:24] have a couple of uh folks whom you've [1:26] heard from before and some new folks [1:28] here. So we'll start with Jim. Can you [1:29] just introduce yourself again to the [1:31] crowd? What do you do and what do you [1:32] find most exciting about networking? [1:35] >> Yeah. So Jim Jolson um chief packet [1:38] polisher uh Coree. Um, so like I guess [1:42] like [1:43] dude that's a hard question, but I think [1:45] like the things that are exciting me [1:47] right now are really about solving some [1:51] of the biggest problems that I've ever [1:53] personally faced, right? Like how do we [1:55] scale out uh everything in from the [1:58] people to the systems and the process [1:59] kind of like my talk talked about [2:01] earlier, right? Um these are complex [2:03] problems and there there's no [2:05] out-of-the-box solution for it. So being [2:07] able to to really stand and be present [2:10] and say like, "Okay, like I've got to [2:12] solve this. Nobody's going to solve this [2:13] for me." It's a pretty cool feeling. [2:15] It's it's the first time for me in my [2:17] career where I've ever had the stakes so [2:18] high. And I and I've kind of found [2:20] myself in this journey and realized that [2:23] like I thrive in in that world and uh AI [2:26] and ML definitely is not short of big [2:29] problems to solve. So [2:31] >> awesome. Excited that you're having so [2:33] much fun. Uh Joseph. [2:35] >> Yeah. Uh I'm Joseph Provin. Uh I'm part [2:37] of the network infrastructure team [2:39] supporting the Nix and uh transport at [2:41] Meta. Um I think we heard a lot this [2:45] morning about you know how traffic [2:47] characteristics keep continuously [2:49] evolving and changing because of [2:50] workloads. So you know we you know we've [2:54] been uh working with many of these and [2:56] if you take a snapshot I think it's [2:58] always been changing a lot. If you take [3:00] a snapshot now, you know, you have [3:02] training synchronized, you have now the [3:05] whole KV cache movement literally [3:07] causing the network to be like a memory [3:10] fabric and uh you also have the agents [3:12] now you know agents are now talking to [3:15] LMS spawning other agents talking to [3:18] tools and then it's a loop. So obviously [3:22] the traffic characteristics and [3:24] workloads are changing and a lot of [3:26] interesting opportunities and problems [3:28] there. Awesome. Ju uh [3:31] >> I'm Jupiti. Um as I said in my talk this [3:33] morning, I do whatever it takes at [3:35] OpenAI to keep the job stepping uh at [3:37] all costs. Uh before that I had a very [3:40] different job as I was at Microsoft and [3:42] it's a public cloud environment. Um so [3:44] the contrast is very stark uh in [3:46] Microsoft and you have to be very [3:49] careful very mindful about how you [3:51] deploy changes because you have many [3:52] customers who depend on you. On the [3:54] other hand, the training fabric is a [3:56] single customer environment where [3:58] performance and speed are paramount. So [4:01] very different challenges and I've been [4:02] fortunate to kind of see a little bit on [4:04] the both. Uh and this is this is this is [4:08] one of the best times to be doing [4:10] networking honestly. [4:12] >> Awesome. Andrew. [4:13] >> Yeah. I'm Andrew Dickson. I'm a software [4:15] architect at Oracle Cloud [4:17] Infrastructure. I work within our [4:19] networking organization. Um I I work [4:22] across both our our our cloud virtual [4:24] networking teams as well as our uh teams [4:26] for software control planes that manage [4:27] our network infrastructure globally. Um [4:29] and I also work on some strategic [4:31] engagements with some of our core uh AI [4:33] networking customers. Um I I've been [4:36] doing cloud networking since 2007. Uh [4:39] basically when it started uh it's it's I [4:41] would agree with you too like it it is a [4:43] a interesting time in networking. uh you [4:46] know compared to to where it's been I I [4:48] think in particular you know what what [4:50] I'm observing is is AI is is changing [4:54] the role of networking in compute uh you [4:57] know I say traditionally the network was [5:01] there to support the compute you know [5:02] kind of get the network out of the way [5:04] make make the network not the problem [5:06] now you know that the network is really [5:08] fundamental to the AI workload uh in a [5:12] way that that the efficiency and [5:14] effectiveness of the expensive GPU use [5:16] is driven by the ability of that network [5:17] to deliver traffic successfully. So it [5:19] really is changing you know how we use [5:21] the network and and and uh and and what [5:23] the network means in in modern you know [5:26] computing. [5:27] >> Awesome. So you know it's it's really [5:29] interesting to have um all of you guys [5:31] here because in some ways if you think [5:33] about it um you know we have Jitu and [5:36] Joseph you are sort of the customers so [5:38] to speak even though Meta builds its own [5:40] data centers and you know Andrew and Jim [5:42] you guys are sort of the service [5:44] providers you're building out of these [5:45] fabrics you're serving a lot of [5:46] customers so maybe I'll start with you [5:48] um Andrew if you can just give us a [5:50] sense of what do you think are the [5:51] critical problems from your perspective [5:53] like what do you see is top of mind [5:57] the the problems in networking aren't [6:00] new, right? Um these are uh you know [6:03] packet loss, congestion, uh uneven [6:06] utilization, uh hardware failures. The [6:10] the thing that I the challenge that I [6:13] see that's new right now is the the [6:14] tolerance uh requirements uh have [6:17] changed, right? So uh the you know [6:20] traditional workloads [6:22] uh you know they run surprisingly well. [6:25] they're fairly tolerant to imperfect [6:27] networks. You know, TCP does a great job [6:29] of of kind of masking some of the pain [6:31] uh and hiding that uh and and these AI [6:34] workloads uh with you know synchronized [6:37] compute running across you know [6:38] effectively a giant supercomputer within [6:40] a building uh that the tolerance is much [6:43] lower and so there's a lot of challenges [6:46] that need to be addressed in terms of uh [6:48] you know those same problems that we've [6:50] always had we need different solutions [6:51] for. you know loss and congestion and [6:53] uneven utilization. We really need the [6:55] ability to drive the network at you know [6:57] nearly 100% in ways that we've never [6:59] been able to do with traditional [7:00] protocols. [7:02] >> Awesome. Does that resonate as a [7:04] customer jun? [7:05] >> Yeah. Um I think uh Andrew said it [7:07] right. The problems are the same. It is [7:10] the requirements of the applications [7:11] that have changed. Um so back again um [7:14] when we did Rocky at Microsoft we used [7:16] to have this slide where TCP was [7:19] depicted as a family minivan. As long as [7:21] you put gas in it, it keeps running. [7:23] >> And Rocky was the Ferrari. Okay, [7:25] >> you need it will go very fast, but you [7:28] have to put high price gas in it. You [7:29] have to know how to drive it. And the [7:31] roads better be spectacularly clean [7:33] because one crash and the whole thing is [7:34] going to go up in flames, [7:35] >> right? And so the goal, as you saw in [7:38] MRC presentation this morning, is to go [7:40] from that back to a little bit of the [7:41] family mini kind. So [7:44] because the networks are going to remain [7:46] what they are, all of a sudden the [7:48] routers are not getting any more [7:49] reliable. All of a sudden the CIC errors [7:51] are not going to stop. Hardware failures [7:53] are going to remain what they are. [7:54] Software problems, deployment problems, [7:56] human errors, they're all going to be [7:58] there. And the job now is to make sure [8:00] that your application is not affected by [8:02] the luxury we have is that it's a single [8:04] application environment. So you own the [8:06] stack in some sense top to bottom [8:08] including the hardware along with your [8:10] vendors, right? And therefore you can [8:12] innovate so that the impact of those [8:17] you know essentially the failures that [8:18] are always going to be there in the [8:19] fabric inevitable failures that's less [8:22] on your application because GPU is such [8:23] a precious commodity you have to keep [8:25] the training job running so that's how [8:27] you structure your works that's how you [8:29] structure your thinking interesting how [8:32] how do you think about heterogenous uh [8:34] environments like what would your ask be [8:38] as a person I guess you don't care where [8:40] the compute comes Right. [8:41] >> So heterogeneous environments you can [8:43] think about this in a number of [8:44] different ways. I was talking to [8:45] somebody at lunch and you can [8:48] potentially one way to structure this is [8:49] islands. Each island is homogeneous and [8:52] then you carefully calibrate the [8:54] communication between the islands. [8:56] That's one way to think about it. The [8:58] other way is true heterogeneity. Suppose [9:00] you had you know accelerators from [9:02] multiple vendors or what have you. And [9:04] then that's where you need um as we saw [9:07] you need standardized protocols from a [9:09] networking perspective right because the [9:11] application is going to remain the same [9:13] but if your hardware is coming from [9:14] different vendors then you obviously [9:16] need standardized protocols you need [9:18] some sort of mechanisms to make sure [9:20] that they're not just bite compatible [9:22] but all the optimizations you do are you [9:25] know working the same way across [9:27] multiple generations of hardware. Uh and [9:29] that would be the way to grow. Makes [9:31] sense. Jim Core is like growing gang [9:34] busters. So I'm sure you know with lot [9:36] of growth comes a lot of challenges. So [9:38] what what do you see uh from your lens? [9:42] >> Yeah. I mean I I feel like I can just go [9:44] what he said [9:45] like um yeah I mean like this kind of [9:50] goes back to you know I I had made a [9:52] comment earlier that that's like trying [9:54] to find things like offtheshelf [9:56] um to do what you need it to do at the [9:59] scale that we need to do it just isn't a [10:02] thing. And so like you know that's great [10:05] you identify that now you have to go and [10:07] build the thing that you need it to do. [10:09] And the problem is is you have a problem [10:10] now. [10:11] >> And so we find ourselves in this [10:13] constant cycle of like buy the thing to [10:15] do the thing, but be building the thing [10:17] while you're doing the thing. [10:19] >> Uh and this this happens in in you know [10:22] all parts of our stack. Like it's a [10:24] constant song and dance back and forth [10:25] about like build verse buy, right? Um [10:28] that piece bleeds into like you look at [10:30] like observability is is a very broad [10:32] term but that's your telemetry streaming [10:34] telemetry your logging your metrics in [10:36] general all your alertings tied to it [10:38] your eventing um that ecosystem consists [10:42] of you know roughly five or six key [10:45] products that I I won't name um but five [10:48] or six key products and all of which are [10:50] are um fungeible in the sense that we [10:53] can you know write our own code up [10:55] against them and and build upon them, [10:57] but sometimes it's still not even enough [10:59] there. And so I think that uh some of [11:01] the things that are front and center for [11:03] us right now are getting the tooling [11:05] stack caught up to the scale of the [11:07] infrastructure. Like that's probably the [11:09] the number one thing on our plate right [11:11] now. [11:11] >> Interesting. So Joseph, as a customer, I [11:17] guess for you, what do you think are [11:19] some of the good things you're getting [11:20] from service providers and vendors that [11:23] gives you the optionality? [11:26] So I mean definitely the um the [11:28] flexibility and the uh you know the [11:31] diverse nature of um you know all the [11:35] systems and uh services that are [11:37] available [11:38] uh enables you know us to get to a [11:41] service very fast. Um so definitely that [11:44] is something that you know that is there [11:46] the quick expansion is always there in [11:48] terms of how we can uh quickly get our [11:51] uh our customers you know enabling our [11:53] customers um the challenges I think is [11:56] obviously um you know I'm going to just [11:59] key on what Jim mentioned here as well [12:01] as in terms of visibility um so you know [12:04] it's it's across the board you know yeah [12:06] there is observability but you know what [12:08] is responsible for you know an adverse [12:11] effect of my input I think getting there [12:14] very fast is very hard because there's [12:15] so many layers and and also within us [12:18] you know we have to figure out as to you [12:19] know what uh what actually went wrong [12:22] because it hurts a lot more especially [12:25] uh you know if you have to restart a [12:26] training job or you know if your you [12:28] know end users are not able to get the [12:30] benefit of the you know the scalable uh [12:33] nature of you know how you're actually [12:36] enabling the capacity that you need so I [12:39] think that's that's probably I think it [12:41] it's been around for it's a problem [12:42] that's been a constant in networking [12:44] that you know more visibility um but [12:47] it's just ex exacerbated by the you know [12:49] actual workload themselves right now [12:51] >> all of our training teams love it when [12:53] there's gray failures that we can't [12:54] detect right [12:57] >> great so I think observability [12:59] operations seems to be a big theme [13:01] >> uh we did we do have a track later on [13:04] which talks about how AI can help NI but [13:07] anybody uh who wants to share some [13:09] insights around how you've been able to [13:11] use actually AI to make things easier [13:16] in terms of operations. Anybody wants to [13:18] take a stab maybe? Yeah. [13:19] >> Yeah. [13:19] >> Jimmy nodding. So, we'll go with you. [13:21] >> Yeah. I I mean like I'll just caveat [13:23] that like we're as a network [13:25] organization, right? Our bread and [13:26] butter is not like riding against LLMs. [13:29] Um, we have some very very smart [13:31] developers and I think some of the [13:33] things that like you take for granted is [13:36] when you're in like onboarding hundreds [13:39] of people a month as a company over the [13:42] course of many years, it creates pockets [13:44] of people that know a lot about a [13:46] little, right? Uh because they have this [13:48] domain uh that they're responsible for, [13:51] but it also creates a a problem in [13:53] getting it's a dissemination of [13:56] information. And so from a network [13:58] standpoint, right, we want to empower [14:00] like we look at everybody that consumes [14:02] our network as customers. First it's [14:04] internal, it's upstack, it's the compute [14:06] teams, it's the storage teams. Um, and [14:08] then it's the customers riding on top of [14:10] the stack. So we like to provide tooling [14:12] for our internal customers to be able to [14:15] selfserve some of their problems, right? [14:17] Uh whether it's capacity and the data [14:19] center teams needing to understand, hey, [14:21] what's inside of your netbox, right? [14:24] what what what is this site and what's [14:26] its address we got to ship infra to it [14:28] and um or whether it's the DCTs looking [14:31] to understand if a device is in [14:32] maintenance mode so that they know that [14:34] they can go service something that the [14:36] automation actually created a ticket [14:37] for. So we've created some agents some [14:39] agentic uh workloads that allow like [14:42] chat op style work where it's just [14:44] simple web hooks that they can go query [14:45] sources of truth passing uh simple [14:48] commands with basic parameters to to [14:50] self-s serve that information. um kind [14:52] of next steps to that is with some [14:55] serious training wheels or guardrails if [14:57] you will uh allowing actual like move ad [15:00] change for basic operations i.e. there's [15:02] no reason that I should care that a [15:04] single spine out of 572 [15:06] of them is pulled out of operation if [15:09] it's warranted. And so we want to be [15:11] able to, you know, have use cases where [15:13] um data center technicians and engineers [15:16] can go self-service some of that stuff [15:17] uh down the road. And and we're [15:18] constantly iterating on things like [15:20] that. [15:20] >> Very interesting. So you know, we've [15:22] talked about different challenges that [15:24] all of us are facing. Maybe we should [15:25] talk about solutions. Spend the the next [15:27] 10 minutes just thinking about how the [15:30] solutions space evolves. So Andrew, from [15:33] your perspective, like what would you [15:34] think are the two or three, you know, [15:37] top level items from your perspective [15:38] that you think would make life easier [15:40] for you? [15:41] >> Yeah, this is um it's fundamentally a [15:44] systems problem. Um, you know, it's not [15:47] a not a specific hardware problem or [15:48] specific software problem. It's really [15:50] the the system at large put together uh [15:52] due due to the kind of the nature of [15:53] these workloads. So, you know, while [15:55] while we need faster networking, we also [15:59] need to, you know, make sure that we're [16:01] actually utilizing it and using it [16:02] effectively and efficiently. So, just [16:04] just having a faster nick does me no [16:06] good if I am not able to get traffic [16:08] into and through a network. Um so you [16:11] know technologies like uh RDMA uh Rocky [16:15] uh MRC uh you know I I think are really [16:18] helping quite a bit to to allow us to [16:21] kind of overcome uh uh you know some of [16:24] the historical challenges that we've had [16:26] um uh and uh you know make better use of [16:31] it. I think to to Jim's point, the [16:32] telemetry and observability is another [16:35] uh key area where uh there's there's a [16:39] lot of opportunity to [16:41] uh uh you know kind of minimize downtime [16:45] and and you know kind of maximize the [16:48] the the the recovery effort and [16:50] prioritize recovery effort as well. um [16:53] uh so you know having a strong stream of [16:55] telemetry and and the corresponding SOPs [16:57] and things like that to go with that I [16:58] think is a key area where uh where that [17:01] solution evolves. So you know it it [17:03] really becomes this whole ecosystem of [17:05] not just the hardware and not just the [17:07] software not just the the network [17:09] engineering or design but also the [17:11] people on the floor that are installing [17:12] it that are managing it that are [17:14] maintaining it and and really the whole [17:16] top to bottom. So it it is a [17:17] comprehensive solution and there's [17:19] there's it's a opportunity for [17:20] improvement throughout all of it. [17:22] >> Makes sense. Ju specifically uh before [17:25] you talk about what you think are top [17:27] solutions. You know in the morning you [17:29] did mention that you got MRC implemented [17:31] with pretty much lot of the vendors in [17:33] the industry. How was that process? Like [17:35] how did you bring everybody to the table [17:37] and how were you able to get you know [17:40] different hardware implementations, [17:41] different limitations, different [17:42] capabilities? you know every solution [17:45] has its strengths and weaknesses. So [17:47] before how did you approach it like what [17:50] was the u big insight um that you got [17:53] out of that? [17:54] >> I think uh the the general idea was you [17:57] know if you have a very solid technical [18:00] vision people will obviously want to [18:02] contribute to it. People will obviously [18:03] want to join the vision. So everybody [18:05] understands that um you got to have [18:08] something like package spraying matter [18:10] of it might be or MRC whatever those [18:12] solutions were obvious right and so once [18:15] you propose this and once you're willing [18:16] to engage in a dialogue to kind of [18:18] figure out a common spec and people work [18:20] on that um it was not always easy [18:23] obviously there are uh typical [18:25] administrative and legal hurdles uh [18:27] those were overcome u but because there [18:30] was a shared technical vision and [18:32] everybody essentially wanted to do more [18:35] or less the same thing. Everybody came [18:37] to the problem from different viewpoints [18:38] but the solution as you saw repeatedly [18:42] essentially coaleses around a few basic [18:44] principles. [18:45] >> So as long as you have that it's easy to [18:47] go from that point onwards right um and [18:50] yeah different hardwarees have different [18:51] uh limitations and different [18:53] capabilities. So sometimes you find [18:55] workarounds uh and we were kind of [18:58] fortunate that we were working with the [18:59] latest generation of hardware. Um so [19:01] that problem was a little bit less [19:03] intense than what I have experienced [19:05] before. Um so that also helped but you [19:08] know to some extent you find problem [19:10] solutions as you go and the shared [19:13] vision is what everybody works towards. [19:15] >> So if we zoom out right because this was [19:17] like a specific solution looking at [19:19] specific piece of hardware [19:21] >> as you are looking at the architecture [19:23] for nextg networks maybe longer [19:24] distances connecting all of these data [19:26] centers. How do you think about that [19:28] solution space? Like [19:31] >> I think technology will continue to [19:34] evolve and the requirements will [19:35] continue to evolve, right? Um the jobs [19:37] are only getting bigger. Uh there was a [19:40] thinking as you guys know that we [19:42] probably don't need to do very large [19:43] training as this no longer true. We [19:45] still on the upswing uh of that curve. [19:49] So one thing that is bound to happen is [19:53] um we may hit the sort of limits on [19:57] where we can go with say two tier or [19:58] three tier networks at which point you [20:00] have to consider the possibility that [20:02] everything all the compute you need is [20:04] not going to fit in one data center. Now [20:07] you have to go between data centers. [20:08] Once you go between data centers you're [20:10] working with potentially much larger [20:12] RDS. You might not have all the power [20:14] you need in one place. Right? uh then [20:16] you have to potentially consider going [20:18] over much longer distances than what we [20:20] have gone so far and that's when you [20:23] have to start thinking next step is [20:25] again obvious um may have to think about [20:27] potentially connecting different islands [20:31] maintained by different providers and [20:32] how do you go there so all of those [20:34] things will come into play and that's [20:35] what those are the challenges that I [20:37] foresee on the purely backend side I [20:40] think [20:42] the scale up and scale out networking [20:44] will continue to come close weather [20:46] that's also a trend that we know that's [20:48] going to happen. the scaleup network [20:49] will continue to grow. uh you know the [20:52] domain the scaleup domain currently what [20:54] Nvidia ships is one rack potentially [20:56] expand that so that will have to be [20:58] factored in how you design the scale out [21:00] network around that there are of course [21:02] newer technologies coming around uh that [21:04] people always talk about like optical [21:06] circuit switching those will also play a [21:08] role eventually [21:10] >> very interesting so Joseph you mentioned [21:12] KV cache so you know I'm going to take [21:14] the bait and uh how do you think about [21:16] this convergence theme do you think u [21:19] something's going to happen between the [21:21] front end and the backend networks. Do [21:22] they remain separate? Um or you feel [21:25] that there are some shared [21:26] characteristics even if they remain [21:28] separate but how do you see this thing [21:30] evolving? [21:31] >> Yeah, there there is going to be uh a [21:34] lot of uh shared characteristics. I [21:35] think you know it's it's not just about [21:38] east, west, north, south anymore. Uh the [21:40] traffic characteristics are starting to [21:43] you know mingle. We heard that in the [21:44] keynote as well and you know the [21:47] colleagues have shared the same thing. [21:48] So it is going to be there and so we [21:50] have to leverage solutions you know [21:52] across um I would say like you know even [21:55] for our vendor partners and ourselves we [21:58] we run into similar challenges or [22:01] different challenges in this environment [22:04] and the key part is to start [22:06] accelerating the solutions. Um you know [22:09] I just want to uh and and we have to be [22:12] flexible you know uh we saw Maraki [22:15] announced today MRC is there. So we are [22:17] trying to solve a problem for our [22:20] workloads in the fabric and for that we [22:22] need to be able to have a very [22:23] performant system performant solution [22:26] low latency [22:28] uh you know KD cache because you brought [22:30] it up you know it can move to multiple [22:32] tiers now there are many frameworks [22:34] there but you know how do we enable that [22:36] and I think Jim mentioned that you know [22:38] go buy off the shelf for you know what [22:40] we do but then if you want to fit that [22:43] into a part of our solution um it has to [22:46] Yeah. So I think it's it's blending in [22:50] terms of you know how um you know we [22:52] have solved the problems let's say in in [22:54] cloud environment to what is happening [22:56] in the AI workload and where things are [22:58] merging together. Um so there is [23:00] leverage on on both sides but I think we [23:03] need to have faster accelerated pieces [23:07] of hardware and very flexible to adapt [23:09] to new and changing um solution space [23:12] that you know we are getting into. Very [23:15] interesting. So Jim, maybe let me ask [23:18] you a different tack on heterogenous. So [23:21] we did here, you know, we're building [23:22] these data centers. They're going to be [23:23] heterogenous equipment, different kinds [23:25] of vendors that are being there uh [23:27] placed together and get all of these [23:28] things to interoperate. But let's talk [23:30] about different kinds of service [23:31] providers, right? Like a meta may come [23:32] and say we have our own data center. [23:34] We're going to connect it to a core data [23:36] center or you may go and take let's [23:38] connect all these different from Oracle [23:40] or all of these guys together. So how do [23:42] you approach it as a provider because [23:45] you need to connect now to all of these [23:47] it's also happening at the macro level [23:49] >> hetrogenity. So how do you think about [23:51] it? [23:52] >> One glass of scotch at a time. Um, no. I [23:56] like I think that's one of the [23:59] interesting things about this particular [24:01] time in networking and this particular [24:03] industry is, you know, there is no [24:05] oneizefits-all, right? The the way that [24:08] we would operate with meta or is [24:10] different from how we would operate with [24:12] open AI. Um, and it's really cool [24:14] because it does leave room for you to [24:16] kind of like think outside the box. It's [24:19] like you always try to model a framework [24:20] as much as you can so that you can rinse [24:22] and repeat and use prior art and and [24:24] that's that's great. That's that's [24:25] architecture 101, design 101, but in [24:29] this world because the language models [24:31] themselves can vary so wildly how they [24:33] behave on a given network across a [24:35] campus across a swath of sites inside of [24:39] a a super region or a geo for that [24:42] matter. It it does mean that you really [24:44] have to look at each individual [24:46] relationship as its own unique thing. uh [24:48] at least that's been our experience and [24:50] and I think in doing so it in itself has [24:53] provided us a blueprint with how to [24:55] approach them and that's um you know [24:57] it's it's the same old requiring or uh [25:00] requirement gathering sorry um and [25:03] looking at the whole picture and then [25:06] solutioning it. Uh it's it's it's it's a [25:09] pretty exciting thing to do um just [25:11] because they are different and it means [25:13] that you're solving different problems [25:14] each time. So [25:15] >> interesting. So it comes down to people. [25:17] >> Comes always comes back to people. [25:19] Always. [25:20] >> That's great. [25:20] >> Yep. [25:21] >> So you know it's great. We sort of went [25:23] through what you guys found exciting. We [25:25] talked about some of the challenges. We [25:27] talked about the solution space. So we [25:29] have you know again we have a lot of uh [25:32] folks who are exploring the networking [25:34] industry. A lot of students in the [25:36] audience. We also have a lot of [25:37] experienced professionals uh mid-career [25:40] professionals just excited about this [25:42] space. So maybe we could try to wrap it [25:44] up with final thoughts and just some [25:46] advice that you would have to folks in [25:48] the room. We can start with you Andrew. [25:50] >> Sure. Yeah. I mean I I'm thinking your [25:52] question earlier about use of AI uh you [25:55] know I think especially for early career [25:57] professionals you know go into Chad GPT [26:00] create a new project called networking [26:02] 101 say I need you to be my teacher. I [26:04] need to know every I want to become a [26:05] world-class expert on this topic. Help [26:08] me. I know nothing. Right? Come up with [26:10] a plan and and you're going to get a [26:11] plan. Right? Hey, you're going to go [26:13] read this book. You're going to read [26:14] this blog. You're going to go do this [26:15] thing, right? Here's some exercises to [26:16] do. Here's some, you know, things to go [26:17] learn on your own. Uh you you you can [26:19] bootstrap yourself quite effectively [26:21] with with that tool. Uh now that said, [26:25] you should talk to other people. You [26:26] should engage. You should find you know [26:28] other people you can learn from. But uh [26:31] it it is absolutely a amplifier on your [26:34] ability or accelerator on your ability [26:35] to learn that that we haven't you know [26:38] we didn't have when we were younger. Um, [26:40] you know, we we had books and people you [26:41] could hunt down and maybe they knew [26:42] something and maybe they didn't. Uh, so [26:44] like take advantage of that technology. [26:45] Absolutely. [26:47] >> Awesome. Sounds good. Final thoughts, [26:48] advice. [26:50] >> I think this is this is great advice. I [26:51] think um what you said, I just did that [26:53] yesterday with BGB VPN. Hey, teach me [26:55] everything. This is something that I [26:56] have not dealt with before. Uh, and [26:59] Jim's giving me hard time. So, I need to [27:00] go figure out uh how it works. So, and [27:03] this is this is how I learned from Chad [27:04] GPD and you guys should do this every [27:06] day, right? Uh the other thing is you [27:10] really have to love what you do. If you [27:12] are not loving what you're doing, go [27:14] find something else. But if you are do [27:16] you have to do what you do, then do [27:18] everything you can to love it. Like [27:19] there is in in any area that you go in [27:22] um the world's really growing in every [27:24] which way and there is so much [27:26] opportunity. There is so much noise out [27:28] there about how AI is taking jobs away [27:30] and things like that. It's quite the [27:32] opposite. [27:33] >> There is it's just deepening what we can [27:35] do. It's making us more empowered. It's [27:38] giving us ability to do bigger projects [27:40] that you would normally require multiple [27:42] people, multiple uh hours. Now you can [27:45] do it all on your own. It's a fantastic [27:47] new world, right? Oh, and then one [27:49] advice, try to not become a manager. [27:53] Okay? Stay stay. [27:58] Okay. Okay. If you stay an individual [28:00] contributor, you learn a lot more. Okay. [28:02] Managing managing AI agents is far [28:05] easier than managing people. So, [28:08] >> shots fired. Shots fired. [28:11] >> Okay, Jim, I think you need to respond [28:13] to that. Final thoughts. Piece of [28:15] advice. [28:16] >> Final thoughts. I'll talk to you on [28:18] Slack. [28:20] >> I kid. I kid. Um, the cool thing about [28:22] being a manager like me is that I'm a [28:24] technical manager and I still stay [28:26] plenty technical. So, uh, no, like like [28:29] jokes aside, right? Um, again, I'm going [28:31] to call back to Andrew and say what he [28:33] said. Like I was having that exact [28:35] conversation with with a group at at one [28:37] of the breaks and um I I think like [28:41] going a step further on like yeah, use [28:43] use AI and and that's that's all great. [28:44] Like it's super important, but I think [28:46] like separating yourself like making [28:50] yourself distinct in that space somehow. [28:52] Figure out what it is that you love and [28:55] lean into it specifically with AI to do [28:58] something different. It's it's I call it [29:00] thinking outside the box. Like you know [29:03] I think that's going to be super [29:04] critical especially for people that are [29:06] getting started in their careers. Um you [29:08] know I I was telling the folks I was [29:10] chatting with like I actually require um [29:12] all of my engineers to use Aentic AI. [29:15] It's not an option. It it is a hard [29:17] requirement. So like um finding people [29:19] that will think outside the box, lean [29:21] into it heavily and then separate [29:23] themselves from the group by showing [29:24] that they can you know be special in a [29:27] world where like Claude Code and and [29:30] Gemini and Codeex are doing all the [29:32] things for you. Um I think that's that's [29:34] important and it's it's everpresent [29:36] right now. [29:38] >> Makes sense. Joseph, maybe you can bring [29:40] it home and [29:41] >> yeah, I mean all great uh excellent [29:44] advice here. You know, I'll just add to [29:46] say that, you know, lean into your [29:47] fundamentals. I think that's that's very [29:49] important. A lot of the concepts, you [29:52] know, I have enough gray hair to say [29:53] that being in networking for so long [29:54] that, you know, I can still take a [29:56] textbook on queuing theory and make a [29:58] lot of things, you know, apply to what [30:00] we have today. So, you know, what you [30:02] learned in school, especially for new [30:04] hires, you know, it's not something that [30:06] you waste. I mean, it it's going to live [30:07] there. The problems change, solution [30:09] space is there. Um, just lean into the [30:12] fundamentals and when you're at your [30:14] work, uh, learn troubleshooting. I think [30:17] that's that's going to be very important [30:18] to all the aha moments. Even if it was [30:21] one of your stupid mistakes, still a [30:24] learning. So, [30:26] >> even if you're a manager, [30:29] >> makes sense. Thank you all. I hope all [30:31] of you guys had uh, you know, as much [30:34] fun as we did uh, exchanging ideas and [30:38] learning something new. U so that brings [30:41] us to the end of at least this session.