CoreWeave Automates Network Telemetry to Prevent AI Outages
@Scale
Large-scale AI clusters demand zero-packet-loss synchronicity across tens of thousands of GPUs, forcing network operations to rely on custom event-driven automation over traditional IT monitoring to prevent job crashes.
## Background
Published by @Scale on YouTube on August 31, 2026, this presentation features Jim Julson, VP of Infrastructure Networking at CoreWeave, speaking live at the Santa Clara Convention Center. According to the page description, the session explores the operational challenges and nuances of managing mission-critical machine learning and AI network infrastructure at scale.
## TL;DR
Jim Julson outlines the key operational differences between traditional IT and massive AI/ML networks, where a single link failure can disrupt training across tens of thousands of GPUs. To overcome these scaling hurdles, CoreWeave structures its strategy around four cornerstones: specialized teams (with ~50% developers), standardized frameworks, custom observability and event-driven automation platforms, and continuous New Product Integration (NPI) cycles. By leveraging dynamic decision trees and the custom CoreWeave Cumulus CLI, the company automates fault isolation, lowers MTTD/MTTR, and maintains uptime across complex NVIDIA-centric fabrics.
## Atmosphere & style
The presentation carries a pragmatic, battle-tested energy, blending high-stakes technical candor with relatable humor—such as comparing rapid hardware iterations to an 'NPI hand grenade' or 'building the plane as it's falling to earth.' Julson speaks with the grounded authority of an experienced operational leader, acknowledging both the technical complexity of AI fabrics and the human toll that network disruptions exact on engineering teams. Watching the talk feels like an insightful, behind-the-scenes look into high-velocity infrastructure engineering, driven by a relentless focus on practical problem-solving over vendor fluff.
## Key takeaways
- [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) Unlike traditional enterprise IT, large-scale AI/ML networks (up to 128,000 GPUs) require strict synchronous, line-rate communication across every endpoint without dropping packets or experiencing link instability.
- [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) A physical or logical failure anywhere in an AI network immediately halts the entire training job, causing severe financial loss and human strain.
- [00:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=40s) CoreWeave structures its network operations around four core cornerstones: People, Frameworks, Systems, and New Product Integration (NPI).
- [00:05](https://www.youtube.com/watch?v=6fd07bX9qQo&t=5s) CoreWeave intentionally uses specialized organizational silos to cultivate deep subject matter expertise, staffing its network engineering department with roughly 50% software developers.
- [00:05](https://www.youtube.com/watch?v=6fd07bX9qQo&t=5s) Off-the-shelf telemetry and monitoring tools fail at the scale of millions of data center ports, necessitating custom-built observability platforms.
- [00:05](https://www.youtube.com/watch?v=6fd07bX9qQo&t=5s) New Product Integration (NPI) cycles occur every six months, constantly disrupting existing telemetry, switch ASICs, fabric designs, and automation schemas.
- [00:10](https://www.youtube.com/watch?v=6fd07bX9qQo&t=10s) CoreWeave implements an event-driven automation decision engine that automatically filters telemetry noise, isolates degraded links, alerts field technicians, and runs validation before re-entry.
- [00:14](https://www.youtube.com/watch?v=6fd07bX9qQo&t=14s) Operational success relies on treating infrastructure as code backed by multiple synchronized sources of truth sharing common software models rather than a single monolith.
- [00:14](https://www.youtube.com/watch?v=6fd07bX9qQo&t=14s) To avoid manual switch management across massive NVIDIA-centric footprints, CoreWeave built a custom Cumulus CLI to centralize and automate multi-site operations.
- [17:26](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1046s) Managing rapid NPI cycles requires tight vendor partnerships where vendors work side-by-side to solve real-time pain rather than relying on annual software release cadences.
- [17:26](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1046s) Evaluating new network generations involves validating the entire ecosystem simultaneously, including optics, fiber plants, switches, telemetry, and GPU infrastructure.
- [19:31](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1171s) The primary operational goal in AI network management is accelerating fault detection and remediation speed to maximize continuous GPU training uptime.
## Detailed walkthrough
### [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) Operating AI and ML Networks at Scale vs. Traditional IT
Jim Julson, VP of Infrastructure Networking at CoreWeave, kicks off the session by laying out the sheer difference in scale and precision required when operating AI/ML infrastructure compared to traditional enterprise IT networks. In a typical IT setup, high availability is straightforward—you place a load balancer with a Virtual IP (VIP) in front of web servers or distribute traffic via Anycast across hundreds or thousands of instances. If a single redundant connection fails, it is usually benign because TCP-based protocol stacks gracefully handle the disruption without impacting the overall end-user experience.
In stark contrast, AI and ML workloads—such as massive training jobs spanning 128,000 GPUs—demand that every single GPU Network Interface Card (NIC) remains continuously online, passing traffic at line rate without dropping packets, link flaps, or Bit Error Rate (BER) sliding window issues. Because training jobs require synchronous communication across the entire cluster, a single compromised link or failing NIC directly impacts the job. Preparing these cluster deployments involves thousands of engineering hours across the entire stack—"from womb to tomb"—meaning any network breakdown inflicts severe financial costs and an emotional, human toll on the teams involved.
- Traditional enterprise networks rely on load balancers, VIPs, Anycast, and resilient TCP protocol stacks to absorb link failures seamlessly.
- Large-scale AI/ML networks require all 128,000 GPUs to synchronously communicate at full line rate without dropping packets or experiencing link instability.
- A physical or logical failure anywhere in an AI network immediately disrupts the entire training job, wasting thousands of prep hours.
[00:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=40s) The Four Cornerstones of AI Infrastructure Operations
To operationalize networks spanning hundreds of thousands of endpoints, millions of ports, millions of optics, and tens of thousands of miles of fiber strands, Julson frames CoreWeave's operational strategy around four primary "cornerstones" modeled after a construction blueprint (a theme inspired by his personal side project of building a home movie theater).
- **People:** Crucial due to the rapid pace of deployment; hiring and organizing highly capable engineering teams is paramount.
- **Frameworks:** Standardized environments that enable early and continuous testing, avoiding costly post-deployment regressions.
- **Systems:** Robust, custom-built telemetry and observability platforms designed to handle massive data center scale and event-driven decisions.
- **New Product Integration (NPI):** The rapid pipeline for bringing new GPU architectures and ASICs to market on compressed (~6-month) timelines.
[00:05](https://www.youtube.com/watch?v=6fd07bX9qQo&t=5s) Deep Dive into People, Frameworks, and Systems
Julson emphasizes that while traditional engineering models often shun organizational silos, CoreWeave leverages specialized silos intentionally. These structural silos allow engineers to become deep subject matter experts rather than generalists. Notably, about 50% of the network engineering organization consists of software developers, enabling the team to build custom tooling alongside infrastructure.
When it comes to frameworks, standard off-the-shelf testing and monitoring stacks fail when forced to scale to millions of ports within a single data center. Monitoring a standard backbone via gNMI collectors across regions is manageable, but rich, real-time metadata processing for massive AI fabrics requires proprietary innovation. Off-the-shelf telemetry solutions simply do not exist at this scale.
Julson jokingly refers to New Product Integration (NPI) as "the architecture and engineering and design hand grenade." Every time a new GPU generation or switch ASIC arrives, all pre-existing telemetry, schemas, event-driven automations, and physical fabric designs are disrupted. The entire engineering lifecycle must be executed again under compressed six-month timelines.
- CoreWeave’s network engineering department is composed of roughly 50% developers to build specialized tools and systems.
- Commercial monitoring systems fail under the telemetry and metadata demands of millions of ports in a single data center.
- NPI cycles repeat every six months, forcing continuous innovation across ASIC architectures, fabric designs, and telemetry schemas.
[00:10](https://www.youtube.com/watch?v=6fd07bX9qQo&t=10s) Observability, Event-Driven Automation, and Sources of Truth
Building an AI cluster under tight timelines is like "building the plane as it's falling to earth." Customers require massive GPU capacity immediately, forcing network teams to perform real-time testing, optical light-level verification, and transmit/receive (TX/RX) error catching prior to handing nodes over to tenants.
To manage this, CoreWeave relies heavily on streaming telemetry and logging to drive down Mean Time to Detection (MTTD) and Mean Time to Resolution (MTTR). However, collecting raw data without filter mechanisms leads to "analysis paralysis"—much like enabling info-level logging and wading through lines of unhelpful logs. To resolve this, CoreWeave constructed a custom, dynamic "if this then that" event-driven decision tree platform.
This automated eventing system actively monitors fabrics (whether built on Spectrum-X, RoCE/IB, or traditional Infiniband) for link flaps, optical degradation, or traffic saturation. When a faulty link is detected, the automated system does not just disable it; it evaluates which data center it belongs to, dispatches the local technicians, and runs validation criteria before allowing the repaired link back into the live fabric.
- High-speed observability targets lower MTTD and MTTR across complex network fabrics.
- Event-driven automation acts as a complex "if this then that" decision matrix to isolate faulty links without manual human intervention.
- Operational success relies on treating infrastructure as code backed by comprehensive metadata across multiple sources of truth.
[00:14](https://www.youtube.com/watch?v=6fd07bX9qQo&t=14s) The Build vs. Operate Divide and the CoreWeave Cumulus CLI
Julson addresses the operational divide between the *Build* side and the *Operate* side of infrastructure engineering. While both domains share common underlying tools, they serve fundamentally different operational goals. Bridging this gap requires reliable infrastructure metadata. Julson explicitly rejects the concept of a single "source of truth," arguing instead that complex scale requires multiple interconnected "sources of truth" sharing common software models.
To solve the sheer operational burden of managing massive networks without infinite headcount, CoreWeave's development team created the **CoreWeave Cumulus CLI**. Managing thousands of switches by logging into individual routers manually is an unscalable nightmare that leads to engineering burnout. By abstracting command-line operations across their NVIDIA-centric infrastructure ("we bleed green and blue"), the custom CLI allows engineers to interact with vast fabric footprints efficiently and safely.
- CoreWeave uses multiple distinct "sources of truth" tied together through shared software layers rather than relying on a single monolith.
- CoreWeave's network footprint heavily features NVIDIA networking ecosystems (such as Cumulus Linux and Spectrum-X).
- The custom-developed CoreWeave Cumulus CLI mitigates the operational strain of manually managing individual switches across massive AI data centers.
[16:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1000s) Scaling Tooling and Systemic CLI Interfaces
Building effective tools requires designing systems that scale seamlessly alongside infrastructure and human capital. The core mission of an internal tool like the Cumulus CLI is to take a collection of complex, multi-layered tasks and wrap them inside an easily consumable interface. This wrapper enables network engineers to interact with infrastructure globally rather than having to execute commands on a site-by-site basis. By allowing engineers to execute chained or trained command workflows directly across the broader architectural design, the tool functions as a highly fungible asset that can be continually massaged and adapted for each operational iteration.
- Multi-site orchestration is centralized through unified CLI abstraction wrappers.
- Executing chained commands globally reduces the cognitive overhead of single-site manual execution.
- Command structures adapt continuously to support New Product Integration (NPI) workflows.
[17:26](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1046s) Managing New Product Integration and Vendor Relationships
Handling New Product Integration (NPI) feels like learning how to "put the pin back in the NPI hand grenade." Successfully navigating these transitions hinges heavily on moving beyond standard vendor dynamics to build genuine operational partnerships. True partners are willing to experience your operational pain firsthand and work side-by-side to solve it. This is a stark contrast to standard vendors who dismiss critical issues by making vague promises about fixes coming in future release cycles that only roll out annually.
Maintaining high engineering velocity during NPI demands an immense effort from technical teams. When introducing a brand-new ASIC, engineers aren't just validating the silicon itself—they must simultaneously evaluate the accompanying optical ecosystem, the associated fiber plant, a complete reference design schema, and all corresponding telemetry, logging, and alerting systems. When stacked on top of evaluating the underlying GPU infrastructure at the same time, the workload is massive. Sustaining this pace requires intentional systems, robust tooling, and repeatable software frameworks where component parts can be constantly refreshed.
- Vendor partnerships must prioritize joint problem-solving over rigid annual software release cadences.
- Full NPI validation covers optics, fiber plants, reference schemas, logging, and telemetry alongside GPU infrastructure.
- Aligning engineering talent with repeatable tasks early and often prevents velocity loss across continuous release cycles.
[19:31](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1171s) Future Directions and Operationalizing Infrastructure at Scale
Looking toward what comes next, the overarching objective centers on maximizing training efficiency and workload continuity. Building on high-profile public industry relationships—such as work with Meta and OpenAI, alongside projects like MRC and Metari—the primary focus remains scaling engineering talent to detect and remediate network issues faster. The ultimate metric of success is keeping machine learning training runs moving without disruption.
While CoreWeave has established a strong, proven build program for rapidly deploying data centers, operationalization is a continuous, evolving journey. Perfection may be an elusive target, but building more resilient systems, hardening operational frameworks, and driving deeper network optimization remain the core targets for managing AI at scale.
- Priority is given to accelerating fault detection and remediation speed to maximize continuous GPU training uptime.
- High-performance data center deployment programs require continuous, iterative operational refinement.
- Scaled infrastructure depends on constantly strengthening software frameworks and engineering tooling.
## Notable quotes
> [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) from womb to tomb — Jim Julson
> [00:05](https://www.youtube.com/watch?v=6fd07bX9qQo&t=5s) the architecture and engineering and design hand grenade — Jim Julson
> [00:10](https://www.youtube.com/watch?v=6fd07bX9qQo&t=10s) building the plane as it's falling to earth — Jim Julson
> [00:14](https://www.youtube.com/watch?v=6fd07bX9qQo&t=14s) we bleed green and blue — Jim Julson
> [17:26](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1046s) put the pin back in the NPI hand grenade — Jim Julson
## Mentioned
Jim Julson, CoreWeave, NVIDIA, Cumulus Linux, Spectrum-X, RoCE, InfiniBand, Meta, OpenAI, MRC, Metari, CoreWeave Cumulus CLI, Santa Clara Convention Center, @Scale
Operating large-scale machine learning networks requires fundamentally different operational paradigms than enterprise IT, as synchronous training across massive clusters cannot tolerate packet drops or link instability. According to CoreWeave's Jim Julson, sustaining these environments requires treating networking as software, deploying automated event-driven remediation, and structuring teams to handle rapid six-month hardware refreshes.
The Zero-Tolerance Demands of Distributed AI Training (0:03)
Traditional enterprise networks rely on load balancers, Virtual IPs, Anycast routing, and TCP protocol stacks to absorb redundant connection failures without affecting end-user performance. In contrast, large-scale AI and machine learning workloads, such as distributed training runs spanning 128,000 GPUs, require synchronous communication where every GPU network interface card (NIC) must continuously pass traffic at line rate. Physical or logical network anomalies, such as bit error rate window degradation or link flapping, directly disrupt the entire training job and waste thousands of engineering preparation hours across the stack.
Because training depends on uninterrupted coordination across millions of ports, optics, and fiber strands, the cost of network instability extends beyond immediate downtime to severe financial and human costs. This sensitivity necessitates operational strategies centered on keeping every link active and clean, rather than relying on standard failover mechanisms that are inadequate for tightly coupled GPU compute fabrics.
Specialized Engineering Silos and Abstraction Tooling (0:05)
To manage infrastructure at this scale, CoreWeave deliberately employs specialized organizational silos rather than generalist engineering models, populating roughly 50% of the network engineering department with software developers. This composition allows the team to build custom tooling tailored to data centers containing millions of ports, where commercial off-the-shelf monitoring solutions fail to scale. Julson also rejects the conventional model of a single monolithic 'source of truth,' arguing instead that massive networks require multiple interconnected sources of truth bridged by shared software data models.
To bridge the gap between building infrastructure and operating it day-to-day, CoreWeave developed the CoreWeave Cumulus CLI. Logging into individual switches manually across vast NVIDIA-centric fabrics (such as Cumulus Linux and Spectrum-X) proved unscalable and drove engineer burnout. The custom CLI abstracts multi-layered switch operations into unified command chains that engineers execute globally across data centers rather than per device, significantly reducing cognitive overhead.
Event-Driven Telemetry and Automated Link Isolation (0:10)
Rapid deployment schedules require testing and validation in real time while clusters are being provisioned for tenants. CoreWeave relies on streaming telemetry and logging to reduce Mean Time to Detection (MTTD) and Mean Time to Resolution (MTTR). However, uncurated logging creates high volumes of noise that result in analysis paralysis. To counteract this, the company implemented a custom event-driven decision tree platform that filters raw telemetry and executes automated intervention workflows.
When the system identifies optical degradation, packet saturation, or link flaps across Spectrum-X, RoCE, or InfiniBand fabrics, it automatically intervenes without human input. Rather than simply disabling the degraded port, the decision engine identifies the facility, alerts regional field technicians, and executes automated validation tests before permitting the repaired link back into the active production fabric.
Absorbing Accelerated Hardware Integration Cycles (17:26)
New Product Integration (NPI) cycles arrive roughly every six months as new GPU architectures and switch ASICs are released. Julson describes each NPI cycle as a design shock that disrupts existing telemetry schemas, eventing rules, and physical fabric designs. Adapting to this cadence requires treating the network as code with repeatable frameworks that can be continually re-engineered under tight delivery windows.
Successfully managing rapid hardware churn also hinges on shifting vendor relationships away from transactional engagements. Standard enterprise vendors often defer critical bug fixes to rigid annual release cycles, which collapses under six-month deployment targets. Julson stresses the need for direct operational partnerships where vendors collaborate on validating full ecosystems—including optics, fiber plants, reference schemas, and telemetry pipelines alongside the silicon itself.
CoreWeave frames machine learning network operations as an evolving discipline where uninterrupted training uptime is the defining metric. While deploying massive data centers can be standardized, Julson acknowledges that operational perfection remains difficult, requiring continuous refinement of internal software abstractions, faster automated fault isolation, and deep alignment with hardware suppliers.
## Background
该视频由 @Scale 频道于 2026 年 9 月 3 日发布在 YouTube 平台上。内容记录了 CoreWeave 基础架构网络副总裁 Jim Julson 在圣克拉拉会议中心（SCCC）的现场演讲，重点剖析超大规模机器学习与 AI 基础设施网络在实际运营中所面临的严苛挑战与工程细节。
## TL;DR
演讲围绕 CoreWeave 在支撑十万卡级别 GPU 集群时的网络运营实践展开，对比了传统 IT 网络与无损高带宽 AI 网络之间的本质差异。Jim Julson 提出了支撑超大规模网络运维的“四大基石”——人员、框架、系统与新产品导入（NPI），并指出网络工程团队必须高度软件化（开发者占比达 50%）。针对链路震荡、TX/RX 误码与误码率超标等常态化故障，团队自研了 CoreWeave Cumulus CLI 与事件驱动的自动化闭环系统，同时破除了对“单一真相来源（SSOT）”的盲信，全力保障分布式大模型训练不发生停滞与中断。
## Atmosphere & style
整场演讲呈现出一种硬核、务实且极具战地前线色彩的工程黑客气质。Jim Julson 的风格坦诚直率、风趣幽默，用“飞机一边下坠一边空中拼装”以及“随时引爆的手榴弹”等生动比喻，真实还原了在海啸般交付重压下工程师死磕故障的高压日常。台下与台上的交流充满默契与共鸣，将超大规模基础设施背后的技术阵痛、自嘲与极致工程追求展现得淋漓尽致。
## Bulletin
- [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) 在大规模 AI/ML 网络中，多达 12.8 万颗 GPU 必须保持全线速无损同步通信，任何微小的链路震荡（link flap）或误码率超标都会直接导致涉及数万卡的大模型分布式训练任务中断崩溃。
- [02:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=160s) CoreWeave 提出支撑超大规模网络运营的四大基石——人员（People）、框架（Frameworks）、系统（Systems）与新产品导入（NPI），其中网络工程团队中软件开发者占比已高达约 50%。
- [12:12](https://www.youtube.com/watch?v=6fd07bX9qQo&t=732s) 针对超大集群运维，CoreWeave 打造了 CoreWeave Cumulus CLI 与自研流式遥测系统，驳斥绝对的单一真相来源（SSOT）神话，主张以多元真相来源集与基础设施即代码（IaC）驱动事件自动化排障。
- [15:25](https://www.youtube.com/watch?v=6fd07bX9qQo&t=925s) NPI 硬件迭代周期已被极致压缩至六个月，每次引入新 ASIC 都伴随光学、拓扑及监控告警体系的全面重构，网络团队的核心战略重心已从单纯数据中心建设转向深度的全生命周期运维化。
## Key takeaways
- [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) 传统企业级 IT 网络依赖 TCP 与冗余路由能够容忍单链路故障，而 AI/ML 训练集群对丢包、抖动及 BER 漂移零容忍，底层网络异常会直接中断分布式训练。
- [01:15](https://www.youtube.com/watch?v=6fd07bX9qQo&t=75s) 大规模训练中断会带来极高的算力沉没成本与人力消耗，网络物理层、逻辑层、人员与流程必须完全同频共振。
- [02:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=160s) 超大规模网络运营依赖四大基石体系：研发型专家团队（人员）、全流程标准规范（框架）、近实时遥测平台（系统）与压缩周期的架构落地（NPI）。
- [05:43](https://www.youtube.com/watch?v=6fd07bX9qQo&t=343s) 市面上现成的开箱即用网络监控工具无法承载单数据中心数百万端口的高密元数据与近实时分析需求，倒逼自研定制化软件平台。
- [07:15](https://www.youtube.com/watch?v=6fd07bX9qQo&t=435s) 新产品导入（NPI）被形容为架构设计的“破片手雷”，每半年一次的芯片迭代必然伴随未知硬件 Bug，需要极强的工程打磨韧性。
- [08:51](https://www.youtube.com/watch?v=6fd07bX9qQo&t=531s) 打破通才思维与职能孤岛定势，通过建立专业细分领域（SME）以应对高精尖网络微观缺陷，同时将开发人员比例提升至约 50% 以全面软件化运营。
- [10:02](https://www.youtube.com/watch?v=6fd07bX9qQo&t=602s) 在客户算力饥渴与极限交付交付节奏下，物理节点并网前必须执行近实时 NCCL 压测，严密筛查收发光功率异常及光纤配线缺陷。
- [11:20](https://www.youtube.com/watch?v=6fd07bX9qQo&t=680s) 运维监控的核心在于追求极致缩短的平均检测时间（MTTD），但必须借助事件驱动过滤低价值噪声，防止团队陷入分析瘫痪。
- [12:12](https://www.youtube.com/watch?v=6fd07bX9qQo&t=732s) 链路隔离仅是运维闭环的开端，自动化的核心难点在于机房匹配、现场工单派发以及重新入网复测准则的完整决策树固化。
- [13:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=820s) 驳斥绝对单一真相来源（SSOT）的迷思，主张大型基础设施应由构建（Build）与运维（Operate）的多元真相来源集协同支撑。
- [15:25](https://www.youtube.com/watch?v=6fd07bX9qQo&t=925s) 自研 CoreWeave Cumulus CLI 替代传统逐台 SSH 单机敲命令的运维方式，实现跨全局多数据中心的链式复合指令执行与抽象屏蔽。
- [17:30](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1050s) 大规模 AI 演化必须依赖能够紧密排障、敏捷修复的生态合作伙伴，而非按年度版本更新的传统被动硬件供应商。
- [20:17](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1217s) 所有网络工程架构与自动化的终极目标，均是确保客户大模型训练持续稳定运转不发生停滞中断。
- [20:35](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1235s) 在完成物理数据中心选址与硬件集群快速建设部署后，团队的战略重心全面转向系统架构的深层次运维化（operationalization）。
## Detailed walkthrough
### [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) 传统网络与大规模 AI/ML 网络的本质差异
CoreWeave 基础架构网络副总裁 Jim Julson 开门见山地指出，在经历了前序演讲（如 MRC 和 Meta 团队的分享）之后，他希望将视角拉回到更贴近工业落地的大规模 AI/ML 网络实际运营中，深入探讨基于 Spectrum-X、传统 RoCE 以及 InfiniBand（IB）承载的大模型训练工作负载所面临的严苛挑战。
传统企业级 IT 网络的架构与故障模型早已是行业常识（“duh，这根本不是什么秘密”）：前端挂载负载均衡器配置 VIP，后端支撑 Web 服务器，或者在成百上千个节点间配置 Anycast 分布式路由。在传统网络中，前端冗余链路如果断开一条，往往无关紧要；上层基于 TCP 的协议栈可以从容应对抖动和丢包，绝大部分服务本身就能自动屏蔽底层异常。
然而，在 AI/ML 世界中，游戏规则彻底改变。一个集群中多达 12.8 万颗 GPU 必须在全线速（line rate）下进行严格的同步通信，整个过程绝对不允许丢包、不允许链路震荡（link flap），更容不得误码率（BER）滑动窗口等深层问题。在这样的训练任务中，每一块 GPU 网卡（NIC）都必须持续保持活跃并顺畅转发流量。一旦某处发生断流或链路抖动，哪怕只是 Tier-1 或 Tier-2 后端交换机上一个微小的 link flap，就会瞬间让涉及数万卡的大规模分布式训练任务整体阻塞或崩溃。
这种中断带来的不仅是直接的硬件停机，更意味着沉重的综合成本。一次大规模训练任务的上线，背后凝聚了横跨软硬件全栈、数百乃至数千名工程师“从摇篮到坟墓”（womb to tomb）倾注的数千小时准备心血。链路故障引发的重试不仅带来巨大的资金损失，更给工程团队带来了极大的精力与情绪消耗。如果物理层、逻辑层、人员、流程以及底层支撑系统无法完全保持同频共振，每一个微小瑕疵都会被放大为全栈工程师的巨大运营负担。
```mermaid
flowchart TD
accTitle: 传统网络与 AI 训练网络特性的对比
accDescr: 对比传统 IT 网络容错机制与大规模 AI 训练集群对无损同步的苛刻要求
subgraph TRAD["传统企业级 IT 网络"]
direction TB
LB["负载均衡器 (VIP)"]
WEB["后端服务 / Anycast 节点"]
LB --> WEB
NOTE1["容错机制强：单链路断开无感知，依赖 TCP 自动重传"]
end
subgraph AIML["大规模 AI / ML 训练网络"]
direction TB
GPUS["128,000 颗 GPU 节点"]
NET["Spectrum-X / RoCE / InfiniBand"]
GPUS <--> NET
NOTE2["零容忍：必须全线速同步通信，零丢包、零震荡、零 BER 漂移"]
end
```
- **通信规模与严苛性**：多达 128,000 颗 GPU 进行无损、全线速、同步通信，任何单点网卡或核心 Fabric 链路抖动均可直接中断训练。
- **协议栈表现差异**：传统面向 TCP 的应用能够吸收 link flap 影响，而基于 RDMA（RoCE/IB）的大模型集合通信对网络时延抖动与微小丢包极其敏感。
- **代价维度**：任务中断涉及昂贵的算力租赁成本、工程排查工时，以及工程师承受的高压心理负担。
---
### [02:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=160s) 运营超大规模网络的“四大基石”
为了系统性地管理分布在数十个数据中心、数以百万计的端口与光模块，以及长度达到数万英里的光纤线路，Jim 借用自己业余生活的一个爱好来打比方——他在工作之余有种“受虐式”的充沛精力，喜欢在自家搞建筑施工、亲手打造家庭影院。因此，他把超大规模网络的运维蓝图提炼为四个关键的“基石”（Cornerstones）：**人员（People）**、**框架（Frameworks）**、**系统（Systems）**与 **新产品导入（NPI, New Product Integration）**。
```mermaid
flowchart LR
accTitle: 超大规模网络运营的四大基石体系
accDescr: 展现人员、框架、系统与新产品导入之间的相互支撑与依赖关系
subgraph FOUNDATION["网络运营四大基石"]
direction TB
P["人员 (People)<br/>高密度的研发型专家团队"]
F["框架 (Frameworks)<br/>跨硬件、软件、流程与测试的标准规范"]
S["系统 (Systems)<br/>支撑百万级端口近实时遥测与事件驱动平台"]
N["NPI (新产品导入)<br/>压缩周期内的架构颠覆与快速上线"]
P <--> F
F <--> S
S <--> N
end
```
- **人员（People）**：尽管大谈“以人为本”略显陈词滥调，但在当前 CoreWeave 爆炸性增长的部署速率和巨大运营规模下，优秀工程师团队的重要性超过了 Jim 过往职业生涯中的任何时刻。
- **框架（Frameworks）**：任何网络问题都“无法在事后补救测试”。必须建立覆盖全流程的测试与规范框架，及早且频繁地进行验证。在 AI 网络中，版本回滚（Regression）通常意味着不可承受的资金与资源代价。
- **系统（Systems）**：这是技术极客最热衷的部分。必须构建足够强壮的自研底层平台，并与框架无缝打通，供工程团队消费调用。
- **NPI（New Product Integration，新产品导入）**：涵盖把全新一代 GPU 从设计、数据中心物理适配、软硬件编码、遥测监控到全面投产的完整流程。Jim 戏称其为“架构与工程设计的破片手雷”，通常新架构刚扔进生产环境时“多半无法直接跑通，全凭工程团队死磕打磨直到可用”。
---
### [05:43](https://www.youtube.com/watch?v=6fd07bX9qQo&t=343s) 框架落地、监控盲区与 NPI “设计手雷”
将基石落到工程实处时，往往伴随着打破常规的架构重塑：
首先是**框架的全面贯穿**。框架一词在业内常被过度包装，但在 CoreWeave，它具体涵盖了组织架构设计（分工与职责权属）、硬件规格标准、软件交互规范、以及全套操作与故障处理流程。面对全球数十个数据中心、数百万网络端点以及庞大的供电规模（“我已经数不清具体的兆瓦甚至吉瓦总容量了”），没有标准框架作为抓手，规模化运营根本无从谈起。
其次是**可观测性系统的极限挑战**。在万卡集群面前，市面上任何开箱即用（off-the-shelf）的现成网络监控、遥测和可观测性工具链均告失效。现成方案或许能在骨干网（Backbone）或跨地域线路上基于 gNMI 收集链路状态，但面对**单个数据中心内即存在数百万个端口**的高密场景，再加上为了实现近实时（near-real-time）事件驱动决策而必须注入的海量元数据（metadata），传统系统完全无法承载这种吞吐与计算压力。为此，CoreWeave 必须大量投入定制软件开发，通过不断的技术自研创新来追赶基础设施的增长速度。
最后是残酷的 **NPI 周期压缩**。前一代 GPU 架构历经数月乃至数年才打磨稳定、摸透遥测指标与事件驱动自动化规则，然而在激烈竞争下，下一次硬件迭代往往只留给团队短短六个月时间。新 ASIC 芯片的引入必然带来未知的新 Bug，这是网络硬件的物理规律，无法完全规避。面对这种每半年重来一次的“设计手雷”，工程师必须具备强大的心理承受力——正如 Jim 开玩笑展示的搞笑配图那样，“解决这些问题的第一步往往是接受现实，大哭一场，擦干眼泪继续干活（尽管照片上那个秃头老哥绝不是我，我头发可多得多）”。
---
### [08:51](https://www.youtube.com/watch?v=6fd07bX9qQo&t=531s) 组织反常识：重塑“职能孤岛”与研发型网络团队
在组织架构与团队构建上，Jim 提出了一个反传统的观点：打破外界对“职能孤岛（Silos）”的负面定势思维。在超大规模 AI 网络这种极度精细化、高复杂度的体系中，通才（Generalist）的认知深度往往难以应对隐蔽的微观网络缺陷。因此，有意识地设立专业“孤岛”，反而能为工程师提供深耕特定垂直领域的空间，培养出能够在细分瓶颈上钻研到底的资深领域专家（SME）。
与这一组织哲学相配合的是团队技能构成的彻底转型。在 CoreWeave 的网络工程团队中，**开发人员（Developers）的比例高达 50% 左右**。网络团队不再只是传统的配置交换机与布线的网工，而是一支重度依靠软件编码驱动、自研自动化控制面与遥测管道的复合型工程研发部队。这种深度的软件工程投资，成为了维系百万级端口高速稳定运转的根本支撑。
- **组织深度优于通识广度**：通过专业分组打造特定网络协议和硬件层面的顶级领域专家，攻克极限故障。
- **50% 开发者占比**：网络工程团队全面向软件研发演进，将底层 Fabric 的管理、调度和监控全面代码化、系统化。
### [10:02](https://www.youtube.com/watch?v=6fd07bX9qQo&t=602s) 边缘失控与实时观测：在“坠落的飞机”上交付算力网络
在动辄超过 100,000 张 GPU 的超大规模集群中运行大模型训练任务时，网络层面的任何微小抖动都可能演变为灾难性事故。最常见的故障包括链路频繁震荡（Flapping Link）、发射/接收端报错（TX/RX Errors）以及误码率滑动窗口超标（sliding BER windows out of spec）。对于负责建设数据中心并交付底层基础设施的团队而言，这已经成了家常便饭。Jim Julson 坦言，许多同行都能深切体会那种“飞机一边下坠、一边在空中拼装飞机”的巨大压力——客户对算力的渴求永远是“昨天就必须要”，市面上的有效容量极度匮乏，工程团队只能以极快的节奏拼命交付。
但在这种极限速度下，如何确保交付的基础设施具备极高的质量？团队必须在物理节点交付给客户使用之前，在近乎实时的状态下进行严格的 NCCL 测试，高强度压测整个网络拓扑（Fabric），把隐藏在光纤链路中的 TX/RX 报错、光功率电平异常、光纤配线问题等各种隐患彻底排查出来。这一切都要求测试工具必须能够持续运转，并在近乎实时的窗口内反馈结果。因此，整个运维架构的核心基石就是可观测性（Observability），涵盖流式遥测（Streaming Telemetry）、集中日志记录与事件系统，其终极目标是将“平均故障检测时间”（MTTD）压缩到最短，并顺畅打通到“平均修复时间”（MTTR）的自动化闭环。
然而，采集数据绝非越多越好。工程师通常都有过错误开启 `INFO` 甚至调试级别日志的惨痛经历——屏幕上瞬间充斥着一行行毫无价值的垃圾信息，让人无所适从。如果在 AI 规模的网络中漫无目的地采集海量低价值数据，团队很快就会陷入“分析瘫痪”（Analysis Paralysis）。因此，必须借助事件驱动的自动化机制将人工排障从这种无谓的信息噪声中抽象剥离出来，让系统能够自主做出判断。
- **超大集群故障常态**：十万卡级别 GPU 训练对链路震荡（Flapping）、TX/RX 误码以及 BER 误码率超标等物理层异常极度敏感，微小异常即可拖垮整个任务。
- **交付与质检并重**：在节点交付用户前，必须进行实时的 NCCL 吞吐与通信测试，全链路排查收发光功率与光纤布线缺陷。
- **观测性核心指标**：以流式遥测为基石，优先追求极致缩短的平均检测时间（MTTD），并无缝对接事件驱动自动化流程以压低平均修复时间（MTTR）。
```mermaid
flowchart TD
accTitle: 链路异常事件驱动自动化闭环流程
accDescr: 展示从链路故障遥测捕获，到自动化决策树分流、工单派发、离线质检及重新并网的闭环生命周期。
subgraph Obs["遥测与异常捕获"]
A["流式遥测 / 误码监控<br/>(TX/RX, BER, 链路震荡)"] --> B{"事件驱动分析引擎<br/>(过滤冗余数据，防分析瘫痪)"}
end
subgraph DecisionTree["自动化 IFTTT 决策决策流"]
B --> C["自动隔离/摘除故障链路"]
C --> D["匹配所属数据中心与机位"]
D --> E["自动生成工单并指派驻场技术员"]
end
subgraph Recovery["离线诊断与复测"]
E --> F["驻场排查光模块与光纤链路"]
F --> G{"重新上线测试套件<br/>(NCCL/光衰/误码率核验)"}
G -- "不合规" --> F
G -- "合规" --> H["安全回归网络 Fabric"]
end
```
### [12:12](https://www.youtube.com/watch?v=6fd07bX9qQo&t=732s) 事件驱动自动化与“单一真相来源”的迷思
在当下的 AI 网络构建中，尽管诸如 MRC（Meta RoCE Consortium）等新兴架构正在快速演进，主流集群依然依赖 NVIDIA Spectrum-X、原生 RoCE（Vanilla RoCE）或者 InfiniBand 架构。当这些高速网络底座发生故障时，会对工作负载产生即时致命的冲击。为此，CoreWeave 自研了一套事件系统，能够在海量 Fabric 规模下快速捕获链路震荡、错误包与带宽饱和状态，并即刻触发自动化决策流。Jim Julson 将这套机制比作一个庞大而精密的“IFTTT”（If This Then That）逻辑决策树。
这套决策树绝非表面看起来那么简单。业界常常轻描淡写地认为“链路震荡摘掉不就行了吗？”，但实际工程复杂度远超想象：这条链路被摘除后应该进入什么维护状态？它属于哪一个数据中心？应该由哪一位驻场技术人员去现场处理？在更换光模块或跳线之后，需要通过怎样的测试矩阵与验收准则，才能确保重新接回 Fabric 时不会引发次生灾害？所有这些极其繁琐的运营逻辑，都必须被提炼并固化到软件系统之中。
谈及基础设施即代码（IaC）时，Jim Julson 针对行业中常年挂在嘴边的“单一真相来源”（Single Source of Truth, SSOT）提出了强烈质疑。他直言“根本不存在什么绝对的单一真相来源，这种说法纯粹是鬼扯”，并欢迎台下观众在会后随时与他辩论。在他看来，现实世界由“构建（Build）”和“运维（Operate）”两个既有交叉又截然不同的领域组成，各自拥有专门的团队和差异化框架。搭建这两个世界之间桥梁的，实际上是多元的“真相来源集”（Sources of Truth）。团队必须在每个对象层级上沉淀高度精准的元数据，唯有将基础设施视作代码严谨管理，才有可能跟上大模型训练算力的扩展步伐。
- **复杂故障处理闭环**：隔离链路仅仅是第一步，后续的机房归属路由、现场人员调度、再入网校验准则才是自动化软件的核心挑战。
- **驳斥 SSOT 执念**：认为大型数据中心无法靠单一数据库作为绝对权威，必须依赖面向构建和运营双重场景的“多元真相来源”（Sources of Truth）协同工作。
- **精细化元数据管理**：在万卡集群规模下，每个网络与算力实体必须拥有严格的元数据描述，这是基础设施代码化（IaC）与水平扩展的刚性前提。
### [15:25](https://www.youtube.com/watch?v=6fd07bX9qQo&t=925s) 规模化管理工具：CoreWeave Cumulus CLI 与 NPI 飞轮
由于底层深度绑定英伟达技术栈，Jim Julson 戏称 CoreWeave 内部“流淌着绿色与蓝色的血液”。随着数据中心以令人发指的速度在各地铺开，团队面临着极其尖锐的结构性矛盾：业务交付合同压在头上，基础设施扩张如海啸般涌来，但工程团队根本不可能以匹配这种膨胀速度去招聘网络工程师，也无法瞬间把所有自动化框架造好。在这种重压下，工程师无法忍受传统运维方式——逐台通过 SSH 登录交换机和路由器并敲命令排查故障，在十万卡 AI 规模下这样做“绝对没人能在这种岗位上干得长久”。
为了解放生产力，CoreWeave 的研发团队打造了 CoreWeave Cumulus CLI。这款工具将复杂的运维与排障逻辑封装成标准化、可复用的抽象接口，使工程师不再受限于单点、单机房视角，而是在全局范围内跨集群架构链式执行组合命令。它具备高度的灵活性（Fungible），能够随着硬件平台的迭代持续演化，为跨站点的大规模基础设施管理提供了统一的交互层。
这一能力在“新产品导入”（NPI, New Product Integration）过程中体现得淋漓尽致。Jim Julson 将 NPI 比作一颗极其危险的“随时可能引爆的手榴弹”。当企业要求工程团队引入一款全新的网络 ASIC 芯片时，他们需要同步验证整个配套的光通信生态、重新铺设的光纤系统、全新的网络拓扑参考架构，并在全新 ASIC 与下一代 GPU 之间同时完成遥测、日志和告警体系的从头适配。在这个高压链条中，拥有能够同甘共苦的真正“合作伙伴”（Partner）而非推诿扯皮的“供应商”（Vendor）是成败的关键——传统供应商往往用“两期之后的年度版本更新再修复”来敷衍故障，这在 AI 时代是不可承受之重。团队必须通过强有力的系统与工具来维持极高的工程迭代速度（Engineering Velocity），在“为人构建框架、向框架注入软件”的循环中不断轮转，确保新硬件能够平稳、规模化地融入生产环境。
- **CoreWeave Cumulus CLI**：针对全局规模化设计的高阶 CLI 工具，替代单机逐台手动敲命令的传统运维，支持跨多数据中心链式下发复合运维指令。
- **真伙伴 vs 伪供应商**：AI 规模化需要能够与客户并肩排障、敏捷迭代软件补丁的生态合作伙伴，而不是守着按年更新路线图的傲慢供应商。
- **NPI 的高难度工程飞轮**：新 ASIC 的引入意味着芯片、光模块、光纤介质、网络拓扑参考架构以及监控告警体系的全面重构，需要凭借模块化工程框架保障高周转速率。
- **未来演进方向**：积极探索 MRC 等下一代网络协议，与 Meta、OpenAI 等行业顶尖团队深度合作，共同探索更极致的网络故障检测与自主修复前沿。
### [20:17](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1217s) 确保训练持续运转：从数据中心建设到运维体系的演进
所有的网络排查、架构设计与系统调优，归根结底都是为了同一个目标——全力确保大模型的训练进程能够持续运转而不被中断（"making sure that training keeps moving"）。Jim Julson 强调，这正是 CoreWeave 当前放在第一位的核心要务。他提到，CoreWeave 已经建立了一套非常强大的基础设施建设体系（build program），在物理数据中心的选址构建与硬件集群的快速部署交付方面，团队不仅经验丰富而且执行力极高。
然而，网络工程的下半场在于长期的“运维化”（operationalization）。Jim 坦率地指出，集群的网络运维是一个动态演进、持续攻坚的过程，团队或许永远无法做到绝对的无懈可击或十全十美，但这正是工程团队必须坚持设立并全力追求的高标准目标。面对未来更大规模的计算集群，CoreWeave 正集中精力提升底层系统的鲁棒性，进一步强化整体运维架构与自动化框架，致力于将大规模 AI 网络运营能力推向下一个全新的水平。
- **核心使命**：所有技术方案与故障处置的首要准则，是保证客户的 AI 分布式训练作业不发生停滞中断。
- **建设与运维的侧重演进**：CoreWeave 在数据中心的高效建设与规模化部署（build & deploy）上已经具备扎实优势，当前战略重心正全面转向深度的系统运维化（operationalization）。
- **未来方向**：持续打磨并增强系统架构与支撑框架的鲁棒性（robustness），向下一阶段的超大规模网络运维体系跨越。
## Notable quotes
> [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) duh，这根本不是什么秘密 — Jim Julson
> [00:03](https://www.youtube.com/watch?v=6fd07bX9qQo&t=3s) 从摇篮到坟墓 — Jim Julson
> [02:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=160s) 无法在事后补救测试 — Jim Julson
> [02:40](https://www.youtube.com/watch?v=6fd07bX9qQo&t=160s) 架构与工程设计的破片手雷 — Jim Julson
> [05:43](https://www.youtube.com/watch?v=6fd07bX9qQo&t=343s) 我已经数不清具体的兆瓦甚至吉瓦总容量了 — Jim Julson
> [05:43](https://www.youtube.com/watch?v=6fd07bX9qQo&t=343s) 解决这些问题的第一步往往是接受现实，大哭一场，擦干眼泪继续干活（尽管照片上那个秃头老哥绝不是我，我头发可多得多） — Jim Julson
> [10:02](https://www.youtube.com/watch?v=6fd07bX9qQo&t=602s) 飞机一边下坠、一边在空中拼装飞机 — Jim Julson
> [10:02](https://www.youtube.com/watch?v=6fd07bX9qQo&t=602s) 昨天就必须要 — Jim Julson
> [12:12](https://www.youtube.com/watch?v=6fd07bX9qQo&t=732s) 链路震荡摘掉不就行了吗？ — Jim Julson
> [12:12](https://www.youtube.com/watch?v=6fd07bX9qQo&t=732s) 根本不存在什么绝对的单一真相来源，这种说法纯粹是鬼扯 — Jim Julson
> [15:25](https://www.youtube.com/watch?v=6fd07bX9qQo&t=925s) 流淌着绿色与蓝色的血液 — Jim Julson
> [15:25](https://www.youtube.com/watch?v=6fd07bX9qQo&t=925s) 绝对没人能在这种岗位上干得长久 — Jim Julson
> [15:25](https://www.youtube.com/watch?v=6fd07bX9qQo&t=925s) 两期之后的年度版本更新再修复 — Jim Julson
> [20:17](https://www.youtube.com/watch?v=6fd07bX9qQo&t=1217s) making sure that training keeps moving — Jim Julson
## Mentioned
CoreWeave, Jim Julson, Spectrum-X, RoCE, InfiniBand, MRC, Meta, OpenAI, CoreWeave Cumulus CLI, NVIDIA, NCCL, gNMI, Santa Clara Convention Center, @Scale
## Video description
Jim Julson from Coreweave presents "Anatomy of @scale Machine Learning Network Operations" live at the Santa Clara Convention Center.
As machine learning continues to grow in scale, so too do the challenges of operating these mission critical networks. Join me in a journey through the operational challenges, and nuances found in Machine Learning and AI at scale around Infrastructure Networks.
Learn more about @Scale here: atscaleconference.com
[Source](https://www.youtube.com/watch?v=6fd07bX9qQo)
Source quotes
[0:03] Jim Julson: duh，这根本不是什么秘密
[0:03] Jim Julson: 从摇篮到坟墓
[2:40] Jim Julson: 无法在事后补救测试
[2:40] Jim Julson: 架构与工程设计的破片手雷
[5:43] Jim Julson: 我已经数不清具体的兆瓦甚至吉瓦总容量了
[5:43] Jim Julson: 解决这些问题的第一步往往是接受现实，大哭一场，擦干眼泪继续干活（尽管照片上那个秃头老哥绝不是我，我头发可多得多）
[10:02] Jim Julson: 飞机一边下坠、一边在空中拼装飞机
[10:02] Jim Julson: 昨天就必须要
[12:12] Jim Julson: 链路震荡摘掉不就行了吗？
[12:12] Jim Julson: 根本不存在什么绝对的单一真相来源，这种说法纯粹是鬼扯
[15:25] Jim Julson: 流淌着绿色与蓝色的血液
[15:25] Jim Julson: 绝对没人能在这种岗位上干得长久
[15:25] Jim Julson: 两期之后的年度版本更新再修复
[20:17] Jim Julson: making sure that training keeps moving
Source quotes
[0:03] Jim Julson: from womb to tomb
[0:05] Jim Julson: the architecture and engineering and design hand grenade
[0:10] Jim Julson: building the plane as it's falling to earth
[0:14] Jim Julson: we bleed green and blue
[17:26] Jim Julson: put the pin back in the NPI hand grenade
Original transcript
[0:03] Hello. Hello. Thank you for joining. My
[0:05] name is Jim Jolson. I'm the uh VP of
[0:07] infrastructure networking at Coreweave.
[0:09] Uh today we're going to kind of go on a
[0:11] journey about uh some of the problems
[0:13] and the solutions of operating AI and ML
[0:16] networks at scale. I think it's fairly
[0:18] fortuitous to go after the likes of uh
[0:21] the MRC talk and the Metari talk. Um,
[0:24] I'm going to kind of bring it back into
[0:26] the more traditional um, uh, AI and ML
[0:29] workloads on top of things like Spectrum
[0:32] X and traditional Rocky as well as as IB
[0:35] and the problems that kind of go with
[0:36] that. So, let's kind of dive in. So,
[0:39] traditional networks, right, some of
[0:41] these things are kind of duh, like it's
[0:43] not a mystery. Uh, it's it's very basic
[0:45] workloads, right? There's the standard
[0:47] enterprise IT kind of you got a load
[0:50] balancer with a VIP in front and you got
[0:52] your services that are backended by like
[0:54] web servers or maybe you got some
[0:56] anycast going on distributed across
[0:57] hundreds or thousands like table stakes
[0:59] these days. Um in AI and ML you've got
[1:04] 128,000 GPUs that all need to
[1:06] synchronously talk to one another at
[1:08] line rate without dropping packets
[1:10] without link flaps without beer sliding
[1:12] window problems. um you know you you
[1:14] need your uh your redundant connections
[1:17] right for your front-end network and one
[1:19] of them can go down and it's kind of no
[1:21] big deal in AI and ML every one of your
[1:24] GPU nicks must remain up and passing
[1:26] traffic or there's actual real world
[1:28] problems that are uh that that will
[1:31] impact these training jobs um and the
[1:33] impact of these training jobs you know
[1:34] there it's not like your traditional um
[1:37] services stack right there are thousands
[1:40] of hours that go into prepping across
[1:42] Ross, you know, hundreds or even
[1:44] thousands of of people and engineers
[1:47] working across the stack from womb to
[1:48] tomb to bring all this stuff to bear to
[1:51] even get it going. So there's monetary
[1:53] impact, there's there's human toll,
[1:55] right? There's the emotional toll of it,
[1:57] which is also equally frustrating.
[2:00] Um so so ultimately at the end of the
[2:02] day if your physical layer, your logical
[2:06] layer, your people, your process, your
[2:08] systems aren't completely in tune with
[2:10] one another, it's felt across these
[2:13] stacks. It it creates an operational
[2:15] burden to everybody involved, right? The
[2:18] littlest things like you'd think a link
[2:20] flap happening um up on a tier one or a
[2:22] tier two for your backend network. uh in
[2:24] a traditional network, I mean you might
[2:26] feel it, but the the protocol stack TCP
[2:29] based is going to usually overcome that
[2:31] kind of thing. Email pretty pretty uh
[2:33] benign in that in that regard. So not so
[2:37] in the AI AI and ML world.
[2:40] Okay. So I've been doing some
[2:42] construction projects uh of my own. It's
[2:44] kind of like this sick thing I have
[2:46] where I like to work more after my
[2:48] workday um building a movie theater. So
[2:50] I thought I'd just like theme this whole
[2:52] thing to like construction and
[2:54] blueprints. And so I want to talk about
[2:56] like how we look at oper
[2:59] operationalizing
[3:00] hundreds of thousands of endpoints and
[3:03] millions of ports and millions of optics
[3:05] and fiber strands that measure in the
[3:07] tens of thousands of miles long. Uh and
[3:10] and what we're going to do is call those
[3:12] the cornerstones. Okay. We look at these
[3:15] in four basic areas. Four cornerstones.
[3:17] um people, frameworks, systems, and then
[3:21] NPI, which is new product integration,
[3:23] right? I I know it's a little like
[3:25] cliche, right? The people thing. Um but
[3:28] it's important. The the people matter.
[3:30] Um and it matters. It's true of any
[3:32] business, but I think that with the
[3:34] speed at which we're growing, the pace
[3:36] at which we're deploying, and the size
[3:39] at which we're operating, the people are
[3:42] everything in this equation. uh more so
[3:44] than than at any other time in my own
[3:46] career. I I can say this statement is
[3:48] true. Uh the frameworks the the general
[3:50] concept here, right, is you know it it's
[3:54] you just heard a second ago that that
[3:56] you can't test after the fact. It it
[3:59] just doesn't work right. You have to
[4:01] have frameworks established and
[4:02] developed so that you're testing early
[4:04] and often. Regression is usually not a
[4:07] good option. Um, in any environment, but
[4:10] in particular AI and ML, regression
[4:13] costs money, it costs time, it costs
[4:15] resource, engineering cycles, creates a
[4:17] lot of headache. On the system side, the
[4:21] systems are like my favorite thing to
[4:22] talk about. That brings out my inner
[4:25] engineering nerd. uh and and for me I
[4:28] think that a big part of this journey in
[4:31] AI and ML overall is is the systems we
[4:35] get to build are extremely robust but
[4:37] you tie those systems into the framework
[4:39] so that the people can consume the
[4:40] frameworks right and it's it's part of a
[4:43] a broader ecosystem um we're going to
[4:45] dive more into this as we go and then
[4:47] finally NPI new product integration
[4:49] right so um new product integration is
[4:52] is taking everything that you do to
[4:54] bring a GPU to market uh whether it's
[4:57] the the first iteration of it for the
[4:59] company or for the industry and you do
[5:01] all the engineering work associated all
[5:03] the coding all the development all the
[5:05] prep and the planning you get the data
[5:07] center physically fit for it you get all
[5:09] the engineering teams behind it your
[5:10] telemetry observability and you deploy
[5:13] it and you hope to god it works it
[5:14] usually doesn't and then you keep going
[5:16] until it does NPI
[5:20] okay so on the people side I'm not going
[5:22] to belver this point too much um because
[5:24] again I I recognize it's cool and sexy,
[5:26] but it is worth just again one one more
[5:28] footnote is it's important. You got to
[5:31] have good engineering teams. You got to
[5:33] treat your engineers well. And I think
[5:34] there's a lot of cool problems to solve
[5:36] right now in AI and ML in general,
[5:38] particularly in networking because it's
[5:40] so incredibly fast-paced.
[5:43] All right, frameworks. This is where I
[5:45] think it it gets more interesting,
[5:46] right? Frameworks is an often overloaded
[5:50] term, but frameworks generally speaking
[5:52] are akin to what you just heard uh with
[5:55] regards to the testing framework built
[5:57] to test the the metari frame the metari
[6:00] implementation so that you're not having
[6:02] those regressions. And when we think
[6:04] about frameworks, we have frameworks for
[6:06] people, right? Org structures, how you
[6:08] divvy up work and who's doing what. We
[6:10] have frameworks for the hardware side,
[6:12] for the software side, and even our
[6:14] process and our tooling all have
[6:15] frameworks. Without them, we could not
[6:17] scale to the uh dozens and dozens and
[6:20] dozens of data centers we have globally
[6:22] across the millions of endpoints. Um I
[6:25] don't even know what our megawatt count
[6:27] is or gigawatt count is at this point,
[6:28] but it's it's insane. Uh without the
[6:31] frameworks in place, we wouldn't be able
[6:32] to to operate any of it.
[6:35] on the system side, right? Some of the
[6:37] problems that you have today, you really
[6:40] can't take an offtheshelf monitoring or
[6:43] observability or telemetry stack and
[6:45] just throw it out there on some compute
[6:47] and then hope it works. Like that's just
[6:49] not going to happen. You might be able
[6:51] to scale that across like a subset of
[6:53] your infrastructure like your backbone.
[6:55] You might be able to get your G&MI
[6:57] endpoints and your collector stood up
[6:58] and collect across, you know, the United
[7:01] States and rinse and repeat that per GIO
[7:03] and you're probably pretty good. You
[7:05] start talking about, you know, millions
[7:08] of ports in a single data center and
[7:10] then all of the metadata that you have
[7:13] to enrichen that with in order to
[7:15] basically make event driven uh decisions
[7:18] in as near real time as possible. you'll
[7:20] quickly discover that those solutions
[7:22] just do not exist, at least not at the
[7:24] scales that you need, right? Uh so
[7:26] there's a there's a constant challenge
[7:28] there. Um the software to manage those
[7:31] endpoints, a lot of that stuff is custom
[7:33] as well, right? So you're constantly in
[7:35] this song and dance of innovation just
[7:37] to even be able to manage the
[7:39] infrastructure that you have on hand.
[7:42] And then finally, NPI, which I like to
[7:44] call the architecture and engineering
[7:46] and design hand grenade. take everything
[7:49] that you just did for months or even
[7:51] years, right, to bring a GPU to market
[7:54] and now go ahead and go ahead and repeat
[7:57] all of that in a super compressed
[7:59] timeline. I don't know, six months and
[8:02] do it all again. All the telemetry that
[8:04] you just figured out and figured out how
[8:06] all your eventing, your event driven
[8:08] automation, all of your schema, uh your
[8:11] fabric design, new ASIC, new problems.
[8:14] It's inevitable. Uh it's not a it's not
[8:16] unique to any one vendor. It's it's just
[8:17] a a fact of life for us in networking.
[8:20] Um and you get to rinse and repeat that.
[8:22] So I like to call the NPI thing uh the
[8:24] the engineering and design hand grenade.
[8:26] Little cheeky there. Okay. So now the
[8:29] like the better stuff. Let's talk about
[8:31] like solving some of these things. Um
[8:34] you know,
[8:36] yeah, we'll we'll we'll dive into that.
[8:38] I'm just going to say this is this is
[8:39] the first step of solving these problems
[8:42] is getting the acceptance part out of
[8:44] the way. just you you just cry about it
[8:46] and you move on. And that's really not
[8:47] me. I have way more hair.
[8:51] Okay, so back to people again. Again,
[8:54] we're not going to we're not going to
[8:55] talk uh a lot about it, but I I just
[8:58] want to highlight one thing, right? So,
[9:00] this is not to show you an org chart
[9:01] here. what you're looking at the the the
[9:04] idea is to say that unlike any other
[9:07] place that for me over the years that
[9:09] I've worked the the silos uh
[9:12] traditionally have a bad they're like a
[9:13] loaded term I think in our industry but
[9:15] in this case I feel like the silos are
[9:17] really good silos allow us to create
[9:19] subject matter experts that can dive
[9:21] deeper than most generalists can go
[9:24] right and so it's been very valuable for
[9:26] us to have these silos that allow people
[9:29] to hyperfocus in key areas and
[9:31] particularly We pour a lot of effort
[9:33] into development. So our our network
[9:36] engineering organization is is comprised
[9:38] of about 50% developers and I think
[9:42] that's super super duper important right
[9:44] and we like to invest heavily in those
[9:46] people across those engineering uh
[9:49] resources.
[9:51] Okay. So frameworks again we're back to
[9:54] our loaded term but um some of the
[9:57] frameworks we've developed in order to
[9:59] operate at this scale you know you you
[10:02] you hear a lot about what what happens
[10:04] when you have a flappy link in a
[10:06] training job that's running across
[10:08] 100,000 plus GPUs or when you have TX
[10:11] errors RX errors sliding beer windows
[10:13] get you out of spec right these are very
[10:16] common problems for us when we're
[10:17] building data centers and trying to
[10:19] deliver this infrastructure
[10:22] I I think that many of my colleagues
[10:24] would would uh confirm how difficult it
[10:26] is to build the plane as it's falling to
[10:30] earth, which is effectively what we're
[10:31] always doing, right? We're trying to
[10:33] deliver as much capacity as possible
[10:35] because all of our customers need it
[10:36] yesterday. There's not enough out there
[10:39] for them. And so some of the things that
[10:41] go along with that are, okay, if I've
[10:43] got to deliver this capacity, how do I
[10:46] make sure that I'm delivering something
[10:47] quality? Am I test am I doing nickel
[10:49] testing in real time before I hand over
[10:52] these nodes to be consumed by a
[10:54] customer? Is my fabric have I exercised
[10:57] my fabric and caught all of the little
[10:59] idiosyncrasies that go with operating
[11:01] these fabrics i.e. the TXRX errors? Am I
[11:04] do I have you know uh light level
[11:07] problems across the stack? Are all of my
[11:09] fiber plants uh you know uh tested and
[11:13] solid? That takes us to having to have
[11:15] tools and testing constantly and doing
[11:18] things in as near real time as possible.
[11:21] So I think one of the biggest pieces in
[11:23] the the framework's cornerstone in
[11:25] particular for us is observability. And
[11:27] that's all your streaming telemetry.
[11:29] It's your logging, your eventing,
[11:31] everything that goes with being able to
[11:34] ultimately drive to your fastest
[11:36] meanantime to detection and then
[11:38] subsequently your meanantime to
[11:40] resolution which takes you into the
[11:42] event driven automation section. Right?
[11:44] So taking all this data like you can
[11:47] collect a lot of data. I think all of us
[11:49] can probably like think back to when we
[11:52] had info level logging enabled on
[11:53] something and you're just like what the
[11:55] crap am I looking at? like it's just
[11:57] like line after line of garbage, right?
[11:59] The same thing can happen here. You can
[12:01] collect too much information that has
[12:03] little to no value and then having to
[12:05] parse through that becomes really
[12:08] difficult over time. Uh you you end up
[12:10] in analysis paralysis, right? So the
[12:12] event- driven automation serves to
[12:14] abstract away the human side and get us
[12:17] to a point where we don't actually care.
[12:20] So when you talk about like MRC and
[12:22] Metari, those are things that are are
[12:24] emerging, right? But in today's world,
[12:28] we're using things like Spectre X or
[12:30] Vanilla Rocky things or Infiniband,
[12:32] right? And when you have fabric
[12:34] problems, they still pose a very real
[12:36] problem to the actual workload. And so
[12:40] what we've done is created an events
[12:42] eventing system that allows us to
[12:44] basically go through all of our fabrics
[12:46] at scale and detect when there are
[12:49] errors, detect when there are flapping
[12:51] links, detect when we have saturation so
[12:54] that we can actually event on those. Uh
[12:57] how many of you have ever seen like an
[12:58] if this then that like application
[13:01] diagram where it's just like a big tree
[13:03] that goes down and if you click this go
[13:04] here. That's effectively what we've
[13:06] built. It's an if this then that. And
[13:09] these trees are not easy to navigate. Um
[13:14] it is not as simple as oh if a link's
[13:17] flapping just take it out. Well yes but
[13:19] also what are you going to do with that
[13:20] link? Which data center does that belong
[13:23] to? Which texts are going to service it?
[13:25] What is the testing criteria to put it
[13:27] back in your fabric so that it doesn't
[13:28] cause another problem? So all these
[13:30] complexities are captured in software
[13:33] for us. Right? And so we put a lot of
[13:35] emphasis on the frameworks. And then
[13:37] finally uh the infrastructure sources of
[13:39] truth. Um you are only as good as the
[13:43] metadata that you have on a per object
[13:45] basis. If you don't have good data about
[13:48] your infrastructure and treat your
[13:50] infrastructure as code, you will not be
[13:53] able to scale it the way that AI and ML
[13:55] needs to. It just you just you just
[13:56] cannot.
[13:59] Okay. So the systems and tooling side,
[14:01] right? Like it's it's the beating heart.
[14:02] I know I've switched from construction
[14:04] to anatomy. Bear with me. I'm just maybe
[14:06] a little add. Um, but the uh the the the
[14:10] the the fact is is there's two worlds
[14:12] that we live in. Um, from an engineering
[14:14] perspective, we got the build side and
[14:16] you have the operate side. Those two
[14:19] worlds definitely have a lot of
[14:21] crossover, right? um the tooling that
[14:24] we've developed, all of the the
[14:26] mechanisms, the process wrapped around
[14:29] all of the build and subsequently the
[14:32] operate has a lot of overlap, but still
[14:34] they're different enough to where we
[14:36] have different teams focusing on those
[14:38] different aspects and as such the
[14:40] frameworks that we develop are even
[14:41] different for those. Um the sources of
[14:44] truth are kind of the common theme
[14:46] between those and you notice I say
[14:48] sources. Uh, it's because I I firmly
[14:51] believe that there's no such thing as a
[14:53] source of truth. I think that is just
[14:54] garbage. Um, you can totally like hit me
[14:57] in the hall and like smack me and say
[14:58] I'm wrong. I'm happy to have that debate
[15:00] with you. Um, but the the sources of
[15:02] truth are really important in building
[15:05] uh, you know, kind of a bridge between
[15:06] those two worlds. uh just because of the
[15:08] fact that the soft there's so much
[15:10] software shared across the two realms of
[15:13] build and operate that without having
[15:15] that common bond you it it you end up
[15:17] recreating the wheel many times over and
[15:20] I've done that and and had to unwind
[15:22] that and it's very difficult. Uh I put
[15:25] something up here to talk about uh as a
[15:28] kind of a like a fun thing that we did
[15:30] like we had some of our dev teams
[15:32] created the corewave cumulus CLI. So
[15:34] we're a big Nvidia shop. Anybody that
[15:36] knows Cororeweave, like we're plastered
[15:38] all over the NVIDIA site, so they know
[15:39] we're like we bleed green and blue. But,
[15:42] uh, the Core Cumulus CLI was a was a way
[15:45] to answer a problem that we had there.
[15:48] We're building all these data centers.
[15:49] We're growing all this infrastructure.
[15:51] We're doing all these things and we
[15:54] can't hire the people fast enough,
[15:56] right? We can't build the frameworks
[15:58] fast enough because we have to be able
[16:00] to deliver the infrastructure that we've
[16:02] committed to on our contracts. And so
[16:04] there's this constant song and dance and
[16:06] balance between that world of delivery
[16:09] and operations. And so we had to start
[16:12] innovating on how we interface with the
[16:15] infrastructure over time. Anybody that's
[16:18] managed any network infrastructure for
[16:20] any size company will will tell you that
[16:24] having to log on to multiple routers and
[16:26] execute commands individually is
[16:29] terrible, right? and doing that at the
[16:32] scales of AI and ML. Uh, I don't know
[16:35] anybody that would stay at that job for
[16:37] long. Like, it just wouldn't happen. I
[16:39] mean, unless there's some crazy people
[16:40] in here that actually enjoy that, but I
[16:42] don't know. I saw somebody nod. That's
[16:45] crazy. So, um, yeah, the the the core of
[16:48] Cumulus CLI, its goal is to take a a a
[16:50] bunch of complex tasks, put them in a
[16:53] wrapper that's consumable and allow
[16:55] engineers to interface with the
[16:57] infrastructure not on a single site
[16:59] basis, but globally to be able to
[17:01] execute trained or chained uh commands
[17:04] rather across our architecture and our
[17:07] design, right? It is a fungeable tool
[17:10] that you that that we can um massage
[17:13] based on each iteration which we get
[17:15] into with NPI next. Uh but it's it's one
[17:18] of those things it goes back to the the
[17:20] frameworks and building things that
[17:22] engineers and people can consume at
[17:26] scale. And then finally, how do you put
[17:29] the pin back in the NPI hand grenade? So
[17:33] I think you know we all have vendors. We
[17:38] don't all have partners and I think
[17:39] having partners is super key in in this
[17:42] journey. People that are willing to like
[17:44] experience your pain and then
[17:46] subsequently partner with you to get out
[17:49] of that pain. That's a huge piece of it.
[17:51] And I know that's not the sexy side of
[17:53] like like talking about networking and
[17:55] whatnot, but it matters. And anybody
[17:57] that's ever had a bad vendor that wasn't
[17:59] there for you when you needed them that
[18:01] just said, "Yeah, that's coming out in
[18:02] two more releases. We got you." And then
[18:04] they have like an annual release cycle.
[18:07] Yeah, you you know that pain and so you
[18:09] need partners that that will that will
[18:11] work with you to be your success with
[18:13] you, right? Finally, I think that the
[18:16] this goes back to a little bit of the
[18:17] people side. The engineering velocity is
[18:21] everything for new product integration.
[18:23] When when you ask your engineering team
[18:25] to vet and validate a brand new ASIC,
[18:28] all of the optical ecosystem that goes
[18:30] with it, all of the new fiber plant that
[18:32] has to go with it, a brand new reference
[18:35] design schema, and then all of the
[18:38] telemetry, the logging, the alerting,
[18:40] and everything just for that new ASIC,
[18:42] let alone the ASIC and the GPU
[18:44] infrastructure simultaneously. You're
[18:47] asking a lot and to build and maintain
[18:49] that velocity requires the good
[18:52] frameworks so that the people can
[18:54] consume the frameworks. It requires good
[18:55] systems and tooling that are intentional
[18:58] that you have designed to facilitate
[19:01] engineering velocity. Right? The thing
[19:03] is is is this is a non-stop cycle. You
[19:06] you build the frameworks for the people
[19:08] and you insert the software into the
[19:10] frameworks and then you keep rinsing and
[19:13] repeating over and over and over again.
[19:15] And that's NPI. And so I think that the
[19:17] the success there is all about making
[19:19] sure that you align people with the task
[19:21] at hand early and often because it is a
[19:25] repeating task and you maintain your
[19:27] velocity in that regard.
[19:31] What's next? Um that's there's a lot. Uh
[19:36] I think that you know I I could I could
[19:39] easily talk for an hour on this. Nobody
[19:41] wants to hear that. In fact, I think if
[19:43] I go over 20 minutes and I stand on this
[19:45] X, I'm going down. Um, but I I think
[19:48] that, you know, holistically, you've
[19:50] heard the talks about MRC and and
[19:52] Metari. Um, it's it's public knowledge
[19:55] that we have some pretty great
[19:57] relationships with with Meta and OpenAI.
[20:00] Um we are definitely on a journey trying
[20:03] to partner with um with with the
[20:07] engineering talent required to become
[20:11] more optimized to be faster at detecting
[20:14] problems to be faster at remediating
[20:17] problems all in the name of making sure
[20:20] that training keeps moving. Right? I
[20:22] think that's probably our number one
[20:24] thing right now is we've got a really
[20:26] good build program. We at Corewave like
[20:27] we build data centers and we deploy and
[20:30] we're really good at that. The
[20:32] operationalization aspect of it is
[20:34] ongoing and and I would s I would say
[20:36] that we'll never be perfect at it, but
[20:39] that's our that's our goal, right? And I
[20:41] think we should have those goals. So for
[20:42] us, we're really focused on on making
[20:44] our systems more robust, our frameworks
[20:46] more robust, and getting to that next
[20:48] level. Thank you for your time. Have a
[20:50] great day.
