CoreWeave Rebuilds Network Software as 1.6T AI Switches Hit Scale
@Scale
CoreWeave VP Jim Julson says zero-tolerance AI network demands force automated systems and specialized vendor partnerships to prevent single bit errors from destroying multi-week training runs across 128,000 GPUs.
## Background
Published on YouTube by @Scale on August 27, 2026, this presentation features Jim Julson from CoreWeave. According to the page description, the session takes viewers through the operational challenges and nuances of managing infrastructure networks for AI and machine learning at scale.
## TL;DR
Jim Julson details CoreWeave's strategy for operating massive AI/ML network environments, where even minor latency glitches or single bit errors can halt multi-week training jobs across thousands of GPUs. To address the physical scale of managing millions of optics and half a million fibers across over 100 data centers, CoreWeave relies on four cornerstones: specialized teams, pluggable frameworks, software-driven automation systems, and continuous New Product Integration (NPI). The presentation highlights their proprietary Tron-inspired Master Control Program (MCP) for zero-touch provisioning and bulk diagnostics, emphasizing the critical role of deep vendor partnerships and a culture of continuous operational retooling as speeds advance toward 1.6T.
## Atmosphere & style
The session feels candid, pragmatic, and high-stakes, blending real-world operational grit with technical depth. Julson speaks from experience, capturing both the daunting pressure of 3:00 AM emergency calls and the exhilarating challenge of integrating next-generation hardware under aggressive timelines. The narrative carries a resilient, highly driven tone—underpinned by a culture of ownership and an upbeat confidence when tackling massive scale.
## Key takeaways
- [00:05](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=5s) AI and machine learning networking requires zero-tolerance for packet jitter, bit errors, or link flaps, as a single micro-disruption can halt a training job across thousands of GPUs.
- [00:05](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=5s) Unlike traditional Infrastructure as a Service (IaaS), AI/ML workloads run at 100% line-rate speed across up to 128,000 GPUs simultaneously.
- [03:26](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=206s) CoreWeave structures its global operational blueprint across more than 100 data center locations around four core cornerstones: People, Frameworks, Systems, and New Product Integration (NPI).
- [05:13](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=313s) The physical scale of AI/ML backend networks reaches over 1,000,000 optical transceivers and 500,000 fiber connections, rendering manual troubleshooting by human operators ('meatware') unviable.
- [07:38](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=458s) New Product Integration (NPI) forces teams to constantly reset automation pipelines and SOPs for next-generation hardware architectures on aggressive time-to-market schedules.
- [11:32](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=692s) CoreWeave utilizes a proprietary Master Control Program (MCP) to manage the full lifecycle of switches, from zero-touch provisioning (ZTP) to bulk diagnostic command execution.
- [11:32](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=692s) Observability systems require enriched device metadata—down to individual optic serial numbers and cable patch plans—to effectively trigger event-driven automated remediation.
- [16:52](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1012s) Strong vendor partnerships with shared accountability are critical for troubleshooting cutting-edge hardware deployments, such as liquid-cooled ORV3 racks.
- [17:46](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1066s) Transitioning between network speeds (100G, 400G, 800G, and 1.6T) and multi-rail topologies requires continuous end-to-end retooling of synthetic monitoring, probes, and telemetry schemas.
- [19:06](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1146s) CoreWeave grounds its operational strategy in an owner mindset ('find a problem, fix a problem') and resilient culture to handle continuous future roadmap expansion.
## Detailed walkthrough
### [00:05](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=5s) The Extreme Demands of AI/ML Network Operations
Jim Julson, Vice President of Engineering for Global Infrastructure Networking at CoreWeave, introduces the core challenges of operating AI and machine learning networks at scale. Reflecting on his four-year tenure at CoreWeave, Julson emphasizes that AI/ML networking is fundamentally different from traditional IT enterprise, hyperscale, or cloud environments. In traditional networking, tasks like retrieving email or downloading a GitHub repository can tolerate minor delays, packet jitter, or transient latency spikes without ruining the underlying work.
Conversely, AI and ML workloads require tens or hundreds of thousands of GPUs to communicate simultaneously at 100% line-rate speed without a single flap, bit error, or performance drop. A single network micro-disruption can halt a massive training job, destroying weeks of collective work across multiple teams. This failure mode carries severe consequences: direct financial losses for both the client and the company, alongside an exhausting emotional toll on engineering teams who are routinely woken up at 3:00 AM to deal with degraded jobs.
- AI/ML networking requires interconnecting up to 128,000 GPUs simultaneously at line-rate speeds with zero tolerance for bit error rates or link flaps.
- Unlike traditional Infrastructure as a Service (IaaS), where physical and logical layers have high tolerance for small glitches, AI/ML networks fail spectacularly if all layers are not performing perfectly.
[03:26](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=206s) The Four Cornerstones of CoreWeave's Operational Blueprint
To tackle the complexities of managing a global footprint distributed across more than 100 unique data center locations, CoreWeave structures its operational strategy around four core cornerstones: **People**, **Frameworks**, **Systems**, and **New Product Integration (NPI)**. Every layer—from hardware up to human processes—must operate in complete alignment.
Julson highlights that regardless of how advanced an organization's software or automation becomes, systems will ultimately fail without great people to build and maintain them. Because AI and ML infrastructure leaves no room for operational error, human expertise and tightly integrated delivery chains are essential to achieving consistent perfection across global deployments.
- CoreWeave's operational footprint spans over 100 unique data center locations linked by a global backbone network.
- The four operational cornerstones:
* **People:** Specialized domain experts who design, build, and run the infrastructure.
* **Frameworks:** Extensible glue code, APIs, and procedures connecting people to systems.
* **Systems:** Software platforms and tooling that replace manual human intervention ("meatware").
* **New Product Integration (NPI):** The process of onboarding next-generation hardware and architectures rapidly.
[05:13](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=313s) Frameworks and the Operational Scale of Systems
Frameworks serve as the crucial glue binding human operators to automated systems. Software must be built as pluggable, extensible architecture—such as API endpoints for querying back-end fabrics or streaming telemetry platforms connected to enriched sources of truth.
When discussing systems, Julson notes how the physical scale of AI/ML infrastructure dramatically outpaces traditional environments. While a standard data center might house 20,000 optical transceivers and 10,000 fiber connections, a modern AI/ML backend network scales to half a million fibers and over a million optics. At this magnitude, relying on human "meatware" to manually log into devices and troubleshoot issues is entirely unviable. Automated systems and software-driven remediations are mandatory to sustain operations.
- AI/ML backend networks scale to over 1,000,000 optical transceivers and 500,000 fiber cables in single large-scale deployments.
- Individual backend network builds frequently incorporate many thousands of network switches, with each switch featuring 64 physical connections.
- Observability platforms, streaming telemetry, and enriched metadata down to individual optic serial numbers are required to trace hardware lifecycle states in real time.
[07:38](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=458s) NPI: The Architectural Hand Grenade
Julson candidly labels New Product Integration (NPI) as the "architectural engineering and design hand grenade." After spending months or years perfecting teams, standard operating procedures (SOPs), and real-time software automation for a specific hardware generation, NPI forces engineering teams to reset and do it all over again for next-generation technology on aggressive time-to-market schedules.
Despite the intensity and unrealistic timelines associated with NPI, Julson views it as one of the most rewarding aspects of the job. It constantly pushes engineers to touch state-of-the-art technology and continuously iterate on their systems, frameworks, and personal skills.
- NPI requires rapidly adapting existing automation pipelines, SOPs, and frameworks to support newly released hardware architectures.
- Success in NPI relies on structured team specialization, allowing engineers to move away from generalist roles and build deep subject matter expertise in specific operational domains.
[11:32](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=692s) Observability, Event-Driven Automation, and Tron-Inspired Tooling
Within CoreWeave's operational ecosystem, observability, event-driven automation, and infrastructure sources of truth act as an interdependent triad. Observability platforms (telemetry, logging, metrics, synthetics) exist primarily to feed event-driven automation engines, allowing software to act as the buffer between physical data center technicians and network engineers.
To execute this, CoreWeave relies on its proprietary **Master Control Program (MCP)**—a system named by Julson in honor of his favorite movie, *Tron*. MCP serves as the primary bridge between infrastructure builds and ongoing operations. It exposes API endpoints and custom CLI tooling (such as a custom-developed Cumulus CLI interface) that allow engineers to securely zero-touch provision (ZTP) new switches or execute bulk diagnostic commands across thousands of switches spanning multiple global data centers simultaneously.
- Telemetry platforms must maintain enriched device metadata—including optic serial numbers, cable patch plans, and lifecycle states—to enable effective event-driven automation.
- The **Master Control Program (MCP)** manages the full lifecycle of switches, from initial ZTP configuration rendering to global bulk CLI execution.
- Operational success in NPI and large-scale management depends on deep vendor partnerships to streamline integration and navigate novel hardware behaviors.
[16:52](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1012s) Vendor Partnerships in New Product Introduction (NPI)
Building robust vendor partnerships is essential when deploying cutting-edge machine learning infrastructure. True partnership requires vendors who are fully committed to sharing both the pain and the success of scaling new systems. This means having a partner who is willing to hop on a plane at 3:00 AM to fly to a Las Vegas data center just to troubleshoot why a switch will not boot inside a brand-new, liquid-cooled Open Rack Standard V3 (ORV3) compliant rack. Because companies like CoreWeave are primarily consumers of newly introduced supply chain and hardware technologies rather than creators of every component, these tight relationships are critical enough to warrant dedicated focus. Having experienced the alternative of missing that level of vendor alignment, maintaining strong, hands-on partnerships is a baseline requirement for success in NPI.
- Strong vendor partnerships depend on shared accountability for hardware failures and field deployment struggles.
- Data center deployment challenges include troubleshooting switch boot issues in liquid-cooled ORV3 (Open Rack Standard V3) infrastructure.
- NPI in large-scale ML infrastructure relies heavily on vendor alignment for supply chain, logistics, and rapid technology integration.
[17:46](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1066s) Engineering Velocity, Operational Retooling, and Core Values
Engineering velocity during New Product Introduction is defined by speed, efficiency, and maintaining uncompromising operational standards. At CoreWeave, this operational drive is anchored in core values like "act like an owner," which translates in practice to a simple mantra: "find a problem, fix a problem." Introducing new generations of hardware—such as new GPUs with entirely different NIC lane configurations to map to higher GPU counts—forces engineering teams to retool their entire operational environment. Scaling network ASICs from 100G to 400G, 800G, and 1.6T speeds requires evaluating complex design choices across multi-rail, single-rail, multi-planar, and multi-tier topologies. Every new architecture forces end-to-end updates to telemetry systems, synthetic testing, network probes, sources of truth, and design schemas.
- High engineering velocity requires balancing rapid NPI cycles with strict stability and release standards.
- CoreWeave’s internal guiding principle of "act like an owner" translates operationally to "find a problem, fix a problem."
- Architectural transitions span network speed upgrades (100G to 400G, 800G, and 1.6T) alongside multi-rail, multi-planar, and multi-tier design iterations.
- Hardware changes ripple across the operational stack, requiring continuous retooling of network probes, synthetic monitoring, and configuration design schemas.
[19:06](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1146s) The Scale Ahead and CoreWeave's Operational Mindset
Looking ahead to future roadmaps, the sheer volume of upcoming infrastructure expansion presents a relentless operational pace. With schedules for 2027 and 2028 already completely packed, managing this scale is both daunting and intense. When facing massive technical hurdles or high-risk deployments—what the team jokingly refers to as "full YOLO mode"—CoreWeave CTO Peter Salanki keeps the focus grounded with a characteristic response to extreme operational challenges: "Well, we haven't failed yet. What makes you think we're going to start now?" That confident, resilient attitude drives how CoreWeave approaches operating AI and machine learning networks at scale.
> "Well, we haven't failed yet. What makes you think we're going to start now?"
## Notable quotes
> [03:26](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=206s) The four operational cornerstones: People: Specialized domain experts who design, build, and run the infrastructure. Frameworks: Extensible glue code, APIs, and procedures connecting people to systems. Systems: Software platforms and tooling that replace manual human intervention ("meatware"). New Product Integration (NPI): The process of onboarding next-generation hardware and architectures rapidly. — Jim Julson
> [07:38](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=458s) NPI forces engineering teams to reset and do it all over again for next-generation technology on aggressive time-to-market schedules. — Jim Julson
> [11:32](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=692s) Observability platforms exist primarily to feed event-driven automation engines, allowing software to act as the buffer between physical data center technicians and network engineers. — Jim Julson
> [16:52](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1012s) True partnership requires vendors who are fully committed to sharing both the pain and the success of scaling new systems. — Jim Julson
> [17:46](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1066s) act like an owner
> [17:46](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1066s) find a problem, fix a problem
> [19:06](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1146s) full YOLO mode
> [19:06](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=1146s) Well, we haven't failed yet. What makes you think we're going to start now? — Peter Salanki
## Mentioned
Jim Julson, Peter Salanki, CoreWeave, Master Control Program (MCP), Tron, Cumulus, Open Rack Standard V3 (ORV3), @Scale
Jim Julson, Vice President of Engineering for Global Infrastructure Networking at CoreWeave, argues that AI and machine learning networks represent a fundamental departure from traditional cloud computing, demanding near-flawless physical and logical execution to prevent catastrophic job failures. Sustaining this standard across massive optical fabrics requires shifting from manual interventions to event-driven software platforms, structured team specialization, and tightly coupled vendor integration.
Zero-Tolerance Workloads Versus Traditional Cloud Infrastructure (0:05)
Traditional enterprise and hyperscale cloud networks are engineered around fault-tolerant workloads where packet jitter, minor delays, or brief link degradation rarely impact the overall operation. In contrast, AI and machine learning clusters require up to 128,000 GPUs to communicate concurrently at 100 percent line-rate speed. In these environments, any packet loss, bit error, or transient link flap can disrupt a distributed training checkpoint, invalidating weeks of computation across dozens of teams and imposing steep financial and operational costs.
Because of these zero-tolerance constraints, Julson contends that standard Infrastructure as a Service (IaaS) operating models are unviable. Rather than relying on soft re-transmissions and tolerance for transient glitches, every physical and logical networking layer must operate with consistent perfection to prevent high-profile cluster halts.
Automation Replaces Manual Operations at Extreme Scale (5:13)
CoreWeave's backend infrastructure supports deployments scaling to half a million fiber cables and more than one million optical transceivers, linking thousands of 64-port switches across more than 100 data center locations. At this volume, relying on human operators—what Julson terms "meatware"—to manually log into network devices or troubleshoot degradation is impossible. Sustainable operations require software-driven abstractions and automated remediation pipelines.
To manage this footprint, CoreWeave relies on a triad comprising observability engines, enriched sources of truth, and event-driven automation. Central to this architecture is the proprietary Master Control Program (MCP), an operational platform providing custom CLI tooling and API endpoints. MCP governs the switch lifecycle, executing zero-touch provisioning (ZTP) and enabling network engineers to safely deploy bulk diagnostics and operational changes across thousands of switches globally.
New Product Integration as an Architectural Disruptor (7:38)
Julson characterizes New Product Integration (NPI) as an "architectural engineering and design hand grenade" because it routinely disrupts stabilized operating environments. Just as engineering teams finish standardizing procedures, automated workflows, and monitoring frameworks for a hardware generation, next-generation architectures reset the baseline on compressed timelines.
This continuous retooling is driven by escalating hardware complexity. Networking speeds are migrating rapidly from 100G to 400G, 800G, and 1.6T ASICs, forcing network designers to evaluate multi-tier, multi-rail, and multi-planar topologies. In tandem, changing GPU NIC lane allocations require continuous structural overhauls to device schemas, telemetry pipelines, synthetic probes, and network configuration models.
Shared Vendor Accountability in Physical Deployments (16:52)
Because infrastructure operators are primary consumers of emerging supply chain hardware rather than custom silicon fabricators, Julson highlights direct vendor alignment as an operational necessity. Cutting-edge deployments routinely expose unforeseen edge cases in new specifications, such as switch boot failures within liquid-cooled Open Rack Standard V3 (ORV3) configurations.
Under CoreWeave's operational model, successful hardware onboarding requires vendors to share direct accountability for deployment setbacks. Real-world readiness depends on partner engineering teams being willing to troubleshoot hardware anomalies in the field at unusual hours, ensuring that hardware delivery problems do not stall broader deployment timelines.
With deployment schedules already committed through 2027 and 2028, Julson indicates that scaling AI backbones will continue to present high-risk technical hurdles. Sustaining rapid deployment velocity without compromising network stability hinges on organizational resilience, structured operational ownership, and continuous automation refactoring.
## Background
本视频由技术大会频道 @Scale 于 2026 年 9 月 3 日发布在 YouTube 平台。视频简介指出，随着机器学习规模的不断扩大，运营关键任务网络的挑战也成倍增加，本期内容由来自 CoreWeave 的 Jim Julson 深入探讨大规模机器学习与 AI 基础设施网络的运营挑战与架构细节。
## TL;DR
CoreWeave 全球基础设施网络工程副总裁 Jim Julson 剖析了超大规模 AI/ML 网络的严酷运维现实，指出其支撑 128,000 张 GPU 全线速通信时对微小抖动与误码的“零容忍”特征。针对单数据中心百万级光模块的运维规模，CoreWeave 提出了由人员、框架、系统与新产品导入（NPI）构成的四大基石，打破通用工程师模式，围绕单一事实源（SSoT）、可观测性与事件驱动自动化构建专才闭环。团队打造了名为 MCP 的中枢 API 框架与定制 CLI 工具，实现从零接触置备（ZTP）到跨站点数千台设备的批量自动化控制。面对高频硬件换代对运维工具链带来的全盘推翻与重塑压力，团队依托深度的供应商共情与极限工程执行力，确保超大规模 AI 集群的高可靠运转。
## Atmosphere & style
演讲洋溢着身处技术最前线、身经百战的实战工程气息，主讲人语言风趣直率，穿插着诸如“学会认命”与“肉体软件”等极客自嘲。在细致描绘凌晨 3 点跨团队排障的窒息重压与庞大光纤拓扑的复杂性时，叙述兼具面对极限工程挑战的紧迫感与纯粹的技术热忱。整场分享从对残酷现实的坦然接纳，逐步推向依托自动化掌控全局的自豪，最终以充满工程师无畏气魄的名言金句收尾，给观众带来硬核、快节奏且极具张力的现场临场感。
## Bulletin
- [01:12](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=72s) AI/ML 基础设施网络要求高达 128,000 块 GPU 并发全线速互联，对单比特误码或端口翻动零容忍，微小网络抖动即可导致运行数周的分布式训练彻底崩溃。
- [06:18](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=378s) 超大规模 AI/ML 后端网络包含超 500,000 根光纤与超 1,000,000 块光模块，传统纯靠人力的“肉机”排障模式在物理上已完全不可行，必须依托全链路自动化系统。
- [14:31](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=871s) CoreWeave 确立了人员、框架、系统与新产品导入（NPI）四大基石，并通过统一 API 中枢“主控程序（MCP）”覆盖交换机从 ZTP 置备到全球并发 CLI 操作的完整生命周期。
- [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) GPU 架构与网络 ASIC 从 100G 向 1.6T 的激进演进迫使运维工具链高频重构，深度供应商协作与“发现问题即刻解决”的主人翁文化是应对 NPI 冲击的核心保障。
## Key takeaways
- [01:12](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=72s) 与传统云计算和企业 IT 允许微小延迟或抖动不同，AI/ML 网络对物理层与逻辑层的任何丢包和误码零容忍，容错空间彻底归零。
- [01:12](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=72s) 跨 128,000 张 GPU 的全线速互联一旦受阻，将导致数周模型训练成果瞬间报废，引发巨额经济损失并对运维团队造成沉重的心智消耗。
- [03:26](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=206s) 支撑全球 100 多个站点的 AI/ML 网络运营依托四大核心基石：人员、框架、系统与新产品导入（NPI）。
- [04:11](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=251s) 系统负责承载规模，但推至性能极限的永远是工程师，AI/ML 极限工况会成倍放大团队协作链路上的任意短板。
- [04:11](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=251s) 技术框架充当人员与系统之间的结构化连接点，为全网络 Fabric 流量分析、流式遥测与单一真实数据源提供标准作业规范与接口。
- [06:18](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=378s) AI/ML 后端网络拥有百万级高速光模块与超 50 万根光纤，体量呈百倍级扩张，迫使运营必须从人工工单转向软件驱动。
- [06:18](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=378s) NPI 宛如“扔向架构的手榴弹”，下一代芯片与架构的推出会瞬间推翻打磨成熟的工具链，倒逼团队高频重构全套系统。
- [10:15](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=615s) 打破传统泛面通才（Generalist）运维模式，围绕四大基石建立垂直专精的领域孤岛，让工程师专注沉淀顶级专业技能。
- [11:32](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=692s) 现代 AI/ML 运维的核心在于基础设施单一真理源（SSoT）、可观测性与事件驱动自动化三位一体的闭环共生。
- [11:32](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=692s) SSoT 必须具备设备级富元数据，向下穿透至光模块序列号、物理跳线拓扑以及精确到分秒的硬件替换生命周期记录。
- [14:31](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=871s) CoreWeave 打造的主控程序 MCP 充当网络中枢，提供统一 API 端点，支持从自动化 ZTP 上电配置到基于 Cumulus CLI 的跨多站点批量并发控制。
- [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) 化解 NPI 挑战需要与硬件供应商建立休戚与共的战友级伙伴关系，确保在新型液冷机柜部署出现故障时能获得第一线的现场支援。
- [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) 网络 ASIC 跨越 100G 到 1.6T 及单轨向多轨/多层架构演进，迫使工程团队频繁重写遥测探针、指标采集与 SSoT Schema 设计。
- [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) 践行“发现问题立即解决”的工程执行力与“迄今从未搞砸过”的工程师底气，构成了 CoreWeave 在极速技术迭代中持续交付的底层文化。
## Detailed walkthrough
### [00:05](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=5s) 自我介绍与主题引入：大规模 AI/ML 网络运营的残酷与魅力
CoreWeave 全球基础设施网络工程副总裁 Jim Julson 开门见山地介绍了自己的背景。他在 CoreWeave 工作了约四年时间，并斩钉截铁地表示：AI 与机器学习（AI/ML）网络与他以往经手过的任何传统网络完全不同。这项工作令人极度兴奋，但也极其艰难、充满挑战，会以最具颠覆性的方式全方位考验工程师的能力。
本次分享的主题围绕大规模 AI/ML 基础设施网络展开。Jim 将整个议程拆解为三个阶段：首先剖析 AI/ML 网络运营中面临的根本性痛点，并以此提炼出应对问题的“蓝图”或四项基石；随后探讨在这些蓝图框架下落地的高效解决方案；最后展望该领域的下一步演进方向。
---
### [01:12](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=72s) 痛点根源：传统网络与 AI/ML 超大规模集群的本质差异
传统网络——无论是企业 IT、超大规模云计算（Hyperscale）还是传统 IaaS——本质上面临的物理与协议约束要少得多。Jim 举了通俗的例子：在传统网络中，用户只想要快速收取一封邮件，或者从 GitHub 下载代码仓库；哪怕网络产生了微小的抖动、延迟增加了 20 毫秒甚至数秒，用户拿到代码后依然能正常工作，性能问题大可留到之后提 PR 时再考虑。
然而在 AI/ML 网络中，这种容错空间彻底归零。网络必须支撑高达 128,000 块 GPU 同时以完全的线速（Line Rate）并发通信，绝不允许任何链路翻动（Flap）、丢包错误或误码率（Bit Error Rate）异常。一旦网络无法维持 100% 的全线速传输，正在运行的大模型分布式训练任务就会受到破坏甚至彻底中断。这意味着跨多个团队数周的心血瞬间付诸东流，不仅给客户和公司带来惨重的直接经济损失，还会带来难以估量的情感消耗与折磨——比如运维人员不得不在凌晨 3 点被紧急叫醒，焦头烂额地排查训练作业为什么性能暴跌或停摆。
- **传统业务场景**：对微小抖动、少量丢包或数十毫秒的额外延迟高度容忍（如拉取代码、收发邮件）。
- **AI/ML 核心诉求**：128,000 张 GPU 并发全线速互联，对单比特误码、端口抖动零容忍。
- **故障外溢效应**：一次网络微小瑕疵即可能导致持续数周的大规模训练作业崩溃，引发巨大的资金亏损与跨团队运维重压。
- **核心结论（TL;DR）**：“没有免费午餐，没有任何容错空间”（There's really no freebies, no forgiveness）。从物理层、逻辑层到软件、人员与流程框架，系统必须时刻保持绝对的一致协同，否则就会以极度壮烈的方式全盘崩溃。
---
### [03:26](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=206s) 应对蓝图：大规模基础设施的四项核心基石
为了应对分布在 100 多个独立物理站点以及覆盖全球骨干网的复杂基础设施，CoreWeave 借鉴建筑学理念，确立了支撑超大规模 AI/ML 运营的四项核心基石（Cornerstones）：人员（People）、框架（Frameworks）、系统（Systems）以及新产品导入（NPI, New Product Integration）。这四者相互嵌套交织，构成高强度运营的底层骨架。
```mermaid
flowchart TD
accTitle: AI/ML 网络运营的四项核心基石
accDescr: 展示人员、框架、系统与新产品导入之间层层支撑与闭环迭代的关系。
subgraph CoreFoundations["运营基石体系"]
P["人员 (People)<br/>组织与工程文化"]
F["框架 (Frameworks)<br/>标准流程与抽象接口"]
S["系统 (Systems)<br/>自动化软件与遥测平台"]
NPI["新产品导入 (NPI)<br/>高频硬件与架构迭代"]
end
P -->|"设计并建立"| F
F -->|"规范与赋能"| S
S -->|"解放人力瓶颈"| P
NPI -->|"持续冲击并重塑"| CoreFoundations
```
---
### [04:11](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=251s) 基石剖析：不可替代的人员与连接两界的框架
在深入技术之前，Jim 强调了一句自嘲但深刻的话：“无论系统多么优秀，如果没有优秀的人才来构建和运营，它们终将走向失败”（No matter how good the systems are, they will fail without great people to build upon and run them）。在 AI/ML 领域，交付和运营链路上的任何一个团队如果未能全功率运转，系统立刻就会在某个不可忽视的角落爆发出致命问题。尽管系统负责规模化，但将系统推至极限的永远是工程师。
而在人员与系统之间充当粘合剂的正是框架。Jim 认为，人的组织管理本身就是最复杂、最难拆解和架构的框架。而在技术侧，团队必须打造高度可插拔、可扩展的软硬件接入框架与标准作业程序（SOP）。无论是提供能够一键查询整套后端网络织构（Backend Fabric）流量特征的 API 端点，还是搭建端到端的数据流式遥测（Streaming Telemetry）平台与单一真实数据源（Source of Truth）的数据富化体系，这些框架定义了工程师与物理网络进行交互的规范与边界。
- **人的重要性**：系统负责支撑规模，但系统是由人来设计、验证和调优的；AI/ML 的严苛环境放大了团队协作链条上的每一个微小断点。
- **框架的核心定位**：作为人员与系统之间的结构化连接点，负责建立可操作的“规范边界”。
- **关键框架举例**：覆盖全 Fabric 的后端流量分析 API、流式遥测订阅管道，以及与单一事实源紧密挂钩的数据补全体系。
---
### [06:18](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=378s) 自动化系统与被戏称为“手榴弹”的新产品导入（NPI）
如果框架是骨架，系统就是真正运转的肌肉。Jim 兴奋地提到，系统是让工程师“内心的极客灵魂得以释放”的地方。没有高度自动化的软件系统，整个运营就会退化为一群“肉机”（Meatware，即纯靠人工排障的人类工程师）每天疲于奔命地登录各种交换机打补丁。在规模较小的场景下人工尚可勉强支撑，但在 AI/ML 领域，运维体量呈现百倍级恶化：
一个传统的数据中心可能仅有 20,000 个光模块和 10,000 根光纤；但在超大规模 AI/ML 集群中，仅后端互联网络就动辄包含超过 500,000 根光纤和 1,000,000 多个高速光模块。面对数以百万计的光器件和物理链路，单靠人力排查光衰、单通或抖动是物理上不可行的，自动化系统因此成为维系集群可用性的生命线。
第四项基石是新产品导入（NPI），被 Jim 戏称为“架构设计与工程团队的手榴弹”。研发团队可能耗费了数月甚至数年时间才把人员组织、SOP 流程、监控系统以及近实时运行的自动化软件打磨得顺畅无比，但只要全新的 GPU 架构或下一代高速网络芯片问世，这个无情的“手榴弹”就会引爆之前的一切假设。团队必须为了抢占市场先机，在看似绝不可能的时间周期内把整套体系从头推倒重构。Jim 笑称做这行确实“脑子得有点毛病才行”，但不断在第一线亲手摆弄最顶尖的技术，正是驱动网络工程师乐此不疲的本质原因。
| 维度 | 传统云计算 / 企业数据中心 | 超大规模 AI/ML 后端网络 |
|:--- |:--- |:--- |
| **光纤规模** | 约 10,000 根 | 500,000 根以上（50 万+） |
| **光模块规模** | 约 20,000 块 | 1,000,000 块以上（百万级） |
| **运维依赖** | 人工工单处理结合脚本 | 必须依靠高度自动化的全流程系统平台 |
| **硬件迭代压力** | 数年稳态生命周期 | NPI 高频推翻重来，追求极限上线时效 |
---
### [09:08](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=548s) 解决方案前瞻与团队阵型构建
在梳理完问题与基石后，演讲切入具体的破局实践。Jim 开玩笑道，面对这些几乎不可能完成的任务，第一步首先是“学会认命与接受”（Acceptance）——他甚至插播了一张滑稽的 PPT 图片，调侃自己每周开会前几乎都要进入这种抓狂或放空的状态。
调侃过后，Jim 开始拆解第一块基石的具体落地方案：人员基石。他明确表示，不会用枯燥的组织架构图来逐个罗列公司里的每一个具体小组，而是要重点剖析如何组织跨职能的技术梯队，让网络运维、系统软件、硬件工程团队在承受极限业务压力时依然能够无缝联动，为后续自动化系统的大规模部署奠定组织保障。
### [10:15](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=615s) 破除“通才”困局：打造高度专精的领域孤岛
CoreWeave 在组织架构上进行了深度剖析与重组，将团队资源紧密围绕运维的四大基石进行战略配置。这种设计的核心目标是在每一个关键技术节点培养出极具深度的领域专家（Subject Matter Experts）。随着各类产品集成在全生命周期内不断推进，由这些垂直专才构成的组织能够反复打磨流程，使庞杂的网络运维如同一台运转平稳的“精密润滑机器”。
```mermaid
flowchart LR
accTitle: 专才型组织架构示意图
accDescr: 展示将通用网络工程师转化为四大基石专属专家的组织架构演进流程。
subgraph Traditional["传统网络运维"]
A["通用型网络工程师<br/>(同时处理琐碎分散任务)"]
end
subgraph Siloed["专精领域孤岛 (四大基石)"]
direction TB
B1["基础设施单一真理源 (SSoT)"]
B2["可观测性与遥测系统"]
B3["事件驱动自动化平台"]
B4["新产品集成 (NPI) 与交付"]
end
Traditional -->|"技能重构与专注沉淀"| Siloed
```
从团队成员的人性角度出发，优秀的工程师往往讨厌枯燥重复，他们渴望尝试新鲜事物、探寻技术盲区。传统的网络工程往往迫使工程师充当“泛面通才”（Generalist），大部分精力被多项繁杂琐事切割。而 CoreWeave 建立紧密技术专精“孤岛”的架构，反倒让工程师彻底跳脱泛泛而谈的维护工作，专注精进某一项顶级技艺。
---
### [11:32](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=692s) 三位一体的协同生态：SSoT、可观测性与事件驱动自动化
在网络运维生态中，常规的基础告警和监控早已算不上前沿，属于几十年前如 Cacti 时代就普及的“起步门槛”（Table stakes）。在现代超大规模 AI/ML 基础设施中，监控运维的真正核心体现在三者的闭环共生：基础设施单一真理源（Infrastructure Source of Truth, SSoT）、可观测性系统（Observability）与事件驱动自动化（Event-Driven Automation）。这三者虽非同物，但在底层高度交织、互为因果。
```mermaid
flowchart TD
accTitle: 三位一体运维生态闭环
accDescr: 展示 SSoT、可观测性以及事件驱动自动化之间的强依赖关系。
subgraph CoreLoop["超大规模网络核心循环"]
SSoT["基础设施单一真理源 (SSoT)<br/>设备全状态/光模块序列号/连线图谱"]
Obs["可观测性平台<br/>遥测 / 日志 / 指标 / 合成探测"]
Auto["事件驱动自动化<br/>自愈 / 调度 / 故障隔离"]
end
SSoT -->|"注入富元数据 & 预期状态"| Obs
Obs -->|"触发异常事件流"| Auto
Auto -->|"同步状态变更与拓扑演进"| SSoT
```
如果 SSoT 缺乏高精度的底层数据和设备级的富元数据（Enriched Metadata），在大规模 AI/ML 场景下根本无法开展任何有意义的事件驱动自动化。仅仅在交换机上标出端口映射是远远不够的，系统必须精确掌握设备处于生命周期的哪一阶段，向下穿透到每一根光模块的物理序列号（Serial Number）、光纤配线架的互联拓扑规范，甚至详细记录某台设备是否在具体某天某分更换了光模块，排查光纤衰减是否为故障元凶。
在 CoreWeave 的部分集群中，仅后端网络就容纳了数千台交换机；若以每台交换机 64 个物理接口来算，单一数据中心的物理连接数便会呈几何级数爆炸，直接导致实时状态查询与状态管理在无自动化介入时面临崩溃。因此，可观测性、遥测、指标、日志及合成探测系统的终极目标，就是为事件驱动自动化提供燃料。在拥有过百个全球站点、单站点连接数达百万级别的版图里，纯靠人力（即所谓的“肉体软件/Meatware”）根本不可能实现横向扩展，软件必须充当一线现场物理运维与网络工程师之间的核心缓冲带。
- **超大后端规模**：部分集群仅后端网络就拥有数千台交换机，单设备 64 个高密物理连接。
- **百万级物理连接**：单一数据中心涉及数百万个物理连接，并伴随全球超过 100 个站点的无序扩张挑战。
- **元数据穿透要求**：必须纳管细微至光模块序列号、光缆互联拓扑及具体分秒的硬件替换追踪记录。
---
### [14:31](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=871s) 系统的“跳动心脏”：主控程序 MCP 与高效 CLI 框架
在大规模网络环境下，软硬件系统与编写这套系统的工程师团队构成了真正的支柱，也是必须倾注巨资的核心。在 CoreWeave 内部，这一中枢系统被称为“主控程序”（Master Control Program, 简称 MCP），其名字源自演讲者所痴迷的科幻电影《电子世界争霸战》（Tron）——以至于团队内部 GitHub 仓库充斥着以 Tron 命名的项目。
MCP 是连接集群构建（Build）与日常运营（Operations）的关键纽带与桥梁。它本质上是一个统一的 API 终结点框架，全面接管网络物理 Fabric 从“零接触配置”（ZTP）到终端交互的每一步生命周期：
```mermaid
flowchart TD
accTitle: MCP (Master Control Program) 生命周期架构
accDescr: 说明 MCP 框架如何从物理设备上电 ZTP 一路支持到工程师执行批量 CLI 操作。
subgraph Hardware["硬件接入阶段"]
P1["新交换机接入物理网络"] --> P2["MCP 渲染目标配置"]
P2 --> P3["自动化零接触置备 (ZTP)"]
end
subgraph Operations["日常运维阶段"]
CLI["工程师本地客户端<br/>(基于 Cumulus CLI 定制开发)"] --> API["MCP 统一 API 端点网关"]
API -->|"并发批量执行指令"| SWs["跨多站点海量交换机 Fabric"]
end
P3 -.-> SWs
```
当交换机初次插电接入时，MCP 负责拉取并渲染配置，通过 ZTP 直接落地到硬件；而在后续运维中，工程师无需逐台登录设备，而是借助基于 Cumulus CLI 深度二次开发的命令行工具，直接在本地输入带参数的指令，便能通过 MCP API 跨越全球多个站点，同时向单台或数千台交换机并发下发复杂或批量的控制指令。MCP 构成了整个网络自动化运营的“跳动心脏”。
---
### [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) 拆解 NPI 引信：深度供应商共情与“发现问题就解决问题”的工程速度
针对前文比喻为“拉开引信的手榴弹”的新产品集成（New Product Integration, NPI），其推进过程极其痛苦，极为考验网络架构师与工程师的耐心和跨组织协作能力。要安全把这枚手榴弹的引信插回，首先离不开与硬件供应商建立休戚与共的深度战略伙伴关系。
这种伙伴关系并非客套的公关辞令，而是实打实的战场共患难——当全新的液冷 Open Rack V3（ORV3）标准机柜部署出现故障、交换机无法上线时，合作伙伴的技术人员必须愿意在凌晨 3 点登机飞往拉斯维加斯数据中心与团队并肩排障。“你需要有人愿意和你一起感受那种痛苦、那种快乐与成功。”NPI 本质上受制于供应链、物流交付与尖端技术的采纳，作为底层硬件技术的消费方而非全栈发明者，失去供应商伙伴的支持将寸步难行。
```mermaid
flowchart TD
accTitle: NPI 迭代与系统重构链条
accDescr: 展示引入新一代硬件技术时对全栈运维工具链带来的全方位重构挑战。
GPU["新型 GPU 接入<br/>(网卡 PCIe/通道重构)"] --> ASIC["新型 ASIC 演进<br/>(100G → 400G → 800G → 1.6T)"]
ASIC --> Arch["拓扑设计全面变更<br/>(Multi-Rail / Multi-Planar / Multi-Tier)"]
Arch --> Retool["全栈运维工具链重构"]
subgraph Retool["全栈运维工具链重构"]
R1["重写遥测与指标采集"]
R2["重塑合成探测探针 (Synthetics/Probes)"]
R3["更新 SSoT 拓扑设计 Schema"]
end
```
支撑 NPI 高效落地的另一支柱是工程迭代速度（Engineering Velocity）。面对产品推向市场的高标准，CoreWeave 遵循“像所有者一样行动”（Act like an owner）的核心价值观，并将其进一步升华为“发现问题，立即解决问题”（Find a problem, fix a problem）的行动纲领。由于硬件演进极其剧烈：
- **网卡通道与 ASIC 速率跃迁**：GPU 换代导致网卡物理通道（Lane）与 GPU 编号的映射关系被全盘改写；网络 ASIC 跨代从 100G、400G、800G 一路激进跨越到 1.6T。
- **架构范式颠覆**：涉及单轨（Single-rail）、多轨（Multi-rail）、多平面（Multi-planar）到多层（Multi-tier）架构的快速切换。
- **全链路重构**：上述变化迫使工程团队对整套运维工具链进行全面重新适配（Retooling），从头修正所有遥测链路、合成探测探针、SSoT 数据结构（Schema）及设计蓝图。
面对 2027 年乃至 2028 年已经完全排满的技术落地日程，技术团队时刻处在极限承压状态。对于这种不断引入疯狂新架构、迈入“彻底的 YOLO 模式”的高压节奏，CTO Peter Salanki 的回应成为了团队内部流传的名言金句：
> “我们到现在可还没搞砸失败过呢。你凭什么觉得我们现在就会开始失败？”（"Well, we haven't failed yet. What makes you think we're going to start now?"）
这种毫无畏惧的工程师底气，正是 CoreWeave 驾驭超大规模 AI/ML 网络运营的真实底层精神。
## Notable quotes
> [01:12](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=72s) There's really no freebies, no forgiveness — Jim Julson
> [04:11](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=251s) No matter how good the systems are, they will fail without great people to build upon and run them — Jim Julson
> [06:18](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=378s) 架构设计与工程团队的手榴弹 — Jim Julson
> [06:18](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=378s) 脑子得有点毛病才行 — Jim Julson
> [09:08](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=548s) 学会认命与接受 — Jim Julson
> [14:31](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=871s) 跳动心脏 — Jim Julson
> [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) 你需要有人愿意和你一起感受那种痛苦、那种快乐与成功 — Jim Julson
> [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) 像所有者一样行动 — Jim Julson
> [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) 发现问题，立即解决问题 — Jim Julson
> [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) 彻底的 YOLO 模式 — Jim Julson
> [16:07](https://www.youtube.com/watch?v=YbXbdwB0XHI&t=967s) Well, we haven't failed yet. What makes you think we're going to start now? — Peter Salanki
## Mentioned
Jim Julson, CoreWeave, @Scale, Peter Salanki, GitHub, Cacti, Master Control Program (MCP), Tron, Cumulus CLI, Open Rack V3 (ORV3)
## Video description
As machine learning continues to grow in scale, so too do the challenges of operating these mission critical networks.
Join Jim Julson, from Coreweave, in a journey through the operational challenges, and nuances found in Machine Learning and AI at scale around Infrastructure Networks.
Learn more about the @Scale conference here: https://atscaleconference.com/
[Source](https://www.youtube.com/watch?v=YbXbdwB0XHI)
Source quotes
[1:12] Jim Julson: There's really no freebies, no forgiveness
[4:11] Jim Julson: No matter how good the systems are, they will fail without great people to build upon and run them
[6:18] Jim Julson: 架构设计与工程团队的手榴弹
[6:18] Jim Julson: 脑子得有点毛病才行
[9:08] Jim Julson: 学会认命与接受
[14:31] Jim Julson: 跳动心脏
[16:07] Jim Julson: 你需要有人愿意和你一起感受那种痛苦、那种快乐与成功
[16:07] Jim Julson: 像所有者一样行动
[16:07] Jim Julson: 发现问题，立即解决问题
[16:07] Jim Julson: 彻底的 YOLO 模式
[16:07] Peter Salanki: Well, we haven't failed yet. What makes you think we're going to start now?
Source quotes
[3:26] Jim Julson: The four operational cornerstones: People: Specialized domain experts who design, build, and run the infrastructure. Frameworks: Extensible glue code, APIs, and procedures connecting people to systems. Systems: Software platforms and tooling that replace manual human intervention ("meatware"). New Product Integration (NPI): The process of onboarding next-generation hardware and architectures rapidly.
[7:38] Jim Julson: NPI forces engineering teams to reset and do it all over again for next-generation technology on aggressive time-to-market schedules.
[11:32] Jim Julson: Observability platforms exist primarily to feed event-driven automation engines, allowing software to act as the buffer between physical data center technicians and network engineers.
[16:52] Jim Julson: True partnership requires vendors who are fully committed to sharing both the pain and the success of scaling new systems.
[17:46] act like an owner
[17:46] find a problem, fix a problem
[19:06] full YOLO mode
[19:06] Peter Salanki: Well, we haven't failed yet. What makes you think we're going to start now?
Original transcript
[0:05] Hello and welcome. My name is Jim
[0:07] Jolson. I'm the vice president of
[0:08] engineering global infrastructure
[0:10] networking with Coree. And today I want
[0:13] to take you on a journey through AI and
[0:16] ML at scale network operations and what
[0:19] it means today to operate at that level.
[0:24] Little bit of background. I've been with
[0:25] Cororeweave now for about four years and
[0:28] I can tell you definitively that AI and
[0:30] ML is nothing like anything I've ever
[0:32] worked on before. It's exciting. It's
[0:35] very hard, very challenging, and it
[0:38] really puts you to the test, but in some
[0:41] of the most awesome ways you can
[0:43] imagine. we're going to go through today
[0:45] and we're going to talk about some of
[0:46] the problem statements associated with
[0:48] AI and ML network operations and how we
[0:52] build blueprints around how to approach
[0:54] those problems. We're then going to talk
[0:56] about some of the cool solutions to some
[0:58] of those problems and how we work within
[1:00] those blueprints to be successful. Then
[1:04] we're going to talk about what's next.
[1:07] So let's dive in. Here's some of the
[1:09] problem statements.
[1:12] AI and ML is fundamentally different
[1:15] from traditional networking. Be it IT
[1:17] enterprise, be it at scale, hypers
[1:19] scale, cloud. Networking traditionally
[1:24] has a lot less constraints than AI and
[1:26] ML. Some of those constraints you can
[1:28] break down into the notion of like,
[1:29] okay, I need my email really quickly.
[1:31] That's a traditional network. Whereas an
[1:33] a IML network, I need to talk to 128,000
[1:37] other GPUs simultaneously at line rate
[1:40] speed. and I can't have a flap, an
[1:42] error, a bit error rate problem,
[1:44] nothing. Additionally, you can look at
[1:47] this like I need to go and I need to
[1:49] download a GitHub repo. You don't care
[1:51] that it takes you an extra 20
[1:52] milliseconds. You don't care that there
[1:54] was a little jitter in the packet. You
[1:56] don't care that it might have even taken
[1:58] seconds. Once you get your repo, you're
[2:00] off, you're working, and it's a problem
[2:01] for when you have to put up a PR later
[2:04] on as to whether or not it's performant.
[2:06] AI and ML, it's quite a different story.
[2:09] You have to make sure that you're
[2:10] operating at line rate 100% of the time,
[2:14] every step of the way. And when you
[2:15] don't, you sacrifice the integrity of
[2:18] the of the training job in question. You
[2:20] stop the training job. It could stop
[2:22] weeks of collective work across numerous
[2:24] teams. The customer can have financial
[2:26] impact. It has financial impact to the
[2:29] company. And then there's the
[2:30] immeasurable toll, the emotional toll,
[2:32] the annoyance, the frustration that we
[2:35] all inevitably deal with. getting woken
[2:37] up at 3:00 in the morning because the
[2:38] jobs aren't performant enough because
[2:40] they're not running properly. These are
[2:42] all problems that you highly associate
[2:44] with AI and ML workloads as compared to
[2:48] traditional cloud.
[2:51] So go too long, didn't read. The TLDDR
[2:55] in traditional IAS, your physical layer,
[2:57] your logical layer, your people, your
[3:00] frameworks, your software, all of it has
[3:02] to work in unison all of the time or
[3:05] else it fails spectacularly. That is
[3:07] probably the easiest way to say it.
[3:09] There's really no no freebies, no
[3:11] forgiveness when it comes to the
[3:13] integrity, performance, and capacity of
[3:16] an AI and ML mail. Even the little
[3:19] things can hurt you and the little
[3:22] things all add up to big things. So, one
[3:26] of the ways that we look at this is we
[3:28] break this down. If you're using
[3:29] construction terminology, right, the
[3:31] blueprints or the cornerstones, if you
[3:33] will, for us is we've looked at things
[3:36] and said, "Okay, so how do we operate at
[3:38] Coreweave? What do we look at things
[3:39] like what's important? How do we
[3:41] approach these big problems? a network
[3:43] that are distributed across a hundred
[3:45] different unique locations and a a
[3:48] global backbone, a global footprint,
[3:51] right? And where we arrive at is four
[3:53] key areas that we focus on and those are
[3:56] your people, those are your frameworks,
[3:59] your systems, and then of course NPI or
[4:02] new product integration. And we're going
[4:04] to talk a little bit about the people
[4:06] side and the frameworks and the systems
[4:07] and how all of those interop together.
[4:11] So, take this with a grain of salt. This
[4:13] quotes from me. I'm nobody, but I
[4:16] thought it was true, so I'm going to say
[4:17] it. No matter how good the systems are,
[4:21] they will fail without great people to
[4:24] build upon and run them. I know it's a
[4:26] little cliche and you can say that of
[4:29] any company, the success always rolls
[4:31] back to the people. It's true 100%. But
[4:33] I can tell you that in AI and ML, if
[4:36] every team involved in the chain of
[4:39] delivery and operations isn't firing on
[4:42] all cylinders, you're going to feel it
[4:44] more there than you are in any other
[4:46] type of network because there is no room
[4:48] as we stated prior for any sort of issue
[4:52] in the long run. It has to be perfect
[4:54] every single time, right? So your
[4:56] systems can scale, but it's really the
[4:59] people that are going to take those
[5:00] systems to get to that point, right? So
[5:03] the people are the ones that build the
[5:04] frameworks. The people are the ones that
[5:06] design the systems. It's the people that
[5:08] you really have to heavily invest in.
[5:11] So let's talk about the next thing. It's
[5:13] the frameworks, right? Well, frameworks
[5:15] are for the people and they're for the
[5:17] systems and they are kind of that glue
[5:19] between those two worlds, if you will.
[5:22] So I think people are by far the most
[5:24] complex framework that there is. It's
[5:26] the hardest problem to dissect and and
[5:28] architect. The people problem, right?
[5:30] The systems though, the systems in
[5:32] place, the things that you do to wrap
[5:34] sanity and build the box in which you
[5:37] operate in are critical. The software
[5:40] that you choose to to deploy or write
[5:42] from scratch, that has to become a
[5:44] pluggable extensible framework. And we
[5:46] we really invest heavily in making sure
[5:49] that we build these frameworks so that
[5:51] people can consume them in a meaningful
[5:53] way. It could be an API endpoint that
[5:56] allows you to query the entirety of a
[5:59] given back-end fabric for for a traffic
[6:02] profile. It can be the telemetry, the
[6:04] streaming telemetry platform that you
[6:06] subscribe to and the data enrichment in
[6:09] your source of truth. All of these
[6:10] frameworks matter as it relates to how
[6:14] you interop with the physical network.
[6:18] So next we're going to talk a little bit
[6:20] about the systems. So the systems, this
[6:22] is this is where I get the most excited,
[6:24] right? This is the fun stuff. This is my
[6:26] inner nerd gets to come out and we have
[6:28] some fun, right? So the systems are the
[6:30] thing that you plug into the frameworks
[6:32] for the people to consume. Without the
[6:35] systems, you just have a bunch of
[6:38] meatware aka humans running around
[6:40] solving problems, logging into devices,
[6:43] and fixing things, and it it's just not
[6:45] going to scale. That's not new. That's
[6:47] been true of IT enterprise. It's been
[6:49] true of hyper cloud, hypers scale. It's
[6:51] been true of IAS, SAS, it doesn't
[6:54] matter. You can't scale humans. We know
[6:56] that. However, in AI and ML, that
[6:58] problem is exacerbated a hundfold
[7:01] because you go from data centers where
[7:02] you have 20,000 optics and 10,000
[7:05] fibers, and then you go to an a IML
[7:08] workload and now you're looking at half
[7:10] a million fibers and you're looking at
[7:13] over a million optics in some cases just
[7:15] for the backend network alone. You
[7:17] cannot rely solely on people and process
[7:21] to remediate the problems that go with
[7:23] operating in that sca at that size and
[7:24] at that scale. And that's why the
[7:26] systems must be so important and so much
[7:29] of a priority in how you structure
[7:32] everything and delivering a good
[7:34] operational platform.
[7:38] Okay, the final piece of it NPI. We like
[7:41] to call this one the architectural
[7:43] engineering and design hand grenade.
[7:45] Why? Why is it so bad, right? NPI, new
[7:48] product integration. That's cool. It's
[7:49] fun to work on new stuff. Yes, 100%.
[7:53] However, take everything we just talked
[7:55] about. You just spent months building
[7:58] the people, the teams, the structure.
[8:00] You just spent months, maybe even
[8:02] quarters and years building out the
[8:04] frameworks and the SOP, your standard
[8:07] operating procedure, your process, all
[8:09] wrapped around this idea that this
[8:11] platform behaves this way, right? And
[8:13] then you spend a ton more time iterating
[8:16] on your software to get it to be able to
[8:19] do what you need near real time to
[8:21] support the intensity of these
[8:23] workloads. Now you get to go do it all
[8:25] over again. And now you got to be first
[8:27] to market. And now you got to go do it
[8:29] in a timeline. It's not going to be
[8:31] realistic. You just accept that.
[8:33] However, it's going to be fun because
[8:35] you get to go iterate on these systems,
[8:37] iterate on the people, iterate on the
[8:39] frameworks, and you get the end result
[8:42] is you're touching new state-of-the-art
[8:43] technology over and over and over again.
[8:46] Is it hard? 100% yes. Is it worth it?
[8:51] Absolutely. I mean, you got to be a
[8:53] little sick in the head, but you know,
[8:55] that's we're engineers. That's kind of
[8:56] how we we're wired that way. So, the NPI
[9:00] piece is is really key for us. So we we
[9:01] we collect that in that cornerstone
[9:04] model, right? Those four corner stellas.
[9:08] Okay, we've talked about kind of
[9:10] identifying the problems. We've talked
[9:12] about sort of like what the problem
[9:14] statement looks like and kind of dove
[9:16] into what each of those key cornerstones
[9:18] are that surround the problem
[9:20] statements, like how we're tackling
[9:21] them. Now, we're going to talk about how
[9:23] we solve them within those cornerstones.
[9:25] I'm going to caveat here. We're on
[9:26] limited time, right? So, I'm not going
[9:29] super super deep into these, and it's
[9:31] worth noting that we could go way
[9:33] deeper. I understand that. We'll have to
[9:35] we'll have to figure something out to to
[9:37] get a little bit deeper afterwards.
[9:39] Oh, well, this is step one. You just
[9:42] have it's acceptance. You just I spend
[9:44] probably once a week in this mode before
[9:46] my meetings. And no, I'm just kidding. I
[9:48] don't even know this guy. I clearly have
[9:50] hair shaved by choice. I I like Dove.
[9:53] That's not Dove soap. I Whatever. Just
[9:56] get that part out of the way. It's
[9:57] mandatory.
[9:58] Okay. People foundation. Now, I didn't
[10:02] list every single team in our
[10:03] organization here to to build you an org
[10:05] chart. That's not exciting. Again, I
[10:08] recognize the people talk isn't the most
[10:09] sexy part of what we do. However, being
[10:12] is critical as it is. It's important to
[10:15] call out kind of some of these
[10:16] constructs and why we have them, right?
[10:18] What you'll notice is we've really
[10:19] dissected our organization and put great
[10:22] emphasis on making sure that we're set
[10:24] up organizationally to tackle each one
[10:27] of these key critical areas that we've
[10:30] identified in our four cornerstones.
[10:32] The idea is that you build subject
[10:34] matter expertise inside each one of
[10:38] these key areas so that as you're
[10:40] executing throughout the life cycle of
[10:42] every one of these product integrations,
[10:44] you have people that can tune this and
[10:46] turn it into a welloiled machine over
[10:48] and over and over again. And then more
[10:51] on the the human side of it, right? At
[10:52] the end of the day, like engineers want
[10:54] options. Engineers get bored. Engineers
[10:57] want to move on. They want to try new
[10:58] things. they they have an itch that
[11:00] needs to be scratched, if you will. And
[11:02] I think this this type of a structure
[11:04] where you build really tight silos of
[11:07] people that have strong subject matter
[11:09] expertise lets people get out of that
[11:11] generalist role and really hone a craft.
[11:14] Whereas traditional networking, you
[11:16] really don't get to do that a lot. Most
[11:18] generalists will tell you like that are
[11:19] network engineers that they they'll
[11:21] spend a lot of time doing a number of
[11:24] things versus a lot of time doing one or
[11:27] two things, right? just because that's
[11:28] the nature of what it takes to operate
[11:30] that way.
[11:32] Okay, so talking about the frameworks,
[11:34] this is kind of the cool stuff right
[11:36] here, right? Within the ecosystem that
[11:38] is operations and network operations,
[11:40] there's what you'd call table stakes,
[11:42] right? It's your given your the things
[11:43] that always come with it. Well, we
[11:45] already know that alerting and
[11:47] monitoring, I mean, hello, that's old
[11:49] news. Like, we've been doing this for
[11:51] decades at this point, going all the way
[11:53] back to cacti. Okay, I'm showing my age.
[11:55] Let's let's leave that well known. But
[11:57] no, from an observability standpoint,
[11:58] right, we look at observability,
[12:00] event-driven automation, and then
[12:02] infrastructure sources of truth. These
[12:05] are all three one and the same thing.
[12:07] Now, hear me out. It's not that
[12:10] infrastructure sources of truth is the
[12:11] same thing as observability. But within
[12:14] the ecosystem, they are all heavily
[12:16] heavily reliant on one another, right?
[12:18] your infrastructure source of truth. If
[12:20] it does not have good data and then
[12:23] enriched metadata on a per device level,
[12:26] you can't do your event proven
[12:27] automation in a meaningful way at the
[12:29] scales required for AI and ML. You just
[12:32] won't be able to. It's not enough to
[12:34] have a switch with the ports mapped out
[12:37] and only the ports and not be able to
[12:39] set state to understand where a device
[12:41] is at in its operational life cycle. You
[12:43] have to be able to get deeper than that
[12:45] down to the serial number of an optic,
[12:47] the interconnected cable plan into cable
[12:50] plants. What that looks like on a per
[12:53] device basis to be able to chase all
[12:54] that through. You have to be able to
[12:56] understand did this device get this
[12:58] optic swapped on this day at this time?
[13:00] Was the fiber the culprit? There's all
[13:03] this data that you have to manage on a
[13:05] regular basis just in order to
[13:07] understand what the heck is happening at
[13:08] any given day. We have some builds just
[13:10] as an example that measure in the
[13:14] many thousands of network switches just
[13:17] on the back end alone. You can imagine
[13:20] it at 64 physical connections per
[13:24] switch. You get up there pretty quickly.
[13:27] So much so that managing that just in
[13:28] one site is almost untenable when you're
[13:31] doing real time query and real-time
[13:33] state. So you're you have to have robust
[13:35] systems in place to support a lot of
[13:37] this from the observability side. Right?
[13:39] We're going to jump over to the left
[13:41] here. You look at what do you what's the
[13:43] observability used for? How are you
[13:44] using it? A lot of what we do is all
[13:48] because of event- driven automation,
[13:50] right? Our observability, telemetry,
[13:52] logging, metrics, and and synthetics
[13:54] platforms exist as a function to serve
[13:57] the event driven automation. Again,
[13:59] going back to its meatware. We don't
[14:01] have meatware humans, right? Software
[14:03] has to be the thing that sits in between
[14:06] the physical boots on the ground and the
[14:08] network engineer. You cannot expect to
[14:11] manage literally millions of physical
[14:13] connections in a single DC with sprawl
[14:16] of over a 100 sites globally and do it
[14:19] in a meaningful way. It simply does not
[14:21] scale. And so we really put that
[14:23] emphasis on the observability platform
[14:25] to drive what we do from an eventing
[14:27] automation standpoint,
[14:31] systems and tooling. So this is again
[14:33] more of our beating heart. You're seeing
[14:35] the theme is I point back to our
[14:37] systems, our software and the people
[14:39] that write them. This is a paramount
[14:42] mindset to have operating at this scale
[14:44] is investing heavily heavily heavily
[14:47] into the systems and tooling within the
[14:49] ecosystem. One of ours is what we call
[14:51] our master control program. Right? I am
[14:53] a huge Tron fan. My team is probably
[14:56] super sick of it because I absolutely
[14:58] ask that they name everything after
[15:00] Tron. They're running out of names. I
[15:03] think we have a not insignificant amount
[15:06] of repos out there in GitHub that are
[15:09] Tronbased and and I'm starting to lose
[15:11] track myself of even what the names are.
[15:13] But anyways, MCP is our master control
[15:16] program, right? It's the glue, the
[15:18] bridge between our build and our
[15:20] operations that allows us to interface.
[15:23] It's a framework, right? It's an API
[15:26] endpoint that allows us to interface
[15:28] with our physical network fabric at
[15:30] every step of the process all the way
[15:32] from the time that that switch is first
[15:34] plugged in and it goes out and we render
[15:37] that configuration and we ZTP it on a
[15:39] box up to the point where an engineer is
[15:42] logging in via the Cumulus CLI that
[15:45] we've custom developed that allows them
[15:47] to run CLI commands from their machine
[15:50] and interface with one or many switches
[15:52] at any number of sites simultaneously.
[15:54] ly to run complex or simple commands in
[15:58] bulk all via simple CLI commands with
[16:00] parameters. That framework is what
[16:03] allows us to do that. It's truly our
[16:05] beating heart.
[16:07] Okay, NPI. We're going to put the pin
[16:10] back in that grenade that we talked
[16:12] about earlier, right? NPI, a new product
[16:15] integration. How do we do that? How do
[16:17] you make NPI not painful? Well, look,
[16:19] I'm not going to sit here and lie. NPI
[16:22] is hard. It it takes a special person
[16:25] with with a lot of patience and
[16:26] understanding to understand like how to
[16:28] work and navigate through the NPI world
[16:30] as a network architect and engineer
[16:32] because it takes takes a village to do
[16:35] it and it takes a lot of collaboration.
[16:38] One of the ways that we you know remove
[16:40] roadblocks is we make sure that we have
[16:43] partnerships with our vendors. I know
[16:45] you're rolling your eyes like it's a
[16:47] little cliche. It's true. You have to
[16:50] have these partnerships. You can't be
[16:52] successful if you don't have that
[16:54] partner that's willing to hop on a plane
[16:56] at three o'clock in the morning and fly
[16:57] to Las Vegas to your data center to
[17:00] troubleshoot why your switch isn't
[17:01] coming up in a brand new OV3 compliant
[17:04] rack that's liquid cooled. Right? You
[17:07] you need people willing to feel that
[17:09] pain with you and that pleasure and that
[17:12] success so that you too can be
[17:14] successful and that will keep the
[17:16] relationship moving forward. NPI is all
[17:19] about supply chain, logistics,
[17:22] introducing new technology, and often
[17:24] we're consumers of it. We're not
[17:26] creating a lot of this technology, at
[17:27] least not all the time. And so, it's
[17:29] paramount to have that relationship in
[17:31] place. And and that's why it gets that's
[17:33] why it gets space on the slide deck,
[17:35] right? Because it's big enough to where
[17:36] when you don't have partners, we felt
[17:38] what that's like missing that partner
[17:40] and that partnership. And it's it's
[17:43] something that everybody needs to be
[17:44] attuned to. And then finally,
[17:46] engineering velocity. So NPI is all
[17:49] about speed and efficiency, right? Being
[17:52] able to get that product out the door,
[17:54] but with the high level of standards
[17:56] that any company really should have. But
[17:58] Core Weave, we absolutely pride
[17:59] ourselves in our rock solid stability
[18:01] and our product launches, right? And so
[18:04] the the core values that we have at Core
[18:07] Weave, one of them is actually act like
[18:09] an owner. And I love that myself because
[18:12] it really can apply to a multitude of
[18:14] things. But I like to take it a step
[18:16] further and call out find a problem, fix
[18:18] a problem. With NPI, as you're going
[18:21] through these new products, you
[18:22] introduce a new GPU. It now has a
[18:25] different lane configuration on the
[18:26] nick, right? Because it has to map to
[18:28] different GPU numbers. You now have a
[18:30] new ASIC. You went from 100 to 400, 400
[18:33] to 800, 800 to 1.6 terra. do multi-rail
[18:38] versus single rail versus multiplaner
[18:40] versus multi-ter. All these different
[18:42] design considerations are the result of
[18:45] this work in NPI. You have to go through
[18:48] and retool everything. You have to go
[18:50] through and fix all your telemetry, your
[18:52] synthetics, your probes. You have to fix
[18:54] your sources of truth, your design
[18:56] schema. It just goes on and on and on.
[18:58] And so having that velocity, having
[19:01] people that find problems and fix
[19:02] problems is absolutely key.
[19:06] Okay, so what's next? Okay, I might have
[19:08] misled you. I don't have a ton next.
[19:11] 2027 is full. You saw me crying in the
[19:13] shower eating a bar of soap. 2028's
[19:16] full. So, I'll just leave you with this.
[19:18] My CTO said something one time that it
[19:20] it's kind of become a joke, right? So,
[19:22] Peter Seni,
[19:24] you know, every time you introduce
[19:25] something crazy or insane, it's full
[19:27] yolo mode, right? His response is
[19:29] simple. Well, we haven't failed yet.
[19:31] What makes you think we're going to
[19:32] start now? That's the attitude we take
[19:34] around network operations at scale for
[19:37] AI and ML and that's the way we like to
[19:39] roll the corweave. Thank you very much
[19:40] for your time and have a great day.
