CoreWeave Bars AI Agents From Network Writes to Protect GPU Clusters

CoreWeave禁止AI Agent执行网络写入以保护GPU集群 闲置硬件的高昂资本成本使得自动化运维失误难以承受，迫使CoreWeave将AI工具限制在只读诊断范围，与此同时，激增的算力需求将网络生命周期缩短至三年以下。 Jim Johnson详细介绍了CoreWeave的运维演进历程，即从混乱的手动部署转向由其MCP引导框架和定制“Echo”可观测性栈驱动的软件定义、API驱动型网络编排。对话探讨了AI集群巨大的资本投入与极严格的容错空间，并将限制自主AI Agent仅进行只读操作的审慎策略，与Claude辅助NPI容量规划等实际落地应用进行了对比。两位领导者还直面了持续无休止扩张带来的人员损耗，强调通过结构化休整来保护核心工程“英雄”、践行“像所有者一样思考（Act like an owner）”的企业文化价值，以及精准定义网络健康度SLO所需的多年复杂探索。 在基础设施快速扩张期过度依赖高产出“英雄”工程师容易导致严重的职业倦怠；保障团队可持续运营需要主动分散工作负载、推行强制休息，并在整个技术架构中落实分布式的主人翁意识。 (5:00) CoreWeave 将网络自动化清晰划分为两部分：通过主控程序（MCP）统一编排新站点的物理引导，并在投入运行后利用事件驱动机制自动隔离降级链路，避免分布式训练任务中断。 (14:40) AI 基础设施需求引发了物理机房规模的急剧演进，CoreWeave 的部署模式已从最初 90 天交付 5MW 的小型机房，跃升至并发建设单体达 200MW 至 250MW 的超大规模算力园区。 (17:10) 出于对高风险操作的审慎把控，CoreWeave 目前对授予 Agentic AI 自主写权限持保守态度，自研 Agent 仍严格限制于只读诊断，以此防范生产网络故障。 (22:32)

CoreWeave全球网络基础设施副总裁Jim Julson与Meta生产工程总监Francois Richard指出，在AI算力需求与集群规模呈指数级膨胀的背景下，基础设施团队必须通过全栈所有权文化、软件驱动的确定性自动化与严苛的SLO量化体系，来应对前所未有的工程交付速度与极高的资本闲置成本。 组织文化与工程人员的防枯竭管理 (0:00) 在动辄数百人、面对7天16小时高压环境的团队中，过度依赖少数“救火英雄”会迅速引发倦怠。Julson指出，面对此类过载，单纯提升薪酬无法替代基本的生活质量。因此，CoreWeave在赞扬高产贡献者的同时，要求管理层主动干预，强制实施不走常规流程的休假机制，并在团队内部全面推行“发现问题、解决问题”和“像所有者一样行动”（Act like an owner）的组织文化。Richard则介绍了Meta的轮换机制，即在高强度发布周期后将骨干调回常规事务以休整。两人一致认为，唯有把所有权分散到各技术栈层级，避免高级骨干陷入无尽的跨组灭火，团队才能保留推动前沿技术创新的带宽。 从交付到运行的双轨网络自动化架构 (12:17) 传统依赖手动编写Jinja模板与直连设备调试的网络运维模式已被废弃。CoreWeave全面采用软件工程范式，通过专用软件工程师搭建生产API后端与前端界面。在自动化体系中，公司将业务清晰划分为两个阶段：一是采用内部称为MCP（Master Control Program）的程序化框架进行新数据中心从零到一的启动交付，负责管理物理布线校验（如核查多达25.6万条链路的拓扑一致性）、配置漂移检测与消缺清单；二是上线后的事件驱动型自愈架构，专门监控包括光衰、端口抖动和BGP会话震荡等指标。例如在支撑7000多节点的RoCE（RDMA over Converged Ethernet）后端网络中，自动化系统能在数十条链路发生翻动时优雅排空流量与控制平面，确保分布式AI训练任务在不发生检查点报错的情况下平稳运行。 物理算力与骨干网周期的跨越式重构 (17:10) AI训练集群高昂的资本支出使得网络中断与硬件闲置成本极高，平均检测时间（MTTD）与平均修复时间（MTTR）已上升为董事会关注的关键指标。Julson回顾了2023年初团队在90天内手工搭建首批5个5MW数据中心的极端经历，而当前公司已转向同时建设3至4个规模达200MW至250MW的超大规模园区。伴随物理规模的剧增，骨干网生命周期经历了前所未有的压缩：过去设计寿命为5至7年的骨干网络在2.5至3年内便面临淘汰，迫使团队在租约与场地合同敲定前便提前数月预订暗光纤与传输波长。在外部拓扑上，CoreWeave将其解构为连接数据中心的全球东西向传输骨干网，以及超过40个部署在电信中立机房、专职处理公网汇聚、区域对等互联和客户直连的边缘Transit POP节点。 风险防范：只读AI代理与人机双重审计 (22:32) 尽管CoreWeave员工在短短6个月内从3000人扩充至4000人，工程团队在引入AI代理执行网络变更时仍保持审慎克制。Julson明确表示，虽然内部使用自研代理进行只读排查、告警比对与跨系统信息关联，但在生产网络中依然全面禁止自主AI执行写入或路由排空等写操作。Richard补充了Meta在应对AI代理介入基础设施时的治理框架：系统必须实现“双重身份”跟踪，即同时记录触发操作的人类证书凭据与执行代理的服务身份，并配合“意图日志”完整存档人类的原始提示词、代理的执行上下文和环境假设，以便在线上事故排查中还原因果链。 消除指标模糊性与建立SLO全景量化 (39:40) 在早期运维中，不同网络子团队对网络“健康”状态的定义往往互有矛盾。为此，CoreWeave重构了跨后端网络、管理网络、骨干网和服务层的SLO（服务等级目标）与SLI（服务等级指标）体系，涵盖可用性、可靠性与容量等核心维度，并将内部SLO评审参与人数从最初的10余人提升至约40人。Richard指出，Meta耗时约三年才建立起能够真实映射生产状态的指标体系，并强调即便在静默故障（如冗余丢失但未导致业务中断）场景下，SLO也必须产生可见回落，以防监控系统失真。同时，CoreWeave在生产事故（内部称为“incies”）管理上坚持由Slack与Jira驱动工单自动化，但将根本原因分析（RCA）与事故定级完全保留给人类工程师把控。 无论是CoreWeave的超大规模GPU云，还是Meta的全球在线系统，AI基础设施的扩展正将传统网络设计推入没有先验经验可循的深水区。当前团队通过限制AI写权限、重构短周期骨干网与强化跨层级SLO来稳固可用性，但随着NVIDIA下一代架构等更高密度硬件的引入，如何在不断缩短的技术迭代周期与极端交付目标下维持组织弹性与系统可靠性，仍是全行业悬而未决的挑战。

## Background 本期节目由 YouTube 频道 @Scale 于 2026 年 9 月 18 日发布。视频由 Meta 生产工程总监 Francois Richard 对话 CoreWeave 全球基础设施网络副总裁 Jim Julson，深入探讨了在 AI 算力狂潮下支撑前沿模型训练所需的基础设施建设、组织扩展以及工程运维挑战。 ## TL;DR 本次对话聚焦于超大规模 AI/ML 网络基础设施的演进历程与工程哲学。Jim Julson 详述了 CoreWeave 如何以 API 优先架构与自研 MCP 软件栈取代传统配置模板，并在数千节点的 RoCE 织网中实现事件驱动的无损故障隔离与容量建模。针对团队管理，两位技术领袖深入剖析了如何打破依赖少数“工程英雄”的救火恶性循环，通过建立全员“极致所有权”文化和成熟的 SLO 体系来守护核心人才。此外，双方还辩证审视了将 AI Agent 引入生产网络的高昂风险，并揭示了全球骨干网架构在算力需求暴增下面临的极限迭代挑战。 ## Atmosphere & style 对话整体展现出极其硬核、务实且充满同行深层共鸣的探讨氛围，两位深谙底层系统复杂度的工程领导者如战友般开怀畅聊。交流中完全剥离了浮夸的概念包装，既有 Meta“风暴计划”主动下线区域演练的幽默自嘲，也有 CoreWeave 狂暴交付背后对工程师身心倦怠的真实关切与战壕体感。整场对谈既流淌着超大规模基建命悬一线的敬畏与紧迫，又穿插着极客亲手搭建自动化家庭影院的纯粹生活情趣，兼具前沿技术的厚重感与浓郁的人文温度。 ## Bulletin - [17:10](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1030s) CoreWeave 的数据中心单体建设已从 2023 年初的 5MW 飞跃至并发交付 3 到 4 座 200MW 至 250MW 的超大型集群，超快演进导致两三年前搭建的骨干网架构已不堪重负，被迫全面推倒重构。 - [12:17](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=737s) 在网络运维中，CoreWeave 坚决弃用脆弱的 Jinja 模板，通过自研分布式软件栈 MCP 与 API 优先框架，在包含 7,000 个节点的 RoCE 网络中实现了链路抖动时的无损自动排空（Graceful Drain）。 - [22:32](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1352s) 面对 AI 智能体进驻生产网的高风险，CoreWeave 严格限制其仅可执行只读排障操作；Meta 则针对智能体变更探索了包含双重身份穿透与推断上下文审计的“意图日志记录”（Intent Logging）机制。 - [26:40](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1600s) CoreWeave 工程师利用 Claude 配合 Google Sheets 构建了网络容量建模计算器，可根据特定 GPU 规模精准反推 T0/T1/T2 各级交换矩阵需求，目前已被直接用于英伟达下一代 Vera Rubin 架构规划中。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=0s) 在狂暴扩张的 AI 算力竞赛中，高负荷运转的顶级技术人才面临严重倦怠，单靠薪酬激励早已失效，团队管理核心在于分担压力并培育全栈责任感（Ownership）。 - [01:12](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=72s) Meta 每季度主动切断一个数据中心 Region 执行“风暴计划”演练容灾；CoreWeave 则主张公开赞赏高产出的“工程英雄”，但需建立制度避免他们成为自身成功的受害者。 - [04:34](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=274s) 破除高级工程师持续救火恶性循环的关键，除了强制执行彻底关机的小长假，更在于从根源治理普通成员的“所有权”缺失，防止故障无休止反弹给核心架构师。 - [07:13](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=433s) GPU 集群的天文数字成本使得停机时间具备极高的商业敏感度，基础设施主管必须具备“敢于说不”的决断力，在繁重会议洪流中守住深入核心技术难题的精力空间。 - [10:05](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=605s) 技术管理者下场参与生产 Debug 不仅能第一视角检验内部工具链的有效性，还能确保制定的 NPI（新产品导入）交付时间表具备底层现实可行性。 - [12:17](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=737s) 针对海量异构站点的管理痛点，CoreWeave 坚决摒弃脆弱的 Jinja 模板脚本，通过招募专职软件开发人员构建 API 优先系统与事件驱动框架。 - [13:58](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=838s) 将基建交付与在网运维明确解耦：自研 MCP 分布式平台负责冷启动阶段数十万条物理链路的质量验证，运营期系统则实时捕捉 RoCE 网络异常并执行优雅排空自愈。 - [17:10](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1030s) 交付规模实现指数级跨越，单体项目迈向 250MW 超级中心并行开建；Meta 同样推进多站点并行交付，并反向推行精简初始足迹的架构改革以减少前期资源锁定。 - [19:45](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1185s) 在缺少先验经验的超大规模无人区，CoreWeave 将企业级监控底座与网络专属团队解耦，基于开源组件与深度定制自研打造了名为“Echo”的可观测性技术栈。 - [22:32](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1352s) 由于缺乏足够人力对写入逻辑进行底层兜底审查，CoreWeave 对生产网中的 Agentic AI 严格限定在只读范畴，防范误操作引发的系统性雪崩。 - [26:40](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1600s) Shiv Patel 借助 Claude 编写出端到端网络容量计算工具，实现了从计算芯片规格到各级网络交换硬件的全链路映射推演，显著提升了前瞻性网络建模效率。 - [29:45](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1785s) 面对 AI 流量对骨干网的贪婪吞噬，CoreWeave 组建专属光纤容量团队，在战略汇聚点布局超 40 个 Transit POP 节点并加速铺设长途暗光纤。 - [32:41](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1961s) 倡导《极致所有权》文化，保持 80 余人团队高度扁平且亲密如家庭的协作氛围，打破直接汇报线藩篱，鼓励成员在 Slack 内部随时跨团队补位攻坚。 - [37:14](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2234s) Jim 仅投入 2.5 小时便自研出关联日志告警、Slack 消息与 Jira 工单的轻量级 AI Agent，显著提升了新中心上线期的排障对齐与监控盲区捕捉效率。 - [39:40](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2380s) SLO 体系正从形式化迈向深刻理解，Meta 实践表明成熟的 SLO 能衍生出错误预算，既为高风险极速交付提供试错空间，也是团队争取系统重构时间的话语权武器。 - [43:06](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2586s) 内部生产故障（Incie）依托 Slack 联动自动生成 Jira RCA 跟踪工单，而在高严重级别故障的复盘审核上仍坚持依靠资深工程师团队的人工深度复盘。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=0s) 播客开场：AI 狂潮下的基础设施与顶尖人才挑战 在虚构作品和现实世界中，人们总是习惯仰望英雄。CoreWeave 全球基础设施网络副总裁 Jim Julson 直言，在他目前约 85 人的团队中，随手就能挑出至少十几个“工程英雄”。这些关键人物像超级英雄一样轮换着披风奔赴前线，时不时也需要退下来为电池充电。然而，当这些顶尖技术人才长期处于极限冲刺状态时，身心消耗极为巨大。在当前的行业周期下，仅靠“砸钱”早就解决不了工程师倦怠的困境——当一个人每周 5 到 6 天、每天连轴转 16 个小时沉溺在繁重的任务中时，生活质量荡然无存，单纯堆砌薪酬毫无意义。如何维系并培育健康的团队文化、让每个人分担责任，已成为过去一两年管理层最重要的课题。 主持人 Francois Richard（Meta 生产工程总监）由此引入了本期 @Scale 播客。作为全球增长最快的专用 GPU 云服务商之一，CoreWeave 正处于 AI 算力革命的风口浪尖。两位工程领导者坐下来深入探讨：在 AI 云平台建设以惊人速度演进的当下，如何设计底层网络与系统框架、如何推动 NPI（新产品导入），以及更关键的——如何在这场史无前例的工程马拉松中留住并保护最核心的技术人才。 --- ### [01:12](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=72s) “风暴计划”与运维高压：我们为何依然需要工程英雄 Francois 与 Jim 开场互相打趣：做播客主持人和做播客嘉宾显然都不是两人的“正职”。Francois 介绍自己的日常工作主要是负责生产工程的被动式事件管理（Reactive Incident Management）与灾难恢复。团队每季度都会执行名为“风暴计划”（Storm Program）的实战演练，直接拔掉 Meta 的某一个数据中心区域（Region），测试全网系统在极端断网断电情况下能否平稳承受。“那场面可真是‘极低压力’啊，”Francois 幽默调侃道。 这种高压环境引出了当下 AI 基础设施领域的共性困境：算力基础设施正在以狂暴的速度扩张，系统架构频繁迭代，所有的压力最终全数倾泻在核心工程师身上。面对“顶尖员工是否快被逼疯”的提问，Jim 坦言这种人性的挣扎古已有之，但在当前的 AI 浪潮下被极大放大了。许多科技公司推崇“去英雄化”，认为依赖英雄会导致团队单点脆弱和员工倦怠；但 CoreWeave 却反其道而行之，选择公开赞扬英雄。在 Jim 看来，真正的英雄不是个人主义狂热者，而是那些身先士卒、为周围所有工程师开辟道路并树立榜样的人。团队需要他们引领方向，但管理者也必须建立机制，将“发现问题、解决问题”（Find a problem, fix a problem）的主人翁意识分散到各个职能孤岛，防止英雄独自累死在沙滩上。 - **团队规模与英雄占比**：Jim 的网络基础设施团队约 85 人，其中约 12 人被视为核心“英雄”（Top Performers）。 - **Meta 的容灾实战**：“风暴计划”（Storm Program）每季度主动切断并下线一个完整 Region，用实战检验系统的弹性和高可用性。 - **英雄的悖论**：英雄往往是自身成功的受害者（"Victims of their own success"），其产出往往是普通工程师的 5 倍甚至 10 倍，导致所有最棘手的问题都会自动汇聚到他们手中。 ```mermaid flowchart TD accTitle: 基础设施救火困境与所有权文化转型 accDescr: 展现高级工程师从被动救火的恶性循环，走向通过建立所有权文化分担负载的演进路径。 subgraph Firefighting["恶性循环：依赖英雄救火"] A["系统故障与高负荷"] --> B["英雄工程师介入响应<br/>(产出高出普通人 5-10 倍)"] B --> C["缺乏局部模块 Ownership<br/>其他成员不敢/不愿接手"] C --> D["英雄 16 小时连轴转<br/>身心俱疲、无法聚焦技术创新"] D --> A end subgraph CultureShift["良性循环：全栈所有权与文化"] E["管理层介入：强制断联休假<br/>(2-4 天小长假，关机脱网)"] E --> F["深挖根因：厘清技术栈所有权<br/>(Find a problem, fix a problem)"] F --> G["负荷分流至基层工程师<br/>资深骨干退回创新与架构设计"] end B -.->|反思管理策略| E ``` --- ### [04:34](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=274s) 摆脱“救火”恶性循环：从休假策略到建立全栈所有权（Ownership） 针对如何防止英雄被压垮，Francois 分享了自己的做法：在重大发布（Release）前夕由英雄顶上冲锋，但在项目落地后，必须强制安排他们回到常规的日常工作节奏，刻意错开下一个高压冲刺周期，通过“波峰-波谷”的高低节奏来调节神经。Jim 对此深表赞同，并透露了自己的管理小技巧：他会私下找核心骨干谈话，要求他们在接下来的两到三个月里，每月至少休两次三天或四天的小长假，甚至直接告诉他们“不用提 PTO 申请流程，直接人间蒸发、把手机彻底关掉”。 然而，休假只能治标，工程师一旦回到工位，面对的依然是一团乱麻。Jim 强调，管理的根本在于追溯“为什么只有这几个人在拼命救火”。其深层根源往往是“所有权”（Ownership）的缺失。如果普通团队成员对技术栈的某个切片缺乏责任感，认为“这不是我的系统”，所有的故障处理最终就会无休止地反弹回高级架构师和技术骨干手中。这种持续的救火状态会彻底剥夺高级人才开展技术创新、重构和长期架构规划的时间窗口，使团队陷入战略停滞。 --- ### [07:13](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=433s) 赌注飙升的 AI/ML 时代：高管领导力的蜕变与学会说“不” 回顾自己从早期就职于 DigitalOcean 到如今执掌 CoreWeave 全球网络的职业生涯，Jim 指出，网络工程的基础物理规律和技术逻辑在过去十几年并没有发生颠覆性突变；正如业内同行所言，网络问题依然是那些网络问题，但在 AI/ML 时代，这些问题被全方位放大了，且解决速度被要求压缩到极致。驱动这种变化的底层原因是极其惊人的资产成本：GPU 集群的投入是一笔天文数字，“这根本不是穷人的游戏，你是在完全不同的平流层里下注”。如此高昂的折旧与停机成本，使得系统的平均检测时间（MTTD）和平均响应时间（MTTR）具有了极高的商业可见度，每一次排障救火都直接关系到数以亿计的业务底线。 谈到自身领导风格的转变，Jim 坦言自己并非生来就是老练从容的管理者，而是一路“向前失败”（Fail forward）跌撞成长起来的。面对这种巨大的压力，他将全部功劳归于身后的团队： - **从执行者到领航者**：从过去亲自操刀底层架构的工程师，转变为高管后，核心职责变成了在技术和团队之间建立愿景，身先士卒地展示什么叫做责任感，并将这种 Ownership 自上而下辐射到整个组织。 - **高管的时间黑洞与边界感**：当 Francois 问及作为高管如今是否还有空深潜技术细节时，Jim 幽默地回答：“如果问我妻子，答案是绝对没有；如果问我的同事，偶尔还能挤出一点。”高管的日常往往被无休止的 Zoom 和 Google Meet 会议吞噬，Jim 认为自己能够保持技术敏锐度的关键，恰恰在于保留了“敢于说不”的决断力，从而在繁忙的协作洪流中守住深入解决核心技术难题的空间。 ### [10:05](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=605s) 技术型领导者的坚守：亲临一线排障与设定现实的 NPI 目标 在工程管理实践中，技术领导者往往面临着在宏观战略与微观代码之间取舍的永恒命题。Jim 谈到，随着企业组织规模极速扩张，自己的精力不可避免地被抽离出底层细节，但他评估自身价值的标准始终是“是否能为公司带来真正的实质增益”，而不仅仅是在会议中发表意见。Francois 对此深有共鸣，坦言自己至今仍然执着于保留排查疑难故障（debugging）与探究底层代码的技能。他认为亲自上手不仅充满乐趣，更能直接检验内部工具的真实水准——只有亲自跑过流程，才能一眼看出“某个工具到底有多烂，或者为何完全无法实现最初预期的功能”。然而，在排期紧凑的日程中挤出两到三小时连续时间，与一线团队共同排查生产环境故障，需要极强的自律与日程管理。 Francois 进一步指出，在制定新产品导入（NPI）等关键工程目标时，领导者如果脱离实际操作，极易脱离现实，制定出“即便连光速都无法达成”的荒谬时间表。因此，领导者必须对执行链条上的每一步骤保持敏锐的体感，即便不必亲力亲为处于关键路径（critical path）上，也必须保持技术敏锐度。Jim 对此高度赞同，特别强调在 AI/ML 基础设施这一复杂度极高且快速演进的领域，管理者若无法熟练使用技术团队的语言，就无法建立起深层的同理心与专业判断力；纯粹的“人员管理者”（people manager）在面对这种极度硬核的技术挑战时，注定会陷入举步维艰的困境。 - 领导者下场参与生产排障的核心价值不仅在于解决具体问题，还在于第一视角验证内部工具链的有效性。 - NPI（New Product Introduction，新产品导入）目标周期的合理制定，高度依赖管理者对底层工程执行步骤的技术体感。 - AI/ML 基础设施的技术密度与变动速度，要求工程管理者必须具备深厚的技术语言能力与同理心。 ### [12:17](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=737s) 告别 Jinja 模板：以 API 优先架构打破配置管理泥潭 当谈及连接工程人员与底层物理系统的“框架”（Framework）时，Jim 剖析了 CoreWeave 在网络自动化领域的设计哲学。针对业内普遍热议的“事件驱动自动化”（Event-driven Automation），CoreWeave 并没有停留在空洞的行业流行词层面，而是坚决告别了网络工程界长期依赖但极易失控的脆弱工具。Jim 直言不讳地指出，如果试图依靠庞杂的 Jinja 模板去规模化管理不同站点，基础设施运营将沦为一场“彻头彻尾的噩梦”。 为了从根本上解决海量异构站点的管理问题，CoreWeave 的核心策略是招募真正的专业软件开发工程师（True Developers），直接构建以 API 为核心的后端系统，并在特定场景下配套开发直观的 Web 前端。这些 API 构成了全网工程师跨规模管理物理网络的主入口与唯一契约，使网络运维彻底摆脱了零散的脚本交互模式。通过将“事件优先”（Eventing First）写入自动化框架与规范，CoreWeave 实现了在全网所有数据中心站点以高度一致、全局可编程的 API 框架进行闭环治理。 ```mermaid flowchart TD accTitle: CoreWeave 基础设施管理分层体系 accDescr: 展示从运维人员到 API 后端，再到自动化编排与底层物理基础设施的分层架构 subgraph InterfaceLayer["交互与抽象层"] A["网络/系统工程师"] --> B["统一 API 接口平台"] C["Web 管理前端"] --> B end subgraph FrameworkLayer["软件自动化框架"] B --> D["MCP 分布式控制程序"] B --> E["事件驱动运营框架"] end subgraph InfrastructureLayer["物理基础设施层"] D --> F["新站点引导 (Bootstrap) & 配置漂移检测"] E --> G["无损故障隔离 (Drain) & RoCE 织网自愈"] end ``` - CoreWeave 坚持引入专职软件开发人员开发后端 API 与运维前端，取代传统的脆弱网络脚本与模板。 - 全网多数据中心采用统一的 API 规范进行系统抽象，使网络状态的管理与下发完全代码化、接口化。 - 自动化策略明确要求以事件驱动（Eventing First）为默认设计准则，仅对极少数非常规场景保留特例处理。 ### [13:58](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=838s) 运维与建设解耦：MCP 站点交付与事件驱动的无损排障 针对 Francois 关于新站点开局（Site Turnup / Bootstrap）如何通过事件驱动落地的追问，Jim 澄清了一个关键的技术认知偏差：在超大规模网络体系中，“自动化”这一术语承载了截然不同的阶段任务，必须在生命周期上进行明确解耦。真正的事件驱动自动化目前主要深植于“生产运营期”（Operational Side），而非“全新建设期”（New Build Side）。在新数据中心冷启动阶段，CoreWeave 依赖一套名为 MCP（Master Control Program，取名自经典科幻电影《电子世界争霸战》*Tron*）的分布式现场 API 软件栈。 MCP 部署在每个独立站点，专职负责硬件的引导安装、初始网络配置下发、全网配置漂移（Config State Drift）监测，并维护极其庞大的光纤链路修复“燃尽清单”（Burndown List）。在进入生产状态前，MCP 必须验证整个网络织网（Fabric）的拓扑完整性。Jim 举例道，在一个预期存在 256,000 条链路互联的 Fabric 中，系统会精确统计出当前已就绪与仍在施工中的端口数量。而在数据中心上线进入运营期后，事件驱动自动化系统便接管了一切：当底层包含数千个计算节点的 RoCE 网络发生链路抖动（Flapping）、端口误码或 BGP 震荡时，系统会捕捉事件并“优雅排空”（Gracefully Drain）故障节点的控制面与数据流，使大规模分布式 AI 模型的训练任务完全不受中断干扰。 - **阶段解耦**：全新站点建设属于计划驱动的程序化交付；在网运营排障则全面采用实时事件驱动。 - **MCP 核心功能**：单站点分布式部署，执行自动化 Bootstrap、配置状态漂移比对与施工期数十万条物理链路的质量验证。 - **运营期自愈**：在规模达 7,000 个节点的 RoCE（RDMA over Converged Ethernet）网络中，一旦监测到链路抖动或 BGP 异常，系统自动执行控制面 Graceful Drain，保障上层大模型训练任务平稳运行。 ### [17:10](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1030s) 从 90 天 4 座 5MW 到同时开建 200MW：两代超大规模数据中心的裂变 谈及数据中心交付规模的演进历程，Jim 分享了自己在 2023 年 2 月入职 CoreWeave 时令人啼笑皆非又惊心动魄的经历。当时他原定于 3 月中旬入职，但联合创始人直接在 2 月第三周将他飞往拉斯维加斯；落地仅仅两天，面对眼前等待交付的极端任务，Jim 当场决定提前入职：“因为我被即将面临的艰巨任务彻底吓到了。”那项看似不可能完成的挑战，是在大约 90 天内冷启动建设 5 座数据中心，而当时现场几乎“毫无自动化系统，毫无成熟设计，一切全凭手工拆箱组装”。Jim 火速招募了 3 名核心工程师，以区区 4 人的初始核心小队拼死攻坚，最终在 90 天内成功交付了 4 座，第 5 座也在随后一个月内顺利交付。 然而时过境迁，Jim 指出当年那些容量仅约 5 兆瓦（MW）的数据中心在今天看来甚至“微不足道”，如今甚至不会再立项这种体量的站点。CoreWeave 目前面对的全新常态，是同时并发建设 3 到 4 座单体规模达 200MW 乃至 250MW 的超大型超级数据中心，这无疑是一项需要全公司数百人紧密协同的“举社之战”（It takes a village）。Francois 对此深表赞同并分享了 Meta 的相似轨迹：Meta 过去在同一时间段仅能支撑 1 到 2 座数据中心的并行建设，如今已激增到 5 到 6 座同时开工。并发规模的急剧放大也催生了全新的工程难题——为了支撑超大规模并行交付，团队正在反向推行精简初始足迹（Footprint）的架构改革，尽可能减少建设初期的核心网络与基础存储容量占用，以降低并行建造过程中的资源冗余与交付阻力。 - **初创期爆发交付**：2023 年初，Jim 组建 4 人初始核心团队，在毫无现成自动化系统的极端条件下，于 90 天内完成了 4 座 5MW 级别数据中心的交付。 - **体量指数级跨越**：CoreWeave 的单体建设项目已从当年的 5MW 飙升至单体 200MW～250MW，且支持 3 至 4 座超大型数据中心同步并行交付。 - **Meta 的并行扩张挑战**：Meta 当前同步开建的数据中心已达 5 到 6 座，推动架构团队探索“缩小初始交付足迹”的新模式，压缩基础网络与存储在前期的资源锁定。 ### [19:45](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1185s) 规模化的无人区与 CoreWeave 的可观测性 Echo 体系 在完成既定部署目标之后，工程团队面对的不是松一口气，而是更为严峻的催促：“恭喜你干得很棒，现在再去干更多。”这种无休止的高强度演进在整个业界都极其罕见。目前全行业只有极少数几家公司正在以如此庞大的规模、极端的体量以及突破极限的速度构建基础设施。这意味着工程团队在遇到深水区问题时，根本没有任何既有的“先验经验”（prior art）可以参照，整个过程就是一场持续不断的摸索与挣扎。为此，CoreWeave 必须在组织层面高度聚焦，围绕诸如新站点上线交付（site turn up）等极其疯狂激进的工期目标展开攻坚。 谈到日常支撑整个基础设施运转的核心系统时，Jim Julson 强调，最核心、最受关注的当属**可观测性与遥测技术栈（Observability and Telemetry Stack）**。CoreWeave 在公司层面设立了专门的企业级可观测性团队，负责全公司海量监控指标的摄取底座（涵盖键值存储、时间序列数据库等基础设施）。与此并行，Jim 的网络工程部门内部也拥有一支专属的网络可观测性团队： - **职责定位**：网络专属团队是底层监控基础设施的使用者与消费者，其核心使命是将全网网络工程师对“网络健康状态”（Network Health）的工程构想转化为具体现实。 - **产品落地**：负责搭建统一的 SLO 仪表盘（SLO Dashboards）以及各种独立的故障排查看板（Troubleshooting Dashboards）。 - **技术选型（Echo 技术栈）**：整套框架并非直接购买单一的现成商业产品，而是以一整套开源组件为骨架，并在其上叠加了大量自研定制代码，内部将其命名为 **“Echo” 技术栈**。 - **厂商协同**：团队与英伟达（NVIDIA）等生态伙伴建立了深度合作渠道，在系统开发演化过程中持续交换底层网络监控需求与最佳实践。 ```mermaid flowchart TD accTitle: CoreWeave 遥测与可观测性系统分层架构 accDescr: 展示企业级基础设施团队与网络专属团队的分工，以及 Echo 观测技术栈的数据流向。 subgraph InfraLayer["底层基础设施层（企业级可观测性团队）"] TSDB["时序数据库 (Time Series Store)"] KVStore["键值存储 (Key-Value Store)"] end subgraph EchoStack["Echo 技术栈（网络专属可观测性团队）"] CustomEngine["开源组件 + 大量自研定制代码 (Echo Engine)"] HealthLogic["网络健康评估逻辑 / NVIDIA 深度协同"] end subgraph PresentationLayer["业务与运维呈现层"] SLODash["统一 SLO 仪表盘"] DebugDash["独立排障分析看板"] end InfraLayer --> CustomEngine CustomEngine --> HealthLogic HealthLogic --> SLODash HealthLogic --> DebugDash ``` --- ### [22:32](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1352s) AI 智能体进驻生产网的风险与“意图可观测性”挑战 Francois Richard 探讨了现代生产环境中执行变更的三类主体： 1. **人类工程师**：行为难以预测，但因为具备谨慎心智，操作时往往极度小心； 2. **自动化系统（Automation）**：行为完全可预期，基于固定脚本或预设程序执行； 3. **AI 智能体（Agents）**：介于人类与系统之间，既具备自主性又存在不确定性。 对此，Jim 明确表示 CoreWeave 目前对直接将 Agentic AI 引入生产网络 Fabric 保持极度谨慎（bearish）。虽然团队在内部开发了部分基于智能体的自研工具，但现阶段严格限制在**只读操作（Read-only Operations）**范畴内。原因在于跨交换机的数据关联诊断、自动切流隔离或生成合成流量测试等写入型动作风险极高。在公司整体以极高速度狂奔的同时，招聘到合适的技术专才本身就是巨大挑战。没有足够的人手去对底层逻辑进行兜底审查，贸然引入写入操作是弊大于利。Jim 直言不讳：“执行写入操作至今仍让我感到害怕。” 随后，Francois 分享了 Meta 在引入 AI 智能体时倒逼出来的全新可观测性技术维度。当智能体代表人类工程师在网络中发起变更突变（mutation）时，排障链路变得极为复杂，必须构建端到端的**身份与意图追踪链条**： ```mermaid flowchart LR accTitle: 智能体变更行为的上下文与意图追踪链路 accDescr: 描述人类工程师通过智能体下发指令，系统记录身份代理、原始意图和执行上下文的审计链路。 subgraph IdentityChain["双重身份穿透"] Human["人类工程师<br/>(人类身份证书)"] -->|发起委托需求| Agent["AI 智能体<br/>(Service Identity)"] end subgraph IntentLogging["意图日志记录 (Intent Logging)"] Agent --> ContextAudit["捕捉上下文假设<br/>& 原始 Prompt 意图"] end subgraph Execution["变更落地与溯源"] ContextAudit --> NetworkChange["网络生产突变变更<br/>(Transit via XYZ)"] end ``` - **双重身份穿透**：传统的自动化采用统一的服务账号身份，人类采用个人证书。但智能体是代理人类发起操作，追踪链必须同时携带“智能体自身的系统身份”以及“其背后的发起人身份”。 - **意图日志记录（Intent Logging）**：排查故障时仅看执行了哪行配置远远不够，必须记录人类向智能体下达指令时的原始意图、智能体在做推断时的假设前提以及推理上下文。这催生出了一门崭新的“启发式推断审计学科”。 - **团队规模制约**：Jim 补充提到，CoreWeave 目前仅有约 4,000 名员工（6 个月前仅为 3,000 人左右）。尽管其他业务团队已经在业务流中探索智能体，但网络底层坚决不能“好大喜功、贪多嚼不烂”（not biting off more than we can chew）。在没有足够的专属工程师去专门为这些代码再写一套监控和保障系统之前，贸然放开智能体控制无异于引火自焚。 --- ### [26:40](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1600s) 基于 Claude 的容量规划计算器与全球骨干网演进 面对新产品导入（NPI, New Product Introduction）中极具挑战性的网络容量规划问题，业务规模往往会从预估的 $X$ 陡增至 $10X$。Jim 特别提到了他在十余年前（约 10 到 12 年前）在 Dell Virtustream 生态工作时结识并一直视作导师的技术大牛——**Shiv Patel**。在 Shiv 历经微软等公司的历练后，Jim 成功将其招募到 CoreWeave，担任负责 NPI 的核心专家。 面对复杂的网络建模痛点，Shiv 展现出极其实用主义的工程创新： - **网络建模即服务（Network Modeling as a Service）**：Shiv 直接借助 Anthropic 的 **Claude** 编写了一整套端到端的网络容量计算器，并配合 Google Sheets 工具。 - **全链路映射推演**：该工具能够直接根据特定类型的 GPU 数量（如 $N$ 块特定规格 GPU），精准反推并计算出支撑该集群所需的各级网络交换机数量（涵盖 T0、T1、T2 交换矩阵层级）；针对 CPU 的特定工作负载，同样能精准输出配套网络硬件需求。 - **面向未来架构**：这种看似简单、优雅却极为高效的工具，近期被直接应用于应对即将到来的英伟达 **Vera Rubin** 产品线的网络容量规划中。 最后，Francois 询问了区域互联与骨干网层面的光纤与运力挑战。Jim 证实，CoreWeave 内部按**区域（Regions）**与区域内的**可用区（Availability Zones）**进行架构划分。随着规模爆发，专职的全球骨干网（Global Backbone）团队正经历重大转型： - 早期阶段，CoreWeave 长期依托第三方供应商采购波分传输服务（Wave Transports），无论数据中转（transit）还是长途传输（transport），全套链路及对应的 NOC 运维、割接维护通知与交付开通均整体委托给第三方代管。 - 尽管第三方合作伙伴表现极其出色，但伴随 AI 超算对骨干带宽需求的指数级攀升，这种外包模式不可避免地迎来了其承载能力的临界瓶颈，倒逼底层传输模式发生蜕变。 ### [29:45](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1785s) 骨干网架构的极限迭代与光纤容量规划 随着业务版图在全球范围内的极速膨胀，CoreWeave 铺设和点亮的光纤网络（Lit Waves 与 Dark Fiber）规模呈爆发式增长，公司甚至在内部组建了专门的全职光纤容量规划团队（Fiber Capacity Planners）。Jim Julson 指出，这支团队几乎是 7×24 小时连轴转，以应对令人难以置信的需求浪潮。然而，网络规模的增长速度远超传统预期，两三年或两年前所设计的骨干网架构，在如今的 AI 流量吞吐面前已经不堪重负。Jim 坦言，这几乎是他职业生涯中所见过架构寿命最短的骨干网设计——团队为了满足产品端不断涌现的新特性以及客户日益复杂的对接需求，一直在架构演进上拼命“追赶”。 ```mermaid flowchart LR accTitle: CoreWeave Global Network Topology accDescr: Diagram illustrating transport network interconnecting data centers and transit POPs for peering and Direct Connect. subgraph DataCenters["全球数据中心集群"] DC1["AI 算力中心 A"] DC2["AI 算力中心 B"] end subgraph TransportCore["全球骨干网 / 传输层"] TN["东西向长途暗光纤<br/>(Global East-West Transport)"] end subgraph EdgeTier["边缘中继层 (40+ 节点)"] POP1["Transit POP<br/>(运营商直连 & 对等互联)"] POP2["Transit POP<br/>(客户专线 Direct Connect)"] end DC1 <--> TN DC2 <--> TN TN <--> POP1 TN <--> POP2 ``` 与此同时，为这些承载海量算力的数据中心进行容量规划也是一项艰巨的挑战。AI 训练对带宽的渴求极度贪婪，所消耗的带宽指标完全颠覆了传统数据中心或网络服务商的认知范畴。这种庞大的体量使得前期准备与规划的时间跨度被拉得极长，团队必须在合同正式敲定、机房“白空间”（White Space）交付之前就提前数月介入。在商业合同与工程部署交织的现实中，团队时刻都在跳着“按 X 规模规划、但随时要应对 Y 变化”的高难度平衡舞步。 - **超短生命周期**：两三年前搭建的骨干网已无法支撑当前规模，网络架构被迫以前所未有的高频次进行全面推倒与重构。 - **全球边缘中继节点（Transit POPs）**：为了满足客户面向公网的服务调用、推理请求（Inference Serving）及私有互联需求，CoreWeave 在战略运营商汇聚点部署了 40 多个 Transit POP 节点，优化了互联网中继（Internet Transit）与对等互联（Peering），并提供客户直连（Direct Connect）服务。 - **短期扩容计划**：目前内部还有 8 到 10 个新的 Transit POP 正在紧锣密鼓地筹备中，预计在接下来的 6 至 8 个月内陆续上线，伴随而来的还有大量长途暗光纤（Long-haul Dark Fiber）的落地。 --- ### [32:41](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1961s) 工程师文化：“极致所有权”与扁平无界的协作 谈及 CoreWeave 与 DigitalOcean 等其他技术公司的文化异同，Jim Julson 结合自身职业生涯的成长轨迹进行了解读。他回忆道，自己入行时连 Google 都算新鲜事物，那时的工程师文化信奉“发现问题，就去解决问题”，而不是指望有现成的答案或依赖别人来兜底。这种在早期职业生涯中沉淀下的强大主人翁意识（Sense of Ownership）成为了他骨子里的工程基因，并在 CoreWeave 得到了发扬光大。“像主人一样思考与行动”（Act like an Owner）不仅写在 CoreWeave 的企业核心价值观里，更是整个网络团队 80 多名工程师的日常实践准则。 Francois Richard 借此向 Jim 推荐了经典管理著作《极致所有权》（*Extreme Ownership*），指出当工程师把主动权融入 DNA 时，能释放出惊人的自主执行力与心理驱动力。Jim 进一步分享，尽管网络团队已扩展至 80 余人，团队依然保持着极为扁平、亲密如家庭（Familial）的沟通氛围。任何工程师在内部 Slack 频道遇到阻碍，都可以打破汇报线（Reporting Lines）的藩篱，随时拍拍同事的肩膀寻求帮助，这种无缝协同的作风确保了组织在高速膨胀中仍能保持高效作战。 - **高度自主权**：强烈的责任意识赋予团队自主操作的信心，促使工程师深入底层技术栈去构建真正有意义的解决方案。 - **打破组织壁垒**：不以“直接汇报给谁”划定职责边界，只要能解决技术难题，全员随时跨团队补位。 - **高阶资深 IC 资源池运作**：Francois 产生共鸣并补充道，类似的做法是将资深独立贡献者（Senior ICs）组建成灵活的特战资源池，针对特定任务攻坚并建立信任纽带，攻克后再轮转至新阵地。 --- ### [37:14](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2234s) 2.5 小时写出的事件关联 Agent 与网络健康度定义 当 Francois 问及最近在工作或生活中最令人印象深刻的 AI 应用时，Jim 透露他自己动手编写了一个轻量级 AI Agent。这个小工具通过对接 Slack 与 Jira，在 CoreWeave 的底层日志系统告警、自动化工作流触发以及工单创建（Ticket Creation）之间建立实时的时序关联。在过去三四个月里，CoreWeave 正在大规模上线新硬件与网络基础设施，初期部署就像一场“持续轰鸣的大火”（Constant Roaring Fire），充满排障镇痛期。 ```mermaid flowchart TD accTitle: AI Event Correlation Agent Architecture accDescr: Flow diagram showing how the AI agent correlates real-time logging alerts, ticketing, and Slack/Jira to detect operational gaps. subgraph RawSignals["底层系统信号"] LogSys["底层日志系统告警<br/>(Real-time Logs)"] Workflows["自动化工作流<br/>(Automation Workflows)"] end subgraph AIAgentLayer["AI 关联处理引擎 (Jim 编写的 Agent)"] Agent["时序关联 & 缺口分析 Agent"] end subgraph CollabTools["协同与追踪平台"] Jira["Jira 故障工单"] Slack["Slack 通知与群聊"] end LogSys --> Agent Workflows --> Agent Agent <--> Jira Agent <--> Slack Agent -.-> Result["校验报警是否遗漏<br/>缩短故障排查链路"] ``` Jim 强调，构建这个 Agent 并非为了追责或找人背锅，而是为了站在全局视角验证监控数据是否吻合、告警链路是否存在盲区。令人惊叹的是，这个极其实用的排障辅助 Agent 仅用了大约两个半小时就构建完成，并在过去的两三个月里释放了巨大的运营排查价值。随后话题转向可靠性工程体系建设，面对团队规模激增，Jim 敏锐地意识到内部对于“什么是健康的网络”缺乏共识——向不同的工程师提问往往会得到完全不同的定义。为此，团队正式启动了标准化治理，针对前端网络等核心支柱（Pillars）及各类设备角色，自顶向下建立全组织层面的 SLO/SLI 与网络健康度评估基线。 - **快速构建与高价值产出**：仅投入 2.5 小时编写的 AI Agent，显著提升了新数据中心从初始化接入到进入平稳运行期（Steady State）过程中的排障与数据校验效率。 - **告警与工单链路对齐**：Agent 重点比对从日志系统发出警报到工单被自动触发的时间差，确保基础设施告警体系毫无遗漏。 - **推动组织级 SLO/SLI 标准化**：针对各团队对“网络健康状态”各执一词的痛点，发起统一规范工程，为前端网络及设备角色量身定制明确的指标度量体系。 ### [39:40](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2380s) 从简单粗暴到深刻理解：建立真正有意义的 SLO 体系 在讨论大规模 AI 基础设施的管理时，Jim 梳理了 CoreWeave 内部的核心支柱：后端网络（Backend Network）、管理网络（Management Network）、骨干网（Backbone）以及上层的服务层（Services Layer）。运行在其中的所有服务与基础设施都绑定了严格的 SLO（服务等级目标），覆盖可用性、可靠性及容量等关键维度。Jim 坦言团队目前正深陷于 SLO 的研发与探索中，去探寻哪些指标才能真实反映系统健康度，这是一场漫长的旅程。回顾初期，Jim 笑着自嘲道，自己最初在脑海中把这件事严重简单化了——“直接把几个 SLI 扔进 SLO 桶里不就完事了吗？”随后他才意识到现实的复杂，“真该骂自己，我本该更懂行的”。但经历这个打磨过程让人极其受益匪浅。起初，整个 80 人的网络团队中只有 10 到 12 人参与 SLO 的讨论会议；而如今参会人数已壮大到 35 至 40 人，近半数团队成员的主动投入是一个极佳的信号，代表大家真正开始敬畏系统健康度，通过“众包”形式共同把控质量。 Francois 对此深有共鸣，并分享了 Meta 的经验：他们足足花了整整三年时间，才让核心内部服务（Big Rocks）的 SLO 真正具有实际业务意义，做到“指标变红时，系统一定是真彻底坏了”。在 Meta，每次发生生产事故（Sev）时，团队必须建立严格的纪律性复盘：检查报警与事故是否准确冲击了相应的 SLO。如果某次内部事故丢失了系统冗余，即便终端用户完全无感知，SLO 也必须有所体现与下跌，否则就说明现有的可观测性体系存在盲区。不仅如此，成熟的 SLO 还能衍生出极具价值的“错误预算”（Error Budget）：当需要冒险快速上线新区域（Region）或抢时间做非计划性维护时，工程团队可以正当地消耗这些预算；而当某项核心指标长期走低时，它也是团队叫停旁支任务、回归基础修缮系统的关键武器。 ```mermaid flowchart TD accTitle: SLO 与故障风险闭环管理架构 accDescr: 展示基础设施四层架构到SLO维度评估，再到故障关联与错误预算决策的完整闭环。 subgraph InfraLayers["基础设施核心支柱"] B["后端网络 Backend"] M["管理网络 Management"] BB["骨干网 Backbone"] S["服务层 Services"] end subgraph SLODimensions["SLO 评估维度"] SLO["核心指标: 可用性 / 可靠性 / 容量"] end subgraph FeedbackLoop["生产关联与业务决策"] SEV["生产事故 Sev / 冗余丢失"] -->|严格复盘对齐| CORR{"是否冲击 SLO?"} CORR -->|未冲击/盲区| FIX["修正指标与盲区"] CORR -->|准确反映| EB["错误预算 Error Budget"] EB -->|预算充裕| RISK["支持高风险操作: 快速上线新 Region / 紧急维护"] EB -->|预算耗尽| BASICS["叫停新业务，回归系统底层重构"] end InfraLayers --> SLO SLO --> FeedbackLoop ``` - **基础设施四大支柱**：后端计算网络、带外管理网络、广域骨干网以及上层运行的基础软件服务。 - **团队参与度跃升**：CoreWeave 网络团队参与 SLO 定期研讨的人数从最初的 10–12 人增长到 35–40 人，健康意识深入工程师日常。 - **Meta 的三年成熟期教训**：SLO 必须与实际故障强相关，即使是“用户无感知”的冗余组件故障也必须扣除相应预算。 - **错误预算的双向赋能**：一方面为高风险的极速交付（如 90 天交付数据中心）提供试错空间，另一方面为工程团队争取重构底层烂摊子的正当话语权。 --- ### [43:06](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2586s) 故障管理机制：事件工单与以人为本的复盘流程 当 Francois 问及 CoreWeave 是否自研了用于网络生产故障的事件管理工具时，Jim 介绍说公司内部将此类故障统称为“Incie”（即 Incidents）。CoreWeave 设立了专门的流程和专职团队负责维护故障响应框架与分级体系（Severity Levels）。在系统整合上，他们构建了事件驱动的工作流体系：工程师通过 Slack 触发创建 Incie 故障群组时，后台系统会自动联动生成用于根因分析（RCA）的 Jira 跟踪工单，保证整个排查链路可溯源。 在关键故障的复盘审核（Review）环节，Jim 表示目前网络团队仍以人工评审为主。虽然每天要处理不少故障事件，但从网络基础设施的视角来看，系统的整体稳定性维持得相当不错（Jim 还幽默地敲了敲木头祈求好运，说“Knock on wood”）。目前核心网络团队并没有引入过于繁重机械的自动化审查手段，而是与负责全公司事件管理生态的独立横向团队紧密结对配合，以确保高严重级别事故得到透彻分析。 - **故障简称与自动化联动**：内部事件统称“Incie”，由专职团队制定严重等级制度，Slack 端指令可一键自动生成关联的 Jira RCA 工单。 - **审核机制的侧重点**：尚未采用完全自动化的审查流程，主要依赖高水平工程师的深度复盘与人工判断。 --- ### [44:25](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2665s) 极客的下班仪式：自建开源家庭影院与全套可调光系统 对话转向业余生活中的技术应用时，Francois 询问 Jim 除了工作是否会在私生活中使用 AI 或工程工具。Jim 爽朗地表示，一旦下班，他最想做的事情就是彻底合上工作笔记本，远离任何纯办公的纯技术屏幕。他深情谈到自己对目前工作与团队的热爱：“我非常尊重 CoreWeave 的几位创始人，是他们录用了我，且毫无保留地信任我；我很享受为他们、为团队以及为自己拼命奋斗的感觉。”但高压运转之后需要一个宣泄口，他的解压方式是在家中亲手建造一个高度自动化的私人家庭影院（Movie Room）。 谈起这个家庭项目，Jim 的极客本质瞬间流露出来。他不仅没有彻底“远离技术”，反而搭建了非常硬核的本地家庭控制中心：基于 openHAB 并通过底层 API 全面接入 Home Assistant，自己写了一整套事件驱动（Event-Driven）的联动控制逻辑。此外，他还在影院内自学并部署了生产级（Production-Scale）的专业 LED 商业照明控制，配备全套自动定时器与场景提示光（Cue Lighting）。Francois 听后哈哈大笑，表示自己也在做类似的事情——折腾全屋智能照明改造，利用全光谱可调白光与色彩搭配，只要能将老旧的筒灯改装并连上网受控，就乐在其中。两人心有戚戚，一致认为将工程极客精神倾注在能与家人一同享受的实体生活空间中，是工程师最高级别的放松。 - **家庭影院技术架构**：采用 openHAB 顶层控制，后端借助 REST API 深度对接 Home Assistant，完全依托事件驱动逻辑运行。 - **照明工程**：落地专业工业级 LED 系统，涵盖全场景氛围光、自动化入场定时器及播放时的场景提示照明（Cue Lighting）。 - **极客共鸣**：高强度基建工程师将物理现实世界的 DIY 改造（智能灯光、影院搭建）作为平衡高压屏幕工作的心理代偿机制。 --- ### [46:32](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2792s) 结语与 @Scale 线下大会预告 随着对智能家居的畅聊结束，Francois 对 Jim Julson 的倾囊相授表达了诚挚感谢，盛赞 CoreWeave 在全球 AI 算力网络基建上的开拓经验。节目最后，Francois 面向广大技术听众预告了即将到来的线下盛会：下一届 Product @Scale 峰会将于 10 月 28 日在加利福尼亚州门洛帕克（Menlo Park）举办。届时来自 Meta、Reactor.inc、Intel、斯坦福大学等顶尖科技机构与学府的专家将齐聚一堂，共同探讨 AI 与大规模基础设施领域的最新前沿创新。 ## Notable quotes > [01:12](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=72s) 那场面可真是‘极低压力’啊。 — Francois Richard > [01:12](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=72s) Victims of their own success — Jim Julson > [04:34](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=274s) 不用提 PTO 申请流程，直接人间蒸发、把手机彻底关掉。 — Jim Julson > [07:13](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=433s) 这根本不是穷人的游戏，你是在完全不同的平流层里下注。 — Jim Julson > [07:13](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=433s) 如果问我妻子，答案是绝对没有；如果问我的同事，偶尔还能挤出一点。 — Jim Julson > [10:05](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=605s) 某个工具到底有多烂，或者为何完全无法实现最初预期的功能。 — Francois Richard > [12:17](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=737s) 彻头彻尾的噩梦 — Jim Julson > [17:10](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1030s) 因为我被即将面临的艰巨任务彻底吓到了。 — Jim Julson > [17:10](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1030s) It takes a village — Jim Julson > [22:32](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1352s) 执行写入操作至今仍让我感到害怕。 — Jim Julson > [32:41](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=1961s) Act like an Owner — Jim Julson > [39:40](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2380s) 直接把几个 SLI 扔进 SLO 桶里不就完事了吗？ — Jim Julson > [39:40](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2380s) 真该骂自己，我本该更懂行的。 — Jim Julson > [39:40](https://www.youtube.com/watch?v=_DYIDfx4g_s&t=2380s) 指标变红时，系统一定是真彻底坏了。 — Francois Richard ## Mentioned Jim Julson, Francois Richard, Shiv Patel, CoreWeave, Meta, DigitalOcean, NVIDIA, Anthropic, Claude, Dell Virtustream, Intel, Stanford University, Reactor.inc, MCP (Master Control Program), Echo, Vera Rubin, Jinja, Slack, Jira, openHAB, Home Assistant, RoCE, BGP, Extreme Ownership, Tron ## Video description What does it take to build and scale the infrastructure powering the AI revolution? In this episode of the @Scale podcast, Francois Richard, Engineering Director at Meta, sits down with Jim Johnson, VP of Global Infrastructure Networking at CoreWeave, to explore the technical and organizational challenges of scaling AI infrastructure. Jim shares lessons from his experience helping CoreWeave grow from standing up four data centers in just 90 days to operating a global infrastructure footprint that continues to evolve alongside the rapidly growing demands of AI. The conversation covers: • Building and scaling AI infrastructure at unprecedented speed • The role of engineering “heroes” — and why teams need a culture of ownership to scale effectively • Event-driven automation and CoreWeave’s approach to infrastructure management • Building observability systems for large-scale AI infrastructure • Capacity planning across GPUs, CPUs, switches, and networks • The opportunities and limitations of using AI agents in production environments • Why infrastructure architectures need to continuously evolve as AI workloads grow • Building an engineering culture where teams collaborate, take ownership, and learn from one another Jim also shares a look at his personal passion for automation: building a fully automated home theater with custom LED lighting and event-driven controls. Tune in for a behind-the-scenes look at the infrastructure, engineering practices, and culture helping power AI at scale. Want to attend conversations like this in person? Register for an @Scale event here: atscaleconference.com [Source](https://www.youtube.com/watch?v=_DYIDfx4g_s)

[1:12] Francois Richard: 那场面可真是‘极低压力’啊。 [1:12] Jim Julson: Victims of their own success [4:34] Jim Julson: 不用提 PTO 申请流程，直接人间蒸发、把手机彻底关掉。 [7:13] Jim Julson: 这根本不是穷人的游戏，你是在完全不同的平流层里下注。 [7:13] Jim Julson: 如果问我妻子，答案是绝对没有；如果问我的同事，偶尔还能挤出一点。 [10:05] Francois Richard: 某个工具到底有多烂，或者为何完全无法实现最初预期的功能。 [12:17] Jim Julson: 彻头彻尾的噩梦 [17:10] Jim Julson: 因为我被即将面临的艰巨任务彻底吓到了。 [17:10] Jim Julson: It takes a village [22:32] Jim Julson: 执行写入操作至今仍让我感到害怕。 [32:41] Jim Julson: Act like an Owner [39:40] Jim Julson: 直接把几个 SLI 扔进 SLO 桶里不就完事了吗？ [39:40] Jim Julson: 真该骂自己，我本该更懂行的。 [39:40] Francois Richard: 指标变红时，系统一定是真彻底坏了。

[2:35] Jim Johnson: There's nothing new under the sun... People have been people for thousands of years. [2:35] Jim Johnson: Find a problem, fix a problem. [5:00] Jim Johnson: Forget about PTO and filing, just go disappear, shut your phone off. [8:10] Jim Johnson: If you ask my wife? No. If you ask my coworkers? Sometimes. [10:05] Jim Johnson: If you can't speak the language and understand it and really empathize with it... I think you're going to find it's a struggle. [17:10] Jim Johnson: I'm like, okay, surely there's automation, there's design. No. None... unboxing. [19:45] Congratulations, you did really well. Now do even more. [22:32] Jim Johnson: write scares me still. [29:45] Jim Johnson: probably the shortest life cycle from an architectural standpoint that I have personally ever seen on a backbone design. [29:45] Francois Richard: Yeah, we do love our bandwidth. [32:44] Jim Johnson: I've always assumed that if I found the problem, I need to fix it. [35:28] Jim Johnson: If somebody can help solve a problem, jump in and help solve it. Who cares who your direct report is? [37:17] Jim Johnson: Are we meeting the mark? Is the data that I see here aligning with the data that I see here? [39:40] Jim Johnson: shame on me, I should know better.

[0:00] people look up to heroes uh both in [0:02] fiction and in and in real life. And and [0:04] I think that like I can look at my team [0:07] of of you know 85 people today and I I [0:10] could pick out at least a dozen heroes, [0:13] right? And and they they change their [0:15] capes from time to time and switch and [0:17] uh recharge their batteries. But um I [0:20] think that the problem there is is you [0:22] know when they're going for as long as [0:24] they are and as hard as they are, it's [0:26] very taxing and and I don't think it's [0:28] enough anymore to throw money at the [0:31] problem. Yeah. [0:31] >> That only takes you so far for me. And [0:34] one thing I've learned over the last [0:35] year and a half, two years is finding [0:37] ways to emphasize keeping culture a [0:41] relevant topic. [0:43] Welcome back to the Atscale podcast [0:45] where we share stories of technical [0:47] innovation from top industry leaders. [0:49] Today I'm talking with Jim Jolson, VP of [0:53] global infrastructure networking at [0:54] Core, one of the fastest growing [0:57] specialized GPU clouds in the world. [0:59] Let's dive into our conversation about [1:02] building and operating AI clouds [1:04] including framework systems and NPI as [1:08] well as retaining top talent. [1:12] So being a podcast host is not my my day [1:16] job. So absolutely [1:17] >> being a podcast guest is not my day job. [1:19] >> It's not my day job. So I you know I at [1:23] the end of the day I kind of like run um [1:26] a part of production engineering. [1:28] Basically the people that do reactive [1:31] incident management and also disaster [1:33] recovery. [1:35] uh we run something called a storm [1:37] program where like once a quarter we [1:40] yank a region away and we test that all [1:42] the systems are are doing okay. [1:45] >> So very low stress low pressure. [1:46] >> Yeah. Very low very low low pressure. So [1:49] I I was like I was like you know I I had [1:52] tons of question about your people [1:54] quadrant especially in these in these [1:56] days where and and I you know I want to [1:58] start digging digging there about a bit [2:01] more about like the [2:02] >> people philosophy because you [2:04] highlighted a point that right now I [2:07] feel that everything is moving super [2:09] fast there's a lot of evolution there's [2:11] a lot of capacity coming online left and [2:14] right and then [2:15] >> we are relying on our people a lot And [2:19] there's a lot of work like if you are a [2:21] top performer like there's a lot of [2:23] pressure on you right now. [2:25] >> Huge. [2:25] >> And I was wondering like you know how do [2:27] you see this in your world? Do you do [2:29] you observe the same thing first and [2:31] then how do you make sure that you keep [2:33] these people sane? [2:35] >> Oh man that's a that's a loaded [2:37] question. [2:39] Um so so the short answer do do we see [2:41] this in our world? Yes. I mean, I like I [2:45] don't want to diminish it, but like I [2:46] feel like there's nothing new under the [2:48] sun, right? People have been people for [2:50] thousands of years. Um the problems that [2:52] we have are the same problems, just you [2:53] know, we're we're more modern today, so [2:56] uh the way they're out there is [2:57] different, but um I think you know the [3:00] the top performer problem, we call it [3:01] the heroes, right? [3:02] >> Yeah. [3:03] >> The we actually like praise our heroes. [3:06] We love our heroes because I know that [3:08] there's some companies that have a [3:10] philosophy. It's like we don't want [3:11] heroes because they get burned out. In [3:13] our experience, um the heroes are often [3:16] the ones that are leading the charge for [3:18] all of the engineers around them and and [3:20] people look up to heroes uh both in [3:23] fiction and in and in real life. And and [3:25] I think that like I can look at my team [3:27] of of you know 85 people today and I I [3:31] could pick out at least a dozen heroes, [3:33] right? And and they they change their [3:35] capes from time to time and switch and [3:38] uh recharge their batteries. But um I [3:40] think that the problem there is is you [3:43] know when they're going for as long as [3:44] they are and as hard as they are, it's [3:46] very taxing and and I don't think it's [3:49] enough anymore to throw money at the [3:51] problem. Yeah. [3:52] >> That only takes you so far because when [3:54] somebody's drowning in work 16 hours a [3:57] day, six days a week, even 5 days a week [4:00] that [4:00] >> they don't who cares if you throw them [4:02] >> Yeah. [4:04] Yeah. They're not there's no quality of [4:05] life. So I think the for me and one [4:08] thing I've learned over the last year [4:09] and a half, two years is finding ways to [4:12] emphasize keeping culture a relevant [4:15] topic and making sure that like we're [4:17] fostering that culture and making people [4:20] accountable. Find a problem, fix a [4:22] problem, right? Y [4:23] >> so that um they can spread the load [4:25] across the people and it's not all the [4:27] heroes doing all the work. And that [4:28] itself is a full-time job for the [4:30] management team of all those different [4:32] silos and Yeah. And are you observing [4:35] also like like you say like the the [4:37] heroes are pushing things forward. [4:39] They're the inspiration but they often [4:41] are the ones that are getting five times [4:44] done more of 10 times in some cases. Are [4:48] you observing the same thing? [4:49] >> Yeah. Absolutely. Victims of their own [4:51] success. Yeah. [4:52] >> Right. Um [4:54] >> yeah I you know I I don't know if it's a [4:56] good thing or a bad thing. Uh but it's a [4:58] thing for sure. [5:00] >> Yeah. I I I am trying to also balance my [5:03] EOS and like I haven't found like any [5:06] any miracle solution but like it's [5:08] almost like I'm at the point where you [5:12] know there are period of times where [5:13] we're about to relieve some release [5:16] something XYZ and then some heroes are [5:21] like upfront and then after a while I [5:24] have to kind of like not make them work [5:27] on the Lex release and kind of like do [5:29] more regular daytoday and kind of like [5:32] balancing these things out about there's [5:34] some [5:35] >> high intensity times and then there's [5:37] low intensity times. [5:38] >> Yep. [5:39] >> Yeah. Yeah. [5:40] >> Yeah. Very well said. I I think another [5:41] like a one tactic that I've used as well [5:43] is I mean it's not it's not rocket [5:44] science but um I will go and talk to [5:46] these these folks individually and just [5:49] tell them like hey for the next two [5:50] three months I want you to take a at [5:53] least a 3-day weekend twice a month [5:55] >> if not a 4-day weekend or three-day [5:58] weekend like all this month and next [6:00] like [6:00] >> forget about PTO and filing just go [6:02] disappear shut your phone off and um [6:05] that that helps a little bit but like [6:07] they just come right back to the chaos [6:08] so I think it always comes back for me [6:10] it comes back to like managing [6:13] the the problems at hand. Like what are [6:15] the problems causing the heroes needing [6:17] to work? Why is it that these guys are [6:20] the only ones engaged at this level so [6:23] often? And and when we start looking at [6:25] that, I think that there's a combination [6:27] of um I think ownership you heard me [6:29] mention earlier. Yep. [6:30] >> When you have people that don't feel [6:32] like they have that ownership over a [6:34] part of the stack, it means that the [6:37] person that is responsible for all that [6:39] is just constantly in this state of like [6:41] like firefighting over and over back and [6:43] forth and there's no real time to [6:45] innovate and you need the heroes to to [6:47] help your your very senior staff more [6:49] often than not to help lead the charge [6:51] on that innovation, right? And if [6:52] they're always just fighting fires and, [6:55] you know, helping everybody else out, [6:56] they're not really pushing anything [6:57] forward. So you spoke about fighting [7:00] fires and I've looked a bit at your [7:03] career from digital ocean now at [7:05] coreweave. Do you feel like fighting [7:08] fire has changed especially in the last [7:11] two years? [7:13] That's a heavy one. Um [7:16] yes I I I don't think the problems are [7:18] any different. Like in the panel just [7:20] now Jatendra was up there and he said [7:23] yeah like there's no new real problems [7:24] or no sorry this was Andrew from Oracle. [7:27] He was saying that there's no real [7:28] problems in networking. Like it's all [7:29] the same stuff in a IML, [7:31] >> but it's exacerbated. [7:32] >> Yep. [7:33] >> And it's faster. Yeah. [7:34] >> Yeah. It's faster. It's it's more it's [7:36] more the stakes are higher is is I think [7:38] the problem. [7:40] >> So I I think it has changed in the sense [7:42] that because the stakes are so high [7:44] because the cost of ownership of these [7:46] GPUs, the price tag is it's not a [7:49] >> Yeah. [7:49] >> poor man's game. [7:50] >> Yeah. it it is uh you're playing in a [7:53] different stratosphere and and the [7:54] expectations around that and the [7:56] infrastructure quality your meanantime [7:58] to response meanantime to detection [8:00] those things matter significantly and [8:02] and so firefighting in in a IML [8:05] workloads is definitely high pressure [8:07] high stakes and high visibility [8:10] >> did you had to adapt some of your [8:13] personal style to to handle like that [8:17] different type of pressure [8:19] >> uh definitely yes I I did not arrive [8:21] here as a polished, seasoned [8:24] professional. No. [8:25] >> Um I have failed forward. Um you know, I [8:28] I I give 100% credit like yes, I've [8:31] shown up and I've done the work, [8:32] >> but I give 100% credit to my team behind [8:34] me, right? Like they're the ones that [8:36] really like dayto-day. Um they've shown [8:39] me what it actually means to to to do uh [8:42] what needs to be done. And so I try I [8:45] try to model that sense of ownership and [8:46] that and get that down the stack, if you [8:49] will. um and then you know model what it [8:52] means to be a technical leader and and [8:54] have a vision and um whether that's [8:56] through people or technology uh I try to [8:59] do all that and then I think the the [9:01] result is you get a bunch of engineers [9:03] at a senior level that do the same thing [9:04] and model that to more juniors and [9:06] >> I think that's been my journey is [9:09] learning that as as I grow from the [9:12] architect yeah [9:13] >> um and and the and one of the engineers [9:15] into senior leadership it's it's more [9:18] about taking all of your collective [9:21] experience and learning to read the room [9:23] and react to the situation at hand in a [9:26] way that models it for everybody else to [9:28] to attune to. [9:29] >> Do you still have time to to go [9:31] technical deep in some area? Like do you [9:35] have like do you have the opportunity? [9:37] >> Um do I have time if you ask my wife? [9:40] No. [9:40] >> No. Yeah, for sure. If you ask my my [9:42] co-workers sometimes um I I spend a you [9:47] know it's it's it's every I think senior [9:49] leader journey is like you spend a [9:50] significant amount of time in Zoom or [9:52] Google Hangouts whatever your flavor is [9:55] >> uh in meetings. Um you're needed [9:57] everywhere. Um I I I have always kind of [10:00] prided myself in that. Um, I I have the [10:02] ability to say no. And I and I do say no [10:05] a lot uh based on judging whether or not [10:09] my value ad is actually real value ad to [10:11] the company um not just to that call [10:14] because there are plenty of initiatives [10:16] that that I'm spread across that where [10:18] my input does matter. And so I try to [10:20] balance that out and much of it is [10:22] technical. So [10:23] >> but that's those those days are changing [10:25] and and I'm further and further pulled [10:27] away as we get bigger and bigger which I [10:29] feel is a very natural progression. [10:30] >> Yeah. Yeah. I I'm still like attached to [10:33] like having some debugging skills and [10:37] some digging. Oh, yeah. I love it. I [10:38] love it. I I do enjoy it. [10:40] >> Yeah. [10:41] >> And and it allows me to do like [10:43] validation about like, oh, that tool [10:46] sucks or like it doesn't work well or [10:49] doesn't do what I'm expecting from the [10:51] get-go. and then allows me to kind of [10:54] stay but it's finding the time finding [10:56] that two or three hours in a row where I [10:59] can sit on a real life production [11:01] incident and actually like debug with [11:03] people I enjoy it but it is it is hard [11:06] it requires a lot of discipline and [11:09] >> I mean so so do you feel like it's [11:10] important for for leaders to remain [11:13] technical [11:14] >> I do feel like especially when you know [11:16] we were discussing recently about like [11:18] you know how we set oursel we set goals [11:20] like you spoke about NPI is like we have [11:22] our own NPI goals about like taking that [11:25] much amount of time. [11:26] >> Mhm. [11:27] >> And if you haven't sat down and kind of [11:29] like understood and executed some of the [11:32] step maybe you will set the goals in a [11:34] very unrealistic fashion that even the [11:37] speed of light could not achieve. So I [11:39] find it like important to get a sense [11:42] but not doing it in a critical pet. [11:46] >> Okay. Wow. Yeah. I I I couldn't agree [11:48] more. I mean like being being attached [11:51] to the technology like how can you [11:52] really effectively manage a technical [11:54] team? Yeah. And not really like I don't [11:56] want to disparage anybody who's a people [11:57] manager like solely that's not my my [11:59] goal but like I do feel like in the a [12:02] IML space in general like more than [12:04] anything [12:05] >> to manage the teams and the technical [12:07] expertise like if you can't speak the [12:09] language and understand it and really [12:11] empathize with it like [12:13] >> uh I I think you're going to find it's a [12:15] struggle. That's that's my take anyways. [12:17] So, [12:17] >> so you spoke about also so cover a [12:20] little bit of a people and I was [12:22] interested in in your concept of [12:24] framework which basically sits between [12:26] people and system type of thing. It's [12:29] kind of like a little bit like the [12:30] policies or the guidelines. [12:34] >> Give me some example of like frameworks [12:36] that you do use every day in kind of [12:39] like network operation that kind of [12:41] helps you guys. Yeah, I I think like the [12:43] the quickest thing I could point to is [12:45] like the system side frameworks that we [12:47] develop, right? So we we have event [12:49] driven automation is kind of like it's [12:51] buzzwordy, I know, but like it's it's a [12:53] thing. Um [12:54] >> the the frameworks that we've developed, [12:56] right? Like everything that we try to do [12:57] from our dev teams is is, you know, we [13:00] hire true developers so that they can [13:02] write API backends. Um in some cases we [13:05] even do web frontends depending on the [13:07] tooling that we're trying to build, [13:09] right? Um, but the API backends, right, [13:11] they're the interface that all the [13:13] engineers have into managing the network [13:15] at scale because it's not realistic to [13:18] try and manage each of these sites [13:19] through ginger templates and, you know, [13:21] that's nightmare [13:23] >> field. [13:24] So, so uh we we have uh event- driven [13:27] automation uh API uh frameworks that [13:30] have been developed that allows us to to [13:32] operate the entirety of all of our sites [13:34] holistically um through the API, right? [13:37] Right. And so these frameworks are the [13:38] the design is that you know you should [13:40] always be using eventing first. Yeah. [13:42] >> And as part of a policy um and then uh [13:46] leveraging the frameworks that do exist, [13:48] you know, where possible. And then [13:50] outside of that scope is where you get [13:51] into like the nuance of like, oh, this [13:53] is a one-off thing that we did, so maybe [13:55] it doesn't apply here or there, stuff [13:57] like that. [13:58] So [13:59] >> when you when you talk about like like [14:01] I'm curious about like when you say like [14:03] um kind of like event based automation [14:07] um [14:08] can you share a little bit more like [14:10] when coreweave as an example decide like [14:13] oh we going to start like you have your [14:15] buildings ready and kind you have [14:17] network connectivity but you're going to [14:18] start provisioning a new site we call it [14:20] site turnup. [14:21] >> Okay [14:22] >> you probably have a a very similar term [14:24] >> bootstrap. [14:25] >> Yeah bootstrap. [14:26] >> Yeah. How do you do it with an [14:29] event-based automation? What's the [14:31] trigger? Like, you know, is there a [14:32] human that start putting like the core [14:34] of the network there and then the rest [14:36] builds up? Like what's the trigger at [14:38] that point? [14:38] >> Oh, you're okay. So, so yeah. So, like [14:41] this is an important distinction, right? [14:43] Um that it's like sometimes there's so [14:45] many automation terms that are [14:46] overloaded. [14:47] >> Uh where we do event- based automation [14:50] is typically on the operational side. [14:52] Okay. So the new build side um we we [14:55] have a very robust automation framework [14:57] built uh we we we have a a program [15:00] called the MCP the master control [15:02] program. [15:02] >> Yeah. I was also like you guys reuse the [15:04] MCP term. [15:05] >> Yeah. Well this was before it got blown [15:08] up like it's been around for for us it's [15:10] been years but um it's the master [15:12] control program from Tron. That's that's [15:14] that's actually like the you know us [15:16] network engineer types we always have to [15:17] have a a theme for the naming right. Uh [15:20] so anyways this is the API framework [15:22] that exists on a per site basis. It's [15:25] distributed right and then it allows you [15:27] to basically interface with the network [15:29] uh programmatically uh via API and it [15:31] handles all of the bootstrap and [15:33] provisioning. It provides the config [15:35] state drift uh reporting. It allows us [15:38] to capture and maintain all of the [15:40] burndown lists that we build for like [15:42] link remediation during build [15:44] >> because uh before we go production [15:46] obviously we're validating the state of [15:48] the fabric. Hey, there's, you know, [15:50] 256,000 links expected to be up. Of [15:52] that, 144,000 are in build or whatever [15:56] the case may be. But, um, that is the [15:58] application stack that handles that for [15:59] us. So, we're less vented on that side. [16:02] >> Okay. On that side, okay, [16:04] >> working towards it though, for sure. the [16:06] the real eventing is on the operational [16:08] side which is like hey you've got 20 [16:11] links that are flapping across your you [16:13] know 7,000 rocky node fabric on the back [16:16] end and we'll go through and gracefully [16:18] drain the control planes so that the [16:20] training jobs will continue to run right [16:22] could be errors on port it could I mean [16:25] any number of things uh BGP flapping [16:27] things of that nature [16:28] >> okay yeah I I'm I'm always curious like [16:31] somehow my team got involved a lot in [16:34] what we call like new region turnup Oh. [16:36] >> Um, the main reason is that we operate [16:39] the the the rest of the network is live [16:43] and you can imagine like there's a lot [16:45] of subsystem like cache and load [16:47] balancing that as soon as you turn [16:49] something up they will be broadcast and [16:51] everything the traffic's going to start [16:52] flowing and it creates production [16:54] incident when you're not expecting it. [16:56] >> Yes. [16:56] >> So we're involved a lot into uh the [17:00] buildout of we call them new regions and [17:03] things like that. I was curious about [17:04] that. [17:05] >> Yeah. Yeah. It sounds like we're [17:07] probably sharing some of the same [17:08] >> Yeah. [17:09] >> same burdens there, same pains. [17:10] >> Yes. Yes. And and I I I'm a bit curious. [17:14] And then we can cut that information if [17:16] you don't want to. Like I'm curious [17:18] about like at Core Weave like how many [17:21] sites can you guys like build up at the [17:24] same time? Like do you have like a a [17:25] sense or so? Well, I'll tell you. When I [17:29] started in February of 2023, there's a [17:32] whole story I won't get into about like [17:34] how I was on boarded. It's kind of [17:35] hilarious. Um, one of the the [17:37] co-founders like I was going to start [17:39] like mid-March and he flew me out to Las [17:42] Vegas in the third week of February and [17:46] by the time the by the second day I was [17:48] there, my start date was that day. [17:50] >> Okay. [17:50] >> Because I was so horrified about the [17:52] task at hand and what was coming. Um it [17:55] was to stand up basically five data [17:58] centers inside of about 90 days. [18:00] >> Okay. [18:00] >> And yeah [18:01] >> and so I you know I'm like okay I'm [18:03] surely there's there's there's [18:04] automation there's design. [18:06] >> No [18:07] >> no no [18:07] >> none. Um and so [18:11] >> unboxing. [18:12] >> Yeah. And so so back then we we we came [18:15] close. We did we did four in 90. The [18:17] fifth one came a month after. [18:19] >> Um so we were just barely off the mark [18:21] which was an impossible task to begin [18:23] with. Um but we did those and that was [18:25] with four of us. Okay. So I had [18:28] immediately hired a team of three [18:30] others. Um and we went right to work and [18:32] and we got the job done. Today we are [18:36] the thing is is those data centers were [18:37] like five megawatts in size if that. [18:40] >> Okay. [18:41] >> Which we don't even bother with that [18:43] anymore. Like it's it's so minuscule. [18:46] Now we are building three and four data [18:49] centers at the same time at 250 megawws [18:52] or 200 megawws and so uh it takes a [18:56] village. [18:56] >> Yeah, [18:56] >> it takes a village. [18:57] >> It does. [18:58] >> It does like like like Yeah. And the [19:00] same goes from Facebook and Meta. We [19:02] were able to do I think you know one or [19:05] two in parallel and now we're doing more [19:07] than five or six in parallel and [19:09] >> which is insane. [19:09] >> It is insane. Yeah. [19:10] >> It's insane. And they're not small. [19:12] >> They're not small. [19:14] >> Yeah. the [19:16] we went almost too far on the [19:18] paralization problem that that we like [19:21] you you talk about like your your [19:22] initial footprint. We're working at [19:25] reducing our initial footprint because [19:27] like [19:28] >> it's just too much capacity when we get [19:30] in which is an interesting problem [19:32] because you have like an amount of [19:33] service and basic services like [19:36] >> you need your core network, you need [19:38] your storage, you need all of that and [19:39] then we're trying to shrink it so we can [19:41] do more in parallel to require less [19:43] capacity during the buildout so that the [19:45] capacity can be used for production [19:46] load. [19:47] >> Congratulations, you did really well. [19:49] Now do even more. [19:50] >> Yeah. Yeah. Do even more. So that's the [19:52] uh Yeah. Yeah. So it's it it is [19:55] constant. It's I have not seen that [19:58] evolution like for a while. [20:01] >> Yeah. And I I think the other thing [20:02] about that it's interesting, right, is [20:04] like [20:05] >> not a lot of people have this problem. [20:08] Like there's only a few of us in the [20:09] industry right now building at this [20:11] scale, at this speed, at this at this [20:13] size that that like are having to solve [20:15] some of these problems. And and so [20:17] there's no like prior art to really pull [20:20] from. Uh which I I find interesting [20:22] because there's it's just a constant [20:23] like struggle. [20:25] >> Yeah. Know it is new. It is it is very [20:28] new like we had to really get organized [20:32] really get focused you know we have [20:34] goals of like timeline for like what I [20:37] call like a site turn up like we have do [20:40] you hit the goal and then yeah those are [20:42] crazy. Um coming back on your [20:46] presentation, you also talk about [20:48] systems. [20:50] Can you describe maybe like a key system [20:54] for core weave? Like you spoke about [20:56] like your kind of like network API [20:59] systems like have you guys come up with [21:02] other system to kind of help you in the [21:05] dayto-day? I you know like I I maybe [21:08] it's because it's so front and center in [21:11] my world day-to-day but like the [21:13] observability and telemetry stack. Um so [21:16] we have a dedicated like we have a [21:19] dedicated observability team for the [21:20] company right and they have a huge [21:22] charter um really really challenging a [21:25] lot of metrics to ingest but um I have [21:28] my own observability team uh inside the [21:30] network or and and their charter is they [21:32] are consumers of the infra that the [21:34] >> other they use the key value pair store [21:36] and all of that and yeah the time series [21:38] store and [21:39] >> exactly but but they're responsible for [21:42] translating the visions that all of the [21:44] engineers have into reality. from a like [21:46] here's how we view health at coreweave [21:48] right like whether it's our solo [21:50] dashboards or whether it's our [21:51] independent troubleshooting dashboards [21:53] like that team is responsible for that [21:55] so they that that is a framework that is [21:58] not an off-the-shelf um singular [22:01] solution it's a collection of [22:02] open-source solutions but with a lot of [22:05] custom code um we call it our echo stack [22:08] um it's yeah echo [22:10] >> um it's pretty cool actually really cool [22:14] um we've we've partnered with companies [22:16] like Nvidia uh to to really make sure [22:19] that like as we're developing if there's [22:21] opportunities to like hey you might want [22:24] to consider putting X into your [22:26] ecosystem like we've we've got some of [22:27] those lines of communication. It's been [22:29] really helpful to partner with them. So [22:32] >> um in terms of like kind of like you [22:34] know monitoring observability [22:37] um do you feel that you needed like I'll [22:41] explain what I meant like in order now I [22:45] have couple of different type of people [22:48] making I would call it like production [22:50] change I have human making production [22:53] change I have system but I also have [22:55] agent [22:56] the humans are highly unpredictable but [22:59] they have they are more careful The [23:01] systems are very predictable that are [23:03] like everything around automation. The [23:06] agents are in between. Did you had to [23:09] build something special to observe what [23:11] the the agents are doing versus other [23:14] entities? [23:15] >> We're I you know we're pretty I think um [23:19] bearish if you will on on just like [23:22] letting loose agentic AI into the fabric [23:25] right now. [23:26] uh we we do make use of agentic um tools [23:29] that we've developed in house. Uh most [23:31] of it is for readon operations today. Um [23:34] I think that you know because of the [23:37] exact thing you just stated that is why [23:40] we have not been quick to just like drop [23:42] something in production that's [23:43] responsible for correlating data across [23:46] switches and then pulling things out or [23:49] you know generating workloads to to test [23:51] for synthetics or things like that. We [23:52] we've definitely talked about it. Um but [23:56] because we are moving at break neck [23:58] speeds already and can like hiring the [24:01] right people is the other side of this [24:03] challenge um that I didn't really talk [24:05] about but it's you know without those [24:08] people to focus on just that thing I [24:10] think it would be more of a detriment [24:11] than a benefit at this point to to go [24:14] write operations read operations cool [24:16] but like [24:17] >> yeah yeah yeah [24:17] >> right scares me still [24:19] >> yeah it does it does we had to build [24:21] like all sorts of things like you can [24:23] imagine [24:25] identity carrying the identity around [24:27] these things like [24:28] >> usually automation they have some sort [24:30] of service identity that you can [24:32] identify where it's coming from [24:34] >> human there's the kind of like the [24:36] humanbased certificate and all that you [24:38] can understand it's a human [24:40] >> usually an agent is acting on the behalf [24:42] of a human that's doing something else [24:44] so you have to carry both identity the [24:46] identity the the fact that it's an agent [24:48] yes [24:49] >> what the agent is and you have to carry [24:51] also the fact that they what who's the [24:53] human behind it. [24:54] >> Oh, [24:55] >> so from a troubleshooting perspective, I [24:57] need to know that oh, this modification [24:59] on this mutation has been done by an [25:01] agent on the behalf of that human. It [25:03] transited via XYZ [25:05] and then we had to get get into the the [25:08] point of I would call it like logging [25:10] intent like what did the human ask the [25:14] agent to do? What were the assumption? [25:17] What was the context of the agent? So it [25:19] becomes like a an entire [25:23] discipline of like observability that we [25:25] didn't have and we had to build that [25:27] like on the fly very quickly. [25:29] >> It's like a heristics exercise where [25:30] you're trying to predict like what's [25:33] what was what as part of that whole [25:35] chain. [25:36] >> Exactly. And I'm I'm always curious [25:38] about like how other big hyperscaler [25:41] have done this because I feel we're like [25:43] we are inventing this on the fly as we [25:46] are going. I mean you guys generally [25:48] like you are the inventors of quite a [25:50] bit of your size. Yeah. So [25:52] >> yeah. I think I I mean just like also [25:54] calling out right like we're we're we're [25:56] like 4,000 people [25:58] >> uh and and like 6 months ago we were [26:00] like 3,000 people. So so we're I think [26:02] you know we're still like definitely on [26:04] a journey and and there's other teams [26:08] outside of my own right that are [26:09] employing agentic models as part of [26:11] their workflows. Definitely. And um I'm [26:13] not really well equipped to speak on [26:15] that, but I will say like there's an [26:17] appetite for it. I just think that like [26:19] we've seen enough bad stuff happen to [26:21] where we're also like, you know, let's [26:23] be mature about like not biting off more [26:25] than we can chew here because we can [26:27] cause more harm than than good if we're [26:30] not careful with it. Like to your point, [26:31] like we're not equipped staff-wise to [26:33] like just go write code for that and [26:35] then have a team to to write code for [26:37] the code, right? Yeah. [26:38] >> So it's it's a real challenge. Like a [26:40] real challenge. So Changing gear a [26:43] little bit. Capacity planning. [26:44] >> Oh yeah. [26:45] >> From a network perspective like what's [26:48] the latest and greatest like can you [26:50] like share like you know recent problem [26:52] about network capacity estimating that [26:54] this type of workload will require X and [26:57] then you figure out like no this is X [26:59] time 10. [27:01] >> Yeah. So this is along the lines of like [27:03] NPI. It's an NPI kind of problem for us. [27:05] So, so prior to um we we've had some [27:08] really great hires like I I I was [27:10] fortunate enough I've always attached [27:12] myself to like a mentor like throughout [27:14] my career. Um and I I'm I'm going to say [27:17] it's been about 10 12 years now. Um I [27:19] had a mentor uh I'm going to name drop [27:20] him. His name is Shiv Patel, right? And [27:22] he comes we worked together back in the [27:24] day at the Dell Virtue Stream ecosystem [27:26] and then we parted ways and I went off [27:28] elsewhere Digital Ocean at the time and [27:30] and he ended up at Microsoft. Well, I [27:32] got to to to uh hire him [27:35] >> and he is uh our our you know the the [27:38] NPI guru. Everybody absolutely adores [27:41] him. [27:42] >> Um he's really good at what he does and [27:44] one of the first things that he did when [27:45] he came in around capacity planning [27:47] seeing that it was such an intense [27:48] painoint um with with NPI specifically [27:52] was he actually went out and used Claude [27:54] Yep. [27:54] >> to go build a full calculator and and [27:58] >> like kind of like a model type of thing. [28:00] >> Yeah. So basically network modeling as a [28:02] service, right? And so he's uh he's got [28:04] a few different iterations of things [28:05] that he's done through through cloud and [28:07] uh Google Sheets as well, but like just [28:09] like basic things like that can provide [28:12] huge value, right? Being able to go [28:14] model like n number of GPUs of a certain [28:16] type uh gives you this many switches [28:19] that you need for your T0, your T1's [28:21] your T2s and so on. Um same thing for [28:23] like oh CPU workload looks like X so we [28:26] need Y. [28:26] >> We need Y in terms of networking. [28:29] >> Yeah. Okay. [28:30] >> Like it it's it's funny. It's not like [28:31] it's an elegant simple solution to a [28:33] very complex problem, but like it it is [28:36] a solution still like and it totally [28:37] worked. So yeah, that's that's probably [28:40] our most recent one and and around like [28:42] the Vera Rubin product line coming up. [28:45] So are you guys running into like Okay, [28:48] so you need to link these I don't know [28:51] if you call them data centers or region [28:52] together. Are you guys also like running [28:55] into like backbone type of issues in [28:57] terms of like capacity planning into [28:59] like getting the right fibers in the [29:01] right place? [29:02] >> Oh yeah. Oh yeah. Um so okay. Yes. So we [29:05] do call them they are regions. Yeah. And [29:07] we have availability zones inside [29:08] regions [29:09] >> inside regions. [29:10] >> Yeah. And so um we we do have a global [29:12] backbone. We have a global backbone [29:14] team. And um we hired for the longest [29:18] time we had just partnered with a third [29:20] party to provide us all of our wave [29:23] transports. Um so whether it was transit [29:26] or transport, right? Um they were just [29:29] third party brokering all of it for us. [29:31] >> They would also handle the knock side of [29:33] the house. So all maintenance [29:34] notification [29:35] >> maintenance and turn up and yeah, [29:37] >> which was awesome until it wasn't. And [29:40] it was by no fault of their own. They [29:41] were amazing partners to us, but they [29:43] were just never structured to handle the [29:45] size that we became with the number of [29:48] lit waves and dark fiber that we had all [29:50] over the globe. So, we hired our own [29:52] fiber capacity planners in-house. [29:55] >> And um you know, we've we've that team [29:58] is I mean, they are 24 by7 it seems like [30:01] just non-stop on the go with this stuff. [30:03] Um but we we're entering a phase now [30:07] where what we built [30:08] >> two and a half three years ago is not [30:11] sufficient for where we're at today. [30:12] >> Yeah. [30:13] >> That is probably the shortest life cycle [30:15] from an architectural standpoint that I [30:17] have personally ever seen on a backbone [30:18] design. [30:19] >> Um it like what's built just isn't going [30:22] to handle what we need to do. And so [30:24] yeah, we're constantly playing catch-up [30:27] on the architecture design side just for [30:28] the backbone just to scale it properly [30:30] and support the software uh software [30:33] feature sets that are being asked of us [30:34] from a product side and a customer [30:36] integration side as well. Uh but then [30:39] also the capacity planning around these [30:41] data centers you know I mean I can use [30:44] you guys you guys you guys love your [30:46] bandwidth. Yeah. Yeah, we do love our [30:47] bandwidth. [30:50] >> And uh they're not it's not normal [30:52] numbers for most data centers or or [30:54] providers. So the planning and the time [30:57] horizons for that stuff is significant. [30:59] And so uh we're finding that that we [31:03] need to get in even earlier than we [31:04] anticipated. And when often times these [31:07] contracts aren't even signed until we're [31:10] about to take whites space over. So it's [31:12] it's a constant delicate dance of like [31:15] hey we got a plan for X but it might be [31:17] Y and and we're we're deep in that game [31:20] right now for sure. The are you guys [31:23] like so so you spoke about backbone and [31:25] you have like your internal network [31:27] probably are you getting use cases where [31:30] people are starting to kind of asking [31:32] you guys hey I need like an edge network [31:35] like which is customer front not data [31:37] centered like us to you but like kind of [31:39] like more like hey I need to start [31:41] serving stuff I need to have like [31:42] inference that serving so you probably [31:44] also need to start building that [31:46] >> so we have that actually yeah we've got [31:48] a pretty robust so when I say backbone [31:50] for us it's just like We have the [31:52] transport network which is your sight to [31:53] sight. Okay. [31:54] >> Uh that's that's a global east west [31:56] network. Um and then we have at every [31:59] one we call them transit pops, right? So [32:00] the transit pops are strategically [32:02] located in facilities that have um very [32:04] specific um carriers. Okay. Uh located [32:08] Yeah. So they we optimize for for [32:10] peering ex basically and transport uh [32:13] internet transit, forgive me. Um and and [32:16] make sure that we can do things like [32:17] direct connects with customers at these [32:19] locations since they're super popular, [32:20] right? So, um, yeah, we have I I don't [32:24] know what the latest count of is like [32:25] 40ome transit pops globally. Okay. [32:27] >> Um, that number is I think we've got [32:29] eight to 10 on the back burner right now [32:32] about to come up in the next 6 to 8 [32:33] months. So, yeah, lots of lots of [32:37] transit, lots of transport coming [32:38] online. [32:39] >> Great. [32:39] >> Lots of long haul dark fiber, too. [32:41] >> Yeah. Yeah. Um couple of cultural [32:44] question I want to know a little bit [32:46] about like like you've been in digital [32:48] ocean you've been other places before [32:51] what is specific that you would say like [32:53] about the culture of carweave in terms [32:55] of like engineering culture and how [32:57] people think about stuff and [33:01] that's a good question um let me start [33:04] by saying so like I came up at a time [33:07] where like even Google was fairly new [33:10] like this I'm dating myself a a little [33:12] bit, but like [33:13] >> engineering was like you find a problem, [33:15] you fix a problem, and you don't [33:16] necessarily have somebody to go to to [33:18] find an answer for that problem. And so, [33:20] like the reason I bring this up is cuz I [33:22] I I've kind of like psychoanalyzed [33:25] myself a little bit about why I am how I [33:27] am and what makes me tick. And I think a [33:29] part of it is like I came up in that [33:32] time where like the sense of ownership [33:34] to problems was a very important aspect [33:37] of career growth. And I've carried that [33:39] with myself um through the entirety of [33:42] my career, right? I've never ever just [33:44] assumed somebody else would just solve [33:46] the problem, right? I've always assumed [33:47] that if I found the problem, I need to [33:48] fix it. I've been very fortunate to be [33:51] around an like an immense number of [33:54] amazing engineers and um technical [33:56] leaders throughout my career uh that [33:58] have had that same mindset. And so I [34:01] think a big piece of like where we're at [34:04] today with coreweave is is we've managed [34:06] to retain a lot of that cultural um like [34:10] we really put a lot of emphasis on like [34:12] act like an owner is an actual core [34:13] weave core value but like internally on [34:15] the network side like it's it's [34:17] something that's super core to who we [34:19] are individually and so like in our org [34:22] of of 80 some odd people today [34:24] >> I I feel like everybody there really has [34:27] a strong sense of ownership. They feel [34:28] like they can operate with autonomy. [34:30] Yeah. Because of that, which makes them [34:32] that much more inclined to dive deeper [34:35] into the stack and solution things that [34:37] have real meaning and value to everybody [34:39] around them. So [34:40] >> get ever had the chance to read the book [34:42] Extreme Ownership. [34:44] >> I I have not I have told I need to read [34:46] that book. [34:47] >> You're now like the third person. [34:48] >> It will it will like if you are like you [34:51] know if it's in your DNA. [34:53] >> Yeah. The book will explain a lot of the [34:57] I would call it like yeah you need to [34:58] own your destiny. You need to take [35:00] charge. You need to like take control [35:02] and you need to take action. [35:05] >> But it explains a lot also around the [35:08] the side effect of the psychology of [35:11] doing so. And I was like ah okay. And [35:14] then you keep on doing it. You keep on [35:15] reinforcing it. So like yeah we did [35:17] that. [35:18] >> You're like the third person that said [35:19] >> as a team exercise is very good to do. [35:22] like it's it's a very good book and [35:24] awesome. [35:24] >> Yes. Yes. [35:25] >> Okay. Yeah. I I should probably take a [35:27] hint. [35:27] >> Yeah. [35:28] >> Um any other cultural values that would [35:31] be interesting? [35:33] >> Yeah. I I mean I like it's it's hard, [35:35] right? Like all of our teams I feel like [35:38] in the in the engineering world, right? [35:40] Like software engineers are are [35:42] functionally different than like [35:43] hardware engineers. They they each bring [35:46] something very unique to the table. But [35:47] I think one thing that we we've still [35:50] retained is even at 80 some odd people, [35:54] you can still go have a Slack [35:55] conversation in in one of our full team [35:57] channels that's private and have it very [36:00] like intimate conversation and and like [36:03] it it feels very familial, right? Um, it [36:07] I think some of that's going to slip [36:08] away as we get even bigger, but but like [36:10] I think with at the team level, at least [36:12] the dynamics today, everybody feels very [36:15] empowered to just go like shoulder tap [36:17] somebody, ask for help, ask for, you [36:19] know, guidance and and it's always [36:21] available regardless of who they report [36:23] to. And so I we've really tried to [36:25] foster that that notion that like, hey, [36:28] engineers are are there for you [36:29] regardless of what team they're on. If [36:31] somebody can help solve a problem, jump [36:33] in and help solve it. Who cares who your [36:34] direct report is, you know? Yeah. [36:36] >> And and that seems to be something still [36:38] very much alive and well. [36:40] >> Yeah. Yeah. I I tend to practice [36:42] something I call it like very similar [36:44] where [36:47] um we operate as a pool of senior IC's [36:51] and then we will direct them to to an [36:53] area of a problem and then we call them [36:56] like okay that would be the kind of like [36:57] your mission right now is to kind of [36:59] like execute on that and then like yeah [37:01] they build a relationship and the [37:02] friendship in the area uh to move things [37:05] around and then once it's done they [37:07] might go to another area and do this so [37:10] I And I can see like the similarity. [37:13] >> Cool. [37:14] >> Awesome. [37:14] >> Um I'll move us to a couple of quick [37:17] questions. Um if I was to ask you today, [37:20] and it doesn't need to be for work, it [37:22] can be at home. What's your favorite and [37:24] most impressive like AI tool or AI [37:27] usage? [37:29] Oh man. Um, so I built a a little agent [37:36] that lets me basically go through Slack [37:39] and Jira and okay, [37:41] >> correlate uh real-time events from our [37:44] logging system and the ticket creation [37:46] from our automation workflows. So I can [37:49] basically go through and look at the [37:51] alerts that have spawned and correlate [37:52] the times to see if we have gaps in any [37:55] of the alerting versus the ticket [37:57] creation versus the workflow. Um it's [37:59] been useful in some of the the growing [38:02] pains that we've had for the last uh uh [38:05] I'd say probably four months been pretty [38:07] rough on the teams like we we're [38:09] onboarding substantial infrastructure [38:11] and with all that comes all the fun pain [38:13] that you get. Um and until it's all [38:16] steady state and you've remediated that [38:18] initial build, it's just it's just this [38:20] constant roaring fire. [38:22] >> Um I don't use it as a it's not a it's [38:24] not a mechanism to like out anybody's [38:27] fault. That's not what it's for. It's [38:28] it's for me to basically go, are we [38:31] meeting the mark? Is the data that I see [38:33] here aligning with the data that I see [38:35] here? Uh is are we missing something? [38:38] Right? And I I think so far the good [38:40] thing is is like it feels like we are [38:42] doing pretty well. Um I I think that by [38:46] and large uh tools like that are are you [38:49] know it took me probably two and a half [38:51] hours to create it and get it working [38:53] right. [38:53] >> Okay, [38:54] >> that was it. uh but but I've gotten a [38:56] lot of value out of it for the last uh [38:58] two three months. [38:59] >> In your organization, are you guys using [39:01] like the concept of SLO, SLI? [39:04] And [39:05] >> it's funny you should ask. Yeah, [39:07] >> we we have an internal project right [39:09] now. Like I I I kind of freaked out [39:12] because I was like I started realizing [39:14] very quickly like if I go ask all of my [39:18] engineers on one team and then another [39:19] and then another like define health for [39:22] me at Corewave on the network [39:24] >> I'm going to get a different answer from [39:25] every engineer. That's a problem. [39:27] >> Yeah. So uh started the initiative to [39:30] define health at a organizational level [39:33] for every role of network device and [39:35] every like we call them the the pillars [39:38] which is like your front-end network [39:40] your backend network your management [39:42] network and then your uh your backbone [39:44] and then the services layer. So any [39:47] services we run all have SLOs's. Um all [39:49] the infrastructure has SLOs's and then [39:51] in those different pillars you have [39:54] availability, reliability, capacity etc [39:57] stuff like that. [39:58] >> So um we're we're heavy heavy heavy into [40:01] development [40:02] >> and building and figuring out what [40:04] metrics represent a good SLO like yeah [40:07] it is it is it is a journey. [40:09] >> I I would have like I oversimplified it [40:12] when I first like materialized some of [40:14] it in my head. Oh, just throw some SLIs [40:17] in in the SLO bucket and Yeah. And then [40:19] you started real it's like shame on me. [40:21] I should know better. Um but I did and [40:23] and and it was it's been really [40:25] enlightening like going through the [40:26] progress process and um it started there [40:29] was only like 10 to 12 people in my org [40:30] out of all 80 that were participating in [40:32] these calls and now there's like 35 to [40:35] 40, right? Which is a great signal that [40:37] people are taking health seriously and [40:38] they understand the implications and [40:40] >> it's uh it's crowd sourcing. The more [40:42] people we have on it, the better we're [40:43] going to be. So [40:45] >> I can tell you the it took us and we've [40:48] been at it for a good three years now [40:50] like defining key SLO for like the let's [40:54] call it like the big rocks the internal [40:56] service [40:57] >> like it took a good three years so that [41:00] the the SL actually becomes meaningful [41:02] like it's really represents reality [41:04] properly [41:05] >> like when it's red that means it's [41:07] really broken [41:07] >> it's really red and then do all the [41:09] correlation like we call production [41:10] incident sev like okay there's a sev [41:14] there's an alert. Was the SLO impacted [41:16] or not? And building that discipline [41:19] about like on every production incident [41:22] going back and figuring out did it [41:24] impact the SLO or not and then question [41:26] it. Like sometimes [41:28] >> with production incident that have no [41:30] impact to the end users or to to [41:33] anything like this like you know you [41:34] lose redundancy of something [41:36] >> for us internally. It is a production [41:38] incident but nobody sees it. It's great [41:40] but we it it needs to dip something in [41:43] the SLO somewhere otherwise there's [41:45] something that we are missing. [41:47] >> Yeah. So all of these things like it [41:49] took like Yeah, I I I probably made the [41:51] same mistake as you did about like [41:53] oversimplifying it and then as we were [41:55] going into like [41:58] >> okay this team has way too many SLO [42:00] versus this team has not enough SLO [42:02] they're not meaningful enough and then [42:04] keeping track and building the culture [42:06] that they're going to watch it they're [42:09] going to look at it they're going to [42:10] integrate it into their on call hand [42:13] yeah exactly so it takes time it takes [42:15] uh [42:16] >> I I I I'm like super glad you brought [42:18] that up because it makes me feel better [42:20] >> knowing that we're not alone on that [42:22] journey. [42:23] >> Yeah, it is not it is like it is more [42:25] subtle [42:27] >> and and the benefits that I've seen over [42:29] time is that when we are taking risk [42:34] >> turning up a new region faster or we are [42:38] doing that maintenance that was not [42:39] supposed to happen but we really need to [42:41] do it faster. We have these now these [42:44] kind of like error budget that we can go [42:46] and dip into [42:48] >> and it also give us like oh kind of [42:51] ammunition where oh this area they have [42:53] been focusing on something else for a [42:55] while and there are trending down so [42:57] okay now it's time to go back to the [42:59] basic and and fix the existing system. [43:02] >> Yes that resonates. [43:04] >> Yeah. Yeah. Yeah. So but it is it is a [43:06] long journey. Um do you guys have like [43:10] internal tooling for like production [43:12] incident or network incident management [43:14] that you guys have built? [43:15] >> Um I I you know I I we have an entire [43:19] process and team that's dedicated to [43:20] like we call them insies. Um so same [43:22] thing right and then they're assigned [43:24] severity levels. Um but like they have [43:27] entire frameworks that they follow [43:28] entire process and and then I think that [43:30] there is even some [43:32] >> tooling integration that happens behind [43:34] the scenes for like when the in's [43:36] created through Slack. Okay. [43:37] automatically creates a tracking Jura [43:39] ticket for the RCA and all that stuff. [43:41] So like the I think the the biggest [43:43] thing would be it's like it's a workflow [43:44] system. [43:45] >> It's a workflow system and then it gets [43:47] into [43:48] >> Have you guys built like a a process to [43:50] review some of the critical [43:53] incident? [43:54] >> Not that's outside like humans. I think [43:56] still it we're very human driven on that [43:58] front thankfully. Like I mean we have a [44:01] lot we have enough insies. There's no [44:02] doubt about that. But like I think that [44:04] from a network standpoint, like we're [44:07] surprisingly pretty good right now. Um [44:11] uh you know, knock on wood. But uh yeah, [44:13] I I don't I don't think um we have [44:15] anything too heavy-handed there yet. So [44:17] definitely probably something coming. [44:19] There's a separate team that manages all [44:21] that ecosystem and we partner with them. [44:23] >> You partner with them. So Okay, great. [44:25] Um, going back to um, AI tooling, quick [44:29] question, like for like you describe it [44:31] like a tool for for work. What about a [44:33] tool for like outside of work? That [44:36] would insinuate that I want to spend [44:37] more time uh, playing on my laptop after [44:40] I'm done for the day. Um, I I love what [44:43] I do. Uh, I love the people that I work [44:46] with now. I I I owe a great like I have [44:50] I have huge respect for like the [44:51] founders of Coree. like they they hired [44:54] me um and and like really believed in me [44:57] and like I I love working hard for them [45:00] and for myself uh and for my team but [45:03] when the day is done I want to close my [45:05] laptop and not look at anything [45:07] technical like that's why I'm doing a [45:09] movie room construction project at home [45:11] like I need that release so yeah no no [45:13] tooling for me today but maybe maybe [45:16] some other time [45:16] >> yeah can imagine like M assistant [45:19] requires a lot of API integration in for [45:22] uh movie. Uh [45:23] >> it's funny you say that. I am I I'm full [45:25] I'm doing a full open HAB setup. Uh [45:28] that's all tied into Home Assistant on [45:29] the back end with the API. I'm doing [45:31] full eventing myself like the whole bit. [45:33] So [45:33] >> there you go. [45:34] >> And I also am learning all of the LED [45:36] like the production scale LED lighting [45:38] systems since I put in custom LED [45:40] lighting into the theater as well. [45:42] >> Oh, like for like accessory lighting or [45:45] like full-blown lighting? [45:46] >> It's all of the above. [45:47] >> All of the above. [45:49] >> All of the above. It's It's a full-blown [45:51] like uh automated system that has timers [45:55] and uh Q lighting and all kinds of [45:57] stuff. [45:57] >> Wow. So [45:58] >> impressive. Yeah. Okay. [46:00] >> Yeah, I that's like if I'm going to nerd [46:02] out, I'm going to nerd out on something [46:03] that's like I'm going to enjoy with the [46:05] family. [46:06] >> I'm doing something similar. Maybe it's [46:08] not I'm redoing some lights in the [46:11] house. [46:11] >> Okay. Okay. And I've been in going in [46:13] the full spectrum tunable white with [46:16] colors and everything like this that she [46:18] can actually control and automate. Yeah. [46:21] Yeah. I know. [46:23] >> As long as I can retrofit the cans. [46:26] >> Yes. [46:27] >> And I can connect wife, I'm good. [46:28] >> I love it. I love it. It's good stuff. [46:30] >> Yes. Yes. Yes. Great. All right. Well, [46:32] that concludes pretty much all my [46:34] questions. So, thank you. It was great. [46:36] Thank you for sharing. [46:37] >> Yeah. Cheers. [46:38] >> Good to meet you. [46:39] >> Thanks for listening. I hope you enjoyed [46:41] the conversation as much as I did. If [46:43] you're interested in connecting with [46:45] more infrastructure and product [46:46] engineers in person, join us for our [46:49] next atscale event, product at scale in [46:52] Menllo Park on October 28th. We'll hear [46:54] about the latest AI innovation from [46:56] expert at Meta, Reactor.inc, [47:00] Intel, Stanford, and many more. Go to [47:03] the Atscale website to register, and I [47:05] hope to see you there.