Meta Deploys AI Agents to Resolve Distributed GPU Training Failure

Meta部署AI Agent解决分布式GPU训练故障 跨rank调度不匹配导致了99%以上的PyTorch NCCL看门狗超时，促使Meta通过Agent工作流与集中式控制器实现代码追踪排障的自动化。 在分布式PyTorch模型训练中，集群规模的扩大会导致任务故障呈指数级增加，其中99%以上极为棘手的NCCL看门狗超时是由跨rank调度不匹配引起的，而非NCCL操作本身的实际超时。针对有效错误日志匮乏的问题，Meta利用PyTorch Flight Recorder收集各rank的轻量级集合通信元数据并可视化堆栈追踪。尽管人工排查指南和静态分析工具有助于分析执行分歧，但人力瓶颈依然显著。为此，Meta转向了Agent调试工作流，以自动化代码追踪并迅速定位根本原因，同时也指明了未来集中式框架调度以及在超时发生前主动检测Bug的发展方向。 在单个任务使用 2,000 块 GPU 的超大规模下，分布式训练的一次性成功率跌破 2%，重新运行失败任务将带来极高的算力与资金损失。 (0:05) 超过 99% 的 NCCL watchdog 超时并非由集合通信耗时过长引起，而是多 rank 间的调度不匹配导致 GPU 卡死，进而使 CPU 在同步点阻塞。 (1:39) PyTorch Flight Recorder 通过内存环形队列轻量记录通信元数据，Meta 结合分布式矩阵可视化工具，帮助工程师一眼识别跨 rank 首次出现分歧的集合操作。 (7:54) Meta 将基于大模型的 Agentic 调试应用于日志与代码分析，自动化完成机械的代码追踪工作，将传统数小时乃至数天的故障排查缩短至几分钟。 (14:17)

At large cluster scales, distributed AI training frequently fails due to uninformative NCCL watchdog timeouts caused by execution divergence across GPU ranks. Meta's Phillip Liu argues that resolving these costly bottlenecks requires moving beyond manual runbooks to automated telemetry, agentic root-cause analysis, and centralized scheduling frameworks. Scale Bottlenecks and Collective Scheduling Mismatches (0:05) In distributed machine learning, failure rates rise exponentially as cluster allocations expand. At a scale of 2,000 GPUs per job, the single-shot success rate falls below 2%, causing thousands of wasted GPU hours and significant compute budget losses per aborted run. The primary operational bottleneck behind these failures is the PyTorch NCCL watchdog timeout, which standard error logs routinely fail to diagnose beyond reporting an uninformative collective timeout. According to Liu, over 99% of NCCL watchdog timeouts are not caused by slow network communication, but by scheduling mismatches across ranks. Because modern distributed training relies on asynchronous execution—where the CPU queues operations ahead of the GPU—any cross-rank divergence causes desynchronization. If ranks queue mismatched collectives, evaluate conditional branching statements differently, experience data imbalances, or hit asymmetric compilation errors, GPUs stall waiting for collective synchronization until the CPU-side watchdog terminates the job. Flight Recorder Telemetry for Cross-Rank Diagnostics (7:54) Standard stack traces and raw console outputs do not capture distributed collective execution history, leaving engineers without visibility into preceding events. Effective diagnostic telemetry must trace collective operations progressively across all participating ranks without introducing runtime performance regressions that slow down active training. To solve this, PyTorch introduced the Flight Recorder, which maintains an in-memory ring queue on each rank to log lightweight metadata for scheduled collectives. The system defers computationally intensive tasks—such as resolving full stack traces, tensor shapes, and argument sizes—until an actual timeout occurs, at which point all ranks dump their queues. Meta utilizes this data in an internal matrix visualization tool, enabling engineers to identify the exact culprit collective by locating the initial column where metadata colors diverge across ranks. Automating Post-Mortem Triage with Agentic Workflows (14:17) While Flight Recorder and static analysis tools reveal what collective mismatched, diagnosing why execution paths diverged historically required on-call engineers to execute manual runbooks. Engineers spent hours or days manually cross-referencing database records, configuration files, and deeply nested code blocks to trace how conditional logic evaluated differently across ranks. To eliminate this manual triage bottleneck, Meta integrated LLM-based agents into the debugging pipeline. These agents process diagnostic telemetry directly into context, combining code comprehension, configuration audits, and rapid log search. By automating the mechanical tracing of conditional branches and stack traces, agentic workflows compress post-mortem root-cause analysis from days down to minutes without requiring human intervention. Structural Elimination and Pre-Execution Code Audits (18:26) Beyond reactive post-mortem triage, advancing AI reliability involves architectural changes that prevent scheduling divergence entirely. PyTorch's Monarch framework introduces a centralized controller responsible for scheduling all collective operations across ranks. Liu states that centralized scheduling structurally eliminates NCCL watchdog timeouts because mismatched operations can no longer be scheduled independently by divergent ranks. Concurrently, agentic capabilities are expanding from reactive debugging to proactive pre-deployment analysis. By inspecting codebases prior to execution, agents can identify hazardous rank-dependent branching logic and irregular collective schedules before jobs ever launch, preventing infrastructure stalls before compute resources are consumed. While agentic debugging dramatically accelerates post-mortem triage for distributed training failures, Liu concludes that full reliability requires eliminating divergence at the source through cleaner cross-rank code logic and centralized scheduling architectures like Monarch.

Meta 工程师 Phillip Liu 指出，在大规模分布式 AI 训练中，算力扩展会导致故障率呈指数级增长，而绝大多数 NCCL 看门狗超时实际上由跨 rank 调度不匹配引发；Meta 正在通过遥测工具、集中式架构与智能体化调试（Agentic Debugging）来消除此类停机损失。 规模扩展下的可靠性危机与超时根因 (0:05) 在分布式模型训练中，集群规模的扩大会导致异常发生率呈指数级上升。根据跟踪数据，在单作业达到 2,000 块 GPU 的规模下，首次尝试即成功的比率已跌破 2%。高规模作业一旦失败并重新运行，将白白浪费数千个 GPU 运行小时，导致单次故障造成数千美元的算力预算损失，因此快速定位问题、减少重试成为控制成本的关键。 然而，传统调试依赖的标准错误信息在分布式 PyTorch 环境中通常缺乏有效参考价值。典型报错仅显示看门狗捕获到集合通信超时，却无法说明超时诱因。Liu 强调，超过 99% 的 NCCL 看门狗超时并非因为 NCCL 底层操作耗时过长，其实质是不同 rank 之间的 CPU 调度出现了不匹配。 集合通信执行机制与调度错位机制 (3:11) 分布式训练需要利用集合通信（Collectives）在跨机架 GPU 之间交换梯度等数据，在英伟达硬件上该通信由 NCCL 处理，上层则由 PyTorch CPU 端的看门狗线程监控执行耗时。底层计算依托 CPU 与 GPU 的异步并行机制，CPU 负责持续派发算子队列，并通过显式或隐式的同步点阻塞等待 GPU 完成任务。 集合通信要求所有参与的 GPU rank 均加入并完成相应操作后才能推进至下一算子。当部分 rank 调度了不同的通信算子、数据与张量类型不一致，或是某一 rank 完全跳过了调用时，就会产生调度不匹配。这种不匹配会导致 GPU 集合通信停滞，CPU 随后在下一个同步点永久阻塞，最终触发看门狗强行终止作业。 PyTorch 飞行记录仪与人工排障瓶颈 (7:54) 由于常规报错缺乏集合通信细节，有效的遥测方案必须满足全 rank 收集、渐进式记录历史轨迹以及超低运行时开销这三大条件。PyTorch 原生引入的 Flight Recorder（飞行记录仪）在每个 rank 上构建内存环形队列，平时仅记录轻量元数据，直到发生超时才回溯转储堆栈跟踪、张量形状与数据类型。Meta 据此开发了分布式堆栈可视化矩阵，工程师可借由颜色差异直观定位首个出现错位的算子列。 尽管如此，导致 CPU 执行分支分歧的诱因极其繁杂，包括条件逻辑分支判断不一致、数据不均衡导致的迭代步数差异、部分 rank 的非对称报错以及 PTX 编译不一致等。此前依赖值班工程师依据操作手册跨数据库、日志与配置文件逐一排查，或使用缺乏自适应推理能力的静态分析工具，往往仍需耗费数小时甚至数天，排障效率严重受制于人工瓶颈。 智能体化调试落地与集中式架构演进 (14:17) 为打破人工诊断的耗时瓶颈，Meta 将静态操作手册转化为由大语言模型驱动的智能体工作流。智能体能够将遥测数据直接载入上下文，自主完成跨代码块回溯、日志检索与多节点配置比对，替代人类工程师在包含数百个分支的代码库中进行机械排查，将故障根因分析时间从原本的数小时或数天缩短至数分钟以内。 在架构演进方面，PyTorch 推出的 Monarch 架构采用集中式控制器来调度各 rank 的所有集合通信操作，从结构上消除了调度错位的可能，旨在根除看门狗超时。未来，智能体还将从故障发生后的事后分析向前延伸，在作业部署前主动扫描代码库中的跨 rank 危险分支逻辑，从而实现超时隐患的事前拦截。 解决分布式代码中的调度错位能够从根源消除看门狗超时，而结合集中式调度架构与智能体自动化排障，正在将工程团队从高频、机械的故障诊断中解放出来，但智能体在更复杂动态分支场景下的预检能力仍有待实际生产环境的持续检验。

## Background 该视频由 @Scale 频道于 2026 年 6 月 26 日发布在 YouTube 平台。视频简介指出，NCCL 看门狗超时（NCCL watchdog timeouts）是 PyTorch 分布式训练中常见且极难排查的故障模式，Meta 团队通过归纳根因分类、制定决策树与排障手册，并探索引入智能体调试（Agentic Debugging）技术，实现了对该问题的极速根因分析。 ## TL;DR 来自 Meta 的 Phillip Liu 深入剖析了大规模 AI 分布式训练中屡见不鲜且代价极高的 NCCL Watchdog Timeout 问题。他指出超过 99% 的超时根源并非底层通信硬件太慢，而是由于多 Rank 间异步执行与控制流分歧导致的“调度不匹配”。针对这一痛点，Meta 和 PyTorch 社区先后引入了零运行时开销的 Flight Recorder 进行可视化追踪，并开发了开源集中编排框架 Monarch。更具突破性的是，Meta 探索利用具备深层代码理解与多源遥测关联能力的 AI 智能体，将原本耗时数小时甚至数天的人工分支回溯压缩至数分钟，并推动排障体系从“事后救火”迈向“事前代码级免疫”。 ## Atmosphere & style 整场演讲呈现出硬核系统工程师特有的务实、严谨与技术纵深感。主讲人以一线研发人员深恶痛绝的“绝望报错信息”为引子，迅速建立起强烈的技术共鸣，随后层层抽丝剥茧，自底向上展开分布式异步执行与死锁链条的技术解构。从展示传统人工交叉比对的笨拙与挣扎，到引入智能体接管排障时的举重若轻，情绪曲线上体现出一种从“被系统死锁折磨”到“用 AI 终结繁琐泥潭”的从容与技术自信。 ## Bulletin - [00:05](https://www.youtube.com/watch?v=9wc0556pIwg&t=5s) 在大规模分布式 AI 训练中，当作业使用 2,000 张 GPU 时，单次端到端运行成功的概率已不足 2%，且每次重启任务都会造成数千 GPU 小时的巨大算力损耗。 - [03:11](https://www.youtube.com/watch?v=9wc0556pIwg&t=191s) Meta 内部调查显示，超过 99% 的 NCCL Watchdog Timeout 本质上并非底层硬件通信故障，而是 CPU-GPU 异步调度过程中出现的多 Rank 集合通信调度不匹配。 - [07:39](https://www.youtube.com/watch?v=9wc0556pIwg&t=459s) PyTorch 内置的 Flight Recorder 通过内存环形队列以近乎零开销记录元数据，仅在超时发生时转储完整上下文，破解了分布式训练中长期的遥测鸿沟难题。 - [14:17](https://www.youtube.com/watch?v=9wc0556pIwg&t=857s) Meta 引入智能体调试（Agentic Debugging）接管逆向堆栈追踪与多源遥测交叉核对，将传统专家耗时数小时乃至数天的排障链路直接压缩至数分钟。 - [18:26](https://www.youtube.com/watch?v=9wc0556pIwg&t=1106s) 通过开源分布式训练框架 Monarch 的集中式控制器，以及利用智能体在静态检查阶段主动识别导致控制流分流的危险分支，团队正致力于从源头根除通信超时问题。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=9wc0556pIwg&t=5s) 分布式 AI 训练的失败率随 GPU 规模呈指数级上升，在 2,000 卡规模下单次成功率不足 2%，故障排查效率直接关乎巨额算力成本。 - [01:39](https://www.youtube.com/watch?v=9wc0556pIwg&t=99s) NCCL Watchdog Timeout 是 PyTorch 分布式训练中最常见且棘手的报错，其崩溃日志往往只暴露 CPU 监控线程状态，缺乏实际卡死的上下文。 - [03:11](https://www.youtube.com/watch?v=9wc0556pIwg&t=191s) NCCL 看门狗超时并非 NCCL 本身的错误，99% 以上是由多 Rank 之间的调度不匹配（Scheduling Mismatch）诱发的死锁所致。 - [05:15](https://www.youtube.com/watch?v=9wc0556pIwg&t=315s) CPU 与 GPU 的异步执行机制加上 CPU-GPU 同步点，导致任何跨 Rank 的集合通信偏差都会造成 GPU 卡死，进而引发 CPU 挂起并触发看门狗熔断。 - [07:39](https://www.youtube.com/watch?v=9wc0556pIwg&t=459s) 解决超时问题需要全 Rank 覆盖、渐进式追踪、近乎零运行时损耗以及详尽元数据上下文的高性能可观测性方案。 - [07:39](https://www.youtube.com/watch?v=9wc0556pIwg&t=459s) PyTorch Flight Recorder 利用轻量级环形队列延迟解析堆栈，实现了开箱即用且对正常训练路径零侵扰的高效遥测转储。 - [10:12](https://www.youtube.com/watch?v=9wc0556pIwg&t=612s) 分布式调用栈可视化工具通过对各 Rank 通信元数据进行哈希着色，帮助工程师通过横向比对第一处颜色分歧瞬间锁定故障 Rank 与算子。 - [10:12](https://www.youtube.com/watch?v=9wc0556pIwg&t=612s) 导致调度分歧的深层根因通常是代码中的 CPU 执行路径分流，包括依赖未同步状态的分支判断（if 语句）和数据不均衡导致的迭代步数差异。 - [12:05](https://www.youtube.com/watch?v=9wc0556pIwg&t=725s) 传统的排障操作手册（Runbook）与静态分析脚本只能指出发生了什么不匹配，无法解释成因，专家人工跨库交叉核对通常耗费数小时。 - [14:17](https://www.youtube.com/watch?v=9wc0556pIwg&t=857s) 智能体调试能将海量异构遥测数据直接读入上下文，具备强大的推理灵活性，能够自如应对静态规则无法覆盖的边界异常场景。 - [14:17](https://www.youtube.com/watch?v=9wc0556pIwg&t=857s) 利用大模型深层代码理解能力，智能体可以自动化沿调用栈向上反向追踪分支点，并结合日志与配置核对，将溯源耗时压缩到数分钟级别。 - [18:26](https://www.youtube.com/watch?v=9wc0556pIwg&t=1106s) Meta 推出的 Monarch 框架通过中央控制器统一编排跨 Rank 的集合通信，从底层架构上彻底避免了调度失步与不匹配。 - [18:26](https://www.youtube.com/watch?v=9wc0556pIwg&t=1106s) 智能体排障的终极愿景是将人类彻底移出故障调试循环，并将能力左移，在代码静态检查阶段提前扫除潜在的分流隐患以实现事前预防。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=9wc0556pIwg&t=5s) AI 可靠性与大规模分布式训练的失败代价 在针对 AI 可靠性（AI Reliability）的讨论开场中，来自 Meta 的 Philip Liu 提出了一个看似简单却直击痛点的问题：“到底什么是 AI 可靠性？”如果给出一个或许有些过度简化的定义，AI 可靠性无非就是“让模型顺利跑完，中途没有任何报错抛错”。在实际工程中，没有任何工程师会喜欢异常报错，因为一旦报错，就意味着必须投入海量时间去排查和定位问题；而如果一番分诊调试之后依然无解，工程师最终往往只能被迫放弃此前在模型上所做的实验尝试。 然而在现代分布式训练的世界里，异常不仅无法彻底避免，反而如同悬在头顶的达摩克利斯之剑。Philip 展示了一张揭示故障率与单任务所用 GPU 数量关系的图表：任务的故障率会随着所使用的 GPU 规模呈指数级激增。当训练规模攀升至 2,000 张 GPU 时，单次作业一次性成功（one shot）跑完的概率已经不足 2%。这意味着 98% 以上的大规模训练任务都会遭遇中断。 规模化训练中的异常更是灾难性的。这不仅意味着可能痛失实验成果，更在于每次重新拉起任务（rerun）都将白白烧掉数千个 GPU 运行小时（GPU hours），折算下来便是数以千计的真金白银打了水漂。因此，如何以极少的重试次数实现极速调试与根因排查，成为了大规模 AI 训练体系中不可或缺的生命线。 - **规模化失败率**：在 2,000 卡 GPU 的集群作业中，单次端到端成功率低于 2%。 - **算力成本损耗**：大任务每次因报错重启都会耗费数千 GPU 小时，带来数千美元级别的直接资金损失。 --- ### [01:39](https://www.youtube.com/watch?v=9wc0556pIwg&t=99s) 臭名昭著的 NCCL Watchdog Timeout 与可观测性缺失 常规的代码调试手段高度依赖于任务崩溃最后打印出来的错误提示（Error Message）和调用堆栈（Stack Trace）。然而，如果底层的报错日志本身几乎不包含任何有效信息，排查工作就会瞬间陷入绝境。Philip 给出了一段现场报错示例：终端里赫然显示“Watchdog caught collective timeout”（看门狗捕获到集合通信超时），后面跟着一串 sequence number、某种 op type，紧接着附带一段监控线程的调用栈。 这段报错极其让人抓狂：它只告诉你发生了集合通信超时，却完全没说“为什么”会超时；而那段堆栈仅仅表明监控线程卡在了检查超时（`check timeout`）的循环里，根本没有提供导致问题的现场上下文。Philip 感慨道，只要你曾被这个异常逼入墙角，“你绝对不是一个人”。事实上，只要在 NVIDIA 硬件上使用 PyTorch 进行分布式训练，几乎每个人都会撞上这个臭名昭著的异常——NCCL Watchdog Timeout。在 Meta 内部，即使经验丰富的顶尖系统工程师，也曾为了调试这类错误耗费数小时、数天甚至数周时间。 ``` 典型报错信息表现： - Watchdog caught collective timeout (附带 sequence number 与 op type) - 堆栈追溯仅仅停留在 CPU 端的监控线程 (check timeout) - 缺乏 GPU 实际卡死位置以及各 Rank 间交互状态的上下文 ``` --- ### [03:11](https://www.youtube.com/watch?v=9wc0556pIwg&t=191s) NCCL 集合通信机制与 99% 超时的真相：调度不匹配 为了揭开这个故障的面纱，需要先厘清分布式训练的核心通信机制。在多节点、多机架的分布式训练中，各个 Rank（计算进程）必须频繁通信以同步训练进度与梯度，这些跨设备的数据交换操作被称为“集合通信”（Collectives）。在 NVIDIA 硬件体系下，底层的通信由 NCCL（NVIDIA Collective Communications Library，集合通信库）实现。现代分布式训练开发者通常不会直接操作复杂的 NCCL 底层 C/C++ 库，而是通过 PyTorch 提供的顶层封装来触发这些调用。 所谓的 NCCL Watchdog（看门狗），本质上是 PyTorch 在 CPU 侧实现的一个监控线程。它的职责是追踪所有已被派发（scheduled）的 NCCL 集合通信生命周期；一旦某个集合通信耗时过长、超出预设阈值仍未完成，PyTorch 的看门狗线程便会介入并抛出超时异常。 Philip 在此处点出了一个颠覆许多人认知的关键事实：“NCCL Watchdog Timeout 根本就不是 NCCL 自身的错误（not a NCCL error）。”尽管名字里带有“NCCL”，但它完完全全是一个 PyTorch 层面在 CPU 侧触发的异常。在 Meta 针对成百上千起甚至上万起该异常的深入调查中，得出了一个惊人结论：**超过 99% 的 NCCL Watchdog Timeout 根本不是因为 NCCL 底层硬件通信太慢导致的，而是因为“调度不匹配”（Scheduling Mismatch）**。 --- ### [05:15](https://www.youtube.com/watch?v=9wc0556pIwg&t=315s) 分布式训练的异步执行流与死锁链条 要理解调度不匹配，必须深入拆解现代训练任务在多 Rank 场景下 CPU 与 GPU 协同工作的异步执行流。在一个包含 Rank 0 与 Rank 1 的两卡作业中，系统的核心行为特征如下： 首先，**CPU 与 GPU 是完全并行且异步解耦的**。CPU 负责以非阻塞的方式将 GPU 算子（Kernels）源源不断地调度发射到 GPU 流中，提交后 CPU 绝不会停下来傻等 GPU 返回结果，而是立即沿着代码路径向下继续推进；GPU 则异步从队列中拉取任务执行，并最终异步写回结果。 其次，**系统依赖同步点防止 CPU“跑得太快”**。为了避免 CPU 调度领先 GPU 太远导致内存或控制逻辑失控，框架会引入 CPU-GPU 同步点（Sync Points），CPU 在到达同步点时会强制阻塞挂起，等待此前所有排队的 GPU 算子全部执行完毕。这些同步点可以是显式调用的（例如 `torch.cuda.synchronize()`），也可以是隐式的（例如发生 CPU 与 GPU 之间的张量数据拷贝）。 最后是 **GPU 之间的跨卡同步（NCCL Collectives）**。NCCL 集合通信要求参与该操作的所有 Rank 必须全部进入并完成该通信算子，各个 Rank 的 GPU 才能继续向下执行后续的计算。而 GPU 上的 NCCL 通信算子全部来自各 Rank CPU 侧的调度发射。如果各 Rank 之间的调度出现了偏差——例如某个 Rank 根本没有调度该集合通信、某个 Rank 调度了类型完全不同的通信算子、或者传入了不一致的数据类型（Data Type）与张量形状（Tensor Shape）——GPU 上的集合通信便会永远无法闭环。 ```mermaid flowchart TD accTitle: 分布式训练调度不匹配与超时触发流程 accDescr: 展示 CPU 异步发射算子到 GPU，因 Rank 间调度不一致导致 NCCL 死锁并最终触发 Watchdog 的全过程 subgraph CPU_Scheduling["CPU 调度与控制层"] A["Rank 0 CPU 发射算子 A"] --> B["Rank 0 到达 CPU-GPU 同步点阻塞"] C["Rank 1 CPU 发生逻辑分歧发射算子 B"] --> D["Rank 1 到达 CPU-GPU 同步点阻塞"] end subgraph GPU_Execution["GPU 异步执行与通信层"] E["Rank 0 GPU 等待算子 A 的集合通信"] F["Rank 1 GPU 等待算子 B 的集合通信"] E <-. 集合通信握手失败/死锁.-> F end subgraph Watchdog_Intervention["看门狗监控与熔断"] G["PyTorch NCCL Watchdog 线程计时"] H["超时阈值触发: 抛出 NCCL Watchdog Timeout 并杀掉作业"] end A --> E C --> F B --> G D --> G G --> H ``` 当 GPU 陷入僵死，紧邻该通信算子之后的第一个 CPU-GPU 同步点便会导致各 Rank 的 CPU 全部被死死卡住。随着时间一分一秒流逝，原本在后台静默计时的 PyTorch NCCL 看门狗最终判定超时，果断介入并抛出异常，整个分布式作业随之轰然崩溃。 --- ### [07:39](https://www.youtube.com/watch?v=9wc0556pIwg&t=459s) 遥测鸿沟与 PyTorch Flight Recorder 的破局 明确了 99% 的根因在于调度不匹配后，调试的焦点就转向了系统可观测性。然而，传统的调试手段在此遭遇了巨大的“遥测鸿沟”（Telemetry Gap）：崩溃时打印的通用日志和调用栈不仅没有记录出故障时的集合通信上下文，更无法跨 Rank 复原调度序列。 为了彻底解决这一问题，理想的遥测方案必须满足四大严苛要求： 1. **全 Rank 覆盖**：必须能够跨所有参与训练的计算 Rank，完整收集集合通信的元数据信息； 2. **渐进式追踪（Progressive Tracking）**：鉴于 CPU 和 GPU 异步执行的本质，超时发生那一刻，GPU 队列中可能同时挂起着多个尚未完成的集合通信，遥测工具必须逐步记录一段历史窗口，而不是仅仅抓取最后发射的那一个算子； 3. **极度轻量与高性能**：所有计算密集型的高开销操作（如回溯符号化堆栈）必须严格推迟到超时发生之后再进行；若在训练主循环中引入任何性能损耗（Performance Regression），该方案就无法在超长周期的生产作业中全局开启； 4. **详尽的上下文维度**：必须完整捕获超时瞬间的通信状态、函数调用栈、涉及张量的数据类型（dtype）、张量形状（Shape）以及入参尺寸。 面对这一挑战，PyTorch 社区早已推出了内置的终极武器——**PyTorch Flight Recorder（飞行记录仪）**。Flight Recorder 在内存中维护了一个轻量级的环形队列（Ring Queue）。在训练运行期间，PyTorch 会在每个 Rank 调度集合通信时以近乎零开销记录元数据；唯有当集合通信超时真正发生时，系统才会为所有 Rank 完整转储（dump）该环形队列，并在此刻才对所有调用栈展开实例化解析。由于这套机制开箱即用（out of the box）且对正常执行路径零侵扰，它成为了攻克 NCCL 超时迷雾的关键基石。 --- ### [10:12](https://www.youtube.com/watch?v=9wc0556pIwg&t=612s) 可视化诊断与导致调度分歧的根本原因 有了 Flight Recorder 捕获的详实轨迹，Meta 在内部构建了一套“分布式调用栈可视化工具”（Distributed Stack Trace Visualization），使工程师“仅凭一眼就能精准锁定肇事集合通信”。 在这套可视化界面中： - **网格单元（Cells）**：每一个矩形格子代表一次被调度的集合通信操作，其内部颜色依据该通信的全部元数据（包括 Python 调用栈、数据类型、张量形状等）综合哈希着色。 - **行（Rows）**：每一行代表一个特定 Rank 按照时间先后顺序调度的集合通信序列（例如第一行就是 Rank 0 的执行历史）。 - **列（Columns）**：代表跨 Rank 的全局通信推进顺序。 由于所有 Rank 必须在集合通信队列上保持严格一致，排查人员只需从左向右横向扫描，找到**第一个出现颜色差异的列**，就能立刻抓出发起调度的罪魁祸首。例如在 Meta 内部展示的一张可视化诊断图中，第 3353 列在其他 Rank 都是同色方块的情况下，Rank 0 突兀地显示为一个绿色的单元格——这直接说明 Rank 0 在该步调度了与其他 Rank 完全不同的集合通信。 锁定了肇事的 Rank 与通信算子后，随之而来的终极问题是：究竟是什么导致了 CPU 侧的调度分歧？Philip 指出，代码执行层面上**任何潜在的 CPU 执行路径分流（CPU execution divergence）**都会让分布式任务陷入调度不匹配的深渊： - **分支判断分歧（Branching Point）**：代码中一个看似人畜无害的简单 `if` 条件判断语句，如果依赖了没有在多卡间广播同步的局部状态，导致某些 Rank 判定为 True、另一些判定为 False，就会导致不同分支发射了不同的通信算子； - **数据不均衡（Data Imbalance）**：各个 Rank 分配到的数据分片数量不一，或者动态 Batch 填充不均，导致部分卡比其他卡少跑或多跑了若干个迭代步骤（Iterations），过早或过晚地退出了通信同步逻辑。 ### [12:05](https://www.youtube.com/watch?v=9wc0556pIwg&t=725s) NCCL 超时的根因复杂性与传统排障流程的瓶颈 在分布式训练场景中，导致 NCCL watchdog 超时的原因极为繁杂且具有隐蔽性。比如部分 rank 上发生的非对称错误（asymmetrical errors），或者跨 rank 之间由于非对称 PTX 编译导致不同节点实际执行的代码产生分歧，都会让系统陷入死锁并最终超时。因为根因类型繁多，PyTorch 官方博客专门发布了题为《Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts》的技术文章，对各种超时诱因进行了极其深入的剖析。为了应对这些线上故障，Meta 内部最初梳理了一份标准的排障操作手册（Runbook），值班专家（on-call expert）需要依据手册执行一套遥测数据交叉比对流程：从各个数据库表中检索遥测指标、核对日志与配置，最终定位导致集合通信不匹配（mismatch）的根本原因。此外，工程团队还开发了在任务因超时被终止后自动运行的静态分析代码，用于输出报告，说明是哪个 rank、哪次集合通信调用出现了分歧。 然而，这套完全依赖人类与静态规则的排障体系在实践中暴露出严重的瓶颈。首要问题是极高的人力成本与时间开销：排查流程需要跨多个系统对异构遥测数据进行交叉验证，值班专家从获取遥测信息、导出 Flight Recorder 追踪记录、展开数据分析，到最终把问题归因归角到具体组件，通常要耗费数小时。其次，现有的静态分析脚本缺乏足够的灵活性，它只能死板地指出“发生了什么不匹配”——例如能够精准揪出罪魁祸首 rank（culprit rank）和发生冲突的通信原语（culprit collective），但完全无法回答“为什么会产生这种不匹配”。查明故障本质根因的重担最终依然落在人类工程师身上，排障效率因此停滞不前。 ```mermaid flowchart TD accTitle: 传统 NCCL Watchdog 超时排障工作流与局限 accDescr: 展示任务被杀后静态分析与人工介入的交叉比对流程及其痛点 subgraph Failure["故障发生"] A["任务触发 NCCL Watchdog 超时终止"] end subgraph StaticAnalysis["静态工具分析（局限：缺乏灵活性）"] B["运行自动化静态检查代码"] C["定位问题 Rank 与通信原语<br/>(Culprit Rank / Collective)"] A --> B --> C end subgraph ManualTriage["专家人工排查（局限：耗时数小时）"] D["提取 Flight Recorder 与遥测数据"] E["跨库查询 Log、Config 与 DB"] F["人工逆向推导根因并完成定责"] C --> D --> E --> F end ``` - **非对称错误形态**：涵盖局部 rank 故障与跨 rank 的非对称 PTX 编译差异。 - **静态工具的边界**：仅能判断“现象”（mismatch 本身及涉及的 rank / collective），无法解释“成因”（why it happened）。 - **专家排查成本**：单次从捞取数据到定位归属组件往往需要数小时的手工交叉比对。 --- ### [14:17](https://www.youtube.com/watch?v=9wc0556pIwg&t=857s) 智能体调试（Agentic Debugging）：代码理解与分支逆向追踪的范式跃迁 面对人工与静态脚本的双重困境，团队转向了智能体调试（Agentic Debugging）。随着大语言模型智能体能力的演进，团队发现只要给予正确的引导，智能体能够极其顺畅地接管大量机械化劳动；它不仅可以精准遵循现有的排障流程，还能在分析过程中融入独特的推理灵活性。Meta 迈出的第一步便是将原先僵化的静态排查工作流改造为智能体驱动的链路。智能体能够直接将海量遥测数据加载到上下文窗口中进行综合推理，轻松处理静态脚本无法覆盖的各种边界异常情况（edge cases）；更关键的是，智能体阅读和消化遥测日志的速度远非人类所能比拟，直接将值班人员从数小时的繁琐排查中解救出来。 智能体展现出的另一项惊人优势在于对庞大代码库的深层语义理解。演讲中以一个典型事故为例：假设系统识别出不同 rank 调度了完全不一致的集合通信原语，例如 Rank 0 调度了 All-to-All，而其他所有 rank 却都在调度 AllReduce。人类专家的常规做法是从堆栈轨迹（stack trace）的最底部开始，逐级向上反向追踪（up trace）寻找第一个分支点（if 语句），因为只有控制流分支才会诱发不同 rank 的行为分歧。 ```mermaid flowchart LR accTitle: 智能体自动化分支逆向验证流程 accDescr: 展示从调用栈底部的集合通信分歧向上回溯条件分支并结合日志与配置验证的过程 subgraph Stack["调用栈逆向回溯"] direction TB S1["发生分歧的通信调用<br/>(如 All-to-All vs AllReduce)"] S2["上溯第 1 个条件分支 (if 语句)"] S3["上溯上一层函数及嵌套分支"] S1 --> S2 --> S3 end subgraph TelemetryCheck["智能体自动化交叉验证"] direction TB T1["检查分支内部 Log 是否被打印"] T2["检索特定节点配置 (Config) 差异"] T3["排除无差异分支，锁定真实分歧点"] S2 -.-> T1 S2 -.-> T2 T1 & T2 --> T3 end ``` 人类专家在这一逆向追踪过程中极其痛苦。找到一个包含函数调用的 `if` 语句后，必须验证其在不同 rank 间的求值是否不同：如果幸运的话，分支内包含打印日志的语句，专家就需要下载原始日志并手动检索该日志是否存在于所有 rank 中；如果没有查到该日志，便排除该分支。接着专家必须转向下一个分支点，例如观察到一个依据配置文件做条件跳出的循环 break，又必须去拉取该任务的配置文件反复核对。如果一个函数内的所有分支都被排除，专家还得跳进更上层的调用函数中，面对里面嵌套的三四个 `if` 语句重复上述枯燥的循环。在现代复杂的大模型训练框架中，代码调用链的复杂度往往高出数十倍甚至上百倍，跨 rank 的条件求值逻辑与可供交叉验证的日志往往被埋藏在极深的调用栈里。人类排查这类问题常常要花上数小时甚至数天，而智能体凭借敏捷的代码理解与近乎瞬时的遥测加载能力，“通常在数分钟的量级内（at the scale of minutes）”就能准确锁定诱发不匹配的控制流分支。 --- ### [18:26](https://www.youtube.com/watch?v=9wc0556pIwg&t=1106s) 总结与展望：消除人工排障与从源头预防超时 在分享的最后，演讲者梳理了面向未来的两项核心结论。首先是 PyTorch 社区对分布式 AI 训练高可靠性的长期深耕：一方面社区推出了用于事后分析 NCCL 超时的诊断利器 Flight Recorder；另一方面，针对超时问题本身，Meta 在前一年开源推出了全新分布式训练框架 Monarch。Monarch 引入了集中式控制器（centralized controller）来统一编排与调度所有 rank 的集合通信操作，从而彻底杜绝了不同 rank 调度不匹配通信原语的可能性，从底层架构上将此类 NCCL watchdog 超时问题“从根本上消除（eliminates fundamentally）”。 其次，智能体调试代表着分布式系统基础设施运维的未来方向。其终极目标是把人类彻底从繁琐的故障排查循环中剥离出去，全面交由智能体接管，让宝贵的系统工程专家不再把时间浪费在机械的日志翻找与代码溯源上，转而去从事“更具影响力的工作（much more impactful work）”。不仅如此，团队目前的探索虽然集中在任务因超时被杀后的被动诊断，但智能体的能力完全可以向前延展到事故发生之前：既然 NCCL watchdog 超时的本质根源是代码中跨 rank 的控制流分支诱发了错误的通信调度，那么完全可以借助具备深厚代码理解能力的智能体，在代码静态检查或集成测试阶段主动扫描代码库，识别出那些隐藏在不同分支路径下可能导致集合通信分歧的隐患。这种从“事后救火”向“事前免疫”的跨越，将从源头上阻断 NCCL 超时的发生。 - **Monarch 集中调度范式**：采用中央控制器调度集合通信，彻底避免多 rank 调度失步与不匹配。 - **智能体调试的核心愿景**：实现 "Eliminate the humans out of the debugging loop"，将排障周期从天/小时级压缩至分钟级。 - **能力前移（Shift-Left）**：从任务崩溃后的根因反推，演进到在代码入库前通过智能体识别潜在的危险分支，实现根本性预防。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=9wc0556pIwg&t=5s) 到底什么是 AI 可靠性？ — Philip Liu > [01:39](https://www.youtube.com/watch?v=9wc0556pIwg&t=99s) Watchdog caught collective timeout — Philip Liu > [01:39](https://www.youtube.com/watch?v=9wc0556pIwg&t=99s) 你绝对不是一个人 — Philip Liu > [03:11](https://www.youtube.com/watch?v=9wc0556pIwg&t=191s) NCCL Watchdog Timeout 根本就不是 NCCL 自身的错误（not a NCCL error） — Philip Liu > [03:11](https://www.youtube.com/watch?v=9wc0556pIwg&t=191s) 超过 99% 的 NCCL Watchdog Timeout 根本不是因为 NCCL 底层硬件通信太慢导致的，而是因为“调度不匹配”（Scheduling Mismatch） — Philip Liu > [10:12](https://www.youtube.com/watch?v=9wc0556pIwg&t=612s) 仅凭一眼就能精准锁定肇事集合通信 — Philip Liu > [10:12](https://www.youtube.com/watch?v=9wc0556pIwg&t=612s) 第一个出现颜色差异的列 — Philip Liu > [14:17](https://www.youtube.com/watch?v=9wc0556pIwg&t=857s) 通常在数分钟的量级内（at the scale of minutes） — Philip Liu > [18:26](https://www.youtube.com/watch?v=9wc0556pIwg&t=1106s) 从根本上消除（eliminates fundamentally） — Philip Liu > [18:26](https://www.youtube.com/watch?v=9wc0556pIwg&t=1106s) 更具影响力的工作（much more impactful work） — Philip Liu > [18:26](https://www.youtube.com/watch?v=9wc0556pIwg&t=1106s) Eliminate the humans out of the debugging loop — Philip Liu ## Mentioned Philip Liu, Meta, @Scale, PyTorch, NVIDIA, NCCL (NVIDIA Collective Communications Library), PyTorch Flight Recorder, Monarch, Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts ## Video description NCCL watchdog timeouts are a common failure mode in distributed AI model training. They impact not only Meta, but broadly affect anyone running PyTorch distributed training—and they’re notoriously hard to debug: even experts can spend hours triaging a single incident, and non-experts may be unable to root-cause them at all. Over the past year, we investigated NCCL watchdog timeouts internally at Meta and partnered with the PyTorch community to categorize the major root-cause buckets. We then distilled these learnings into a practical decision tree and runbook to speed up triage and make debugging more accessible. We also explored using agent-based approaches to assist root-cause analysis and saw strong early results. Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=9wc0556pIwg)

[0:05] Philip Liu: 到底什么是 AI 可靠性？ [1:39] Philip Liu: Watchdog caught collective timeout [1:39] Philip Liu: 你绝对不是一个人 [3:11] Philip Liu: NCCL Watchdog Timeout 根本就不是 NCCL 自身的错误（not a NCCL error） [3:11] Philip Liu: 超过 99% 的 NCCL Watchdog Timeout 根本不是因为 NCCL 底层硬件通信太慢导致的，而是因为“调度不匹配”（Scheduling Mismatch） [10:12] Philip Liu: 仅凭一眼就能精准锁定肇事集合通信 [10:12] Philip Liu: 第一个出现颜色差异的列 [14:17] Philip Liu: 通常在数分钟的量级内（at the scale of minutes） [18:26] Philip Liu: 从根本上消除（eliminates fundamentally） [18:26] Philip Liu: 更具影响力的工作（much more impactful work） [18:26] Philip Liu: Eliminate the humans out of the debugging loop

[1:40] Phillip Liu: If you have been ever cornered by this exception, you're not alone. [1:40] watchdog caught collective timeout [11:33] Flight Recorder and PyTorch's New Lens for Understanding NCCL Watchdog Timeouts [14:17] Phillip Liu: As the agent develops, we realize that it can do many mechanical work fluently if guided correctly. [18:26] Phillip Liu: fundamentally because there will be no mismatch scheduled. [20:04] Phillip Liu: happy debugging.

[0:05] Hi. My name is Philip. I'm from Meta. [0:08] As we gather here for AI reliability, [0:12] there's a quick question, is that what [0:14] is AI reliability? [0:17] In a maybe oversimplified definition, [0:19] AI reliability is to run your model [0:22] successfully with no exception. [0:25] Nobody likes exception because that [0:26] means you have to [0:28] spend a lot of time debugging the [0:30] problem. And if the triage and debugging [0:33] didn't resolve the exception, [0:35] eventually you will have to give up on [0:37] the experiments you made on the model. [0:41] However, in a distributed training [0:43] world, [0:44] sometimes the exceptions are inevitable. [0:47] Let's look at this chart that shows the [0:50] failure rate in regards with number of [0:52] GPU used in a single job. [0:55] As you can see here, the failure rate [0:57] increase exponentially [0:59] correlated to the number of GPU used. [1:01] And at the scale of 2,000 GPU jobs, [1:05] the successful rate is less than 2%. [1:08] That means that less than 2% of the [1:10] training jobs [1:11] will succeed in one shot. [1:14] In addition, [1:15] exceptions on those big jobs are even [1:17] worse. [1:19] As it's not meaning that you can [1:20] potentially give up on your experiment, [1:23] but also every single rerun cost [1:26] thousands of GPU hours, [1:28] which translates to thousands of dollars [1:30] wasted. [1:32] So, as a result, [1:33] fast debug with minimal number of [1:35] retries [1:36] is essential in those jobs. [1:39] With common debugging techniques, [1:41] which requires reasoning from error [1:44] message, stack traces [1:46] that got raised and printed out at the [1:48] end of the job. [1:50] But what if the error message doesn't [1:51] really have information? [1:54] Let's look at this example. [1:58] We have here [2:00] watchdog caught collective timeout, give [2:02] a sequence number, some op type, and [2:05] then followed by a stack trace that from [2:08] a monitoring thread. [2:09] This is obviously not enough [2:11] information. [2:13] The error message tells you that it is a [2:15] collective timeout, but it didn't tell [2:17] you why the collective timeout. [2:20] And for [2:22] the stack trace this it gives you a [2:24] monitoring thread stack trace that stuck [2:26] in check timeout, which is not telling [2:29] you what exactly causing the problem. [2:32] If you have been ever cornered by this [2:35] exception, [2:36] you're not alone. And actually, as long [2:39] as you're using PyTorch to train your [2:41] distributed job using Nvidia hot wear, [2:44] you're likely see this exception before. [2:47] This is the infamous NCCL watchdog [2:49] timeout. [2:50] And inside Meta internally, our [2:52] engineers struggled to debug this for [2:55] hours or even days or weeks. [2:59] Today, we're going to talk about how [3:02] Meta triage NCCL watchdog timeout. [3:04] How do we expand the debuggability of [3:07] NCCL watchdog timeouts using agents? [3:11] Let's start with some context setting on [3:14] what is a NCCL collective first. [3:17] In the world of distributed training, [3:19] since the work is put on multiple racks, [3:23] it is important to communicate [3:26] with each other to share the progress, [3:28] let's say gradients, [3:31] and other information. [3:33] Those communication operations are [3:35] called collectives. [3:38] For different hardware, [3:40] the communication implementations may [3:42] vary. [3:43] And NCCL, [3:45] which stands for Nvidia collective [3:47] communication library, implements the [3:50] collective operation on Nvidia hardware. [3:53] In modern distributed training, we don't [3:55] really interact with this deep [3:58] sophisticated nickel library, [4:01] but instead PyTorch wraps this [4:03] collective call for us. [4:06] So that we are just calling on PyTorch [4:07] level. [4:09] And the nickel watchdog is also a [4:12] PyTorch implemented CPU side monitoring [4:16] thread, [4:18] which basically monitors the life cycle [4:21] of all the scheduled nickel collective. [4:24] And when a nickel collective taking too [4:27] long and didn't finish within a certain [4:30] time, [4:31] the PyTorch nickel watchdog will kicking [4:33] and fire a nickel watchdog timeout. [4:38] One thing is that to note is that [4:40] nickel watchdog timeout is not a nickel [4:43] error. [4:45] And it's as you can see here, the nickel [4:47] watchdog timeout, [4:49] even though it has the word nickel, it's [4:51] the PyTorch level CPU side error. [4:54] And in fact, during our investigations [4:57] for hundreds or even thousands of nickel [5:00] watchdog timeouts, [5:02] we come to a conclusion that over 99% of [5:05] all the nickel watchdog timeouts [5:07] are not even caused by nickel operations [5:10] taking too long, but instead a [5:12] scheduling mismatch. [5:15] So, [5:16] you may question what is a scheduling [5:18] mismatch? And we have to Before we [5:21] understand that, we need to figure out [5:23] the execution flow of a modern training [5:26] job. [5:27] Let's look at this two rank distributed [5:30] training job execution flow with both [5:32] CPU and GPU. [5:35] There are a few takeaways from this. [5:37] Firstly, [5:38] the CPU and GPU executions are in [5:40] parallel, meaning that the CPU will [5:42] schedule GPU kernels onto GPU. It will [5:45] not wait for the GPUs to return the [5:48] result, but instead will proceed to the [5:50] next scheduling and other CPU [5:52] executions. On the other hand, for the [5:54] GPUs, they will pick up all the [5:56] scheduled work and execute them [5:58] asynchronously. And eventually, they [6:00] will return the result asynchronously as [6:03] well. [6:04] Secondly, we cannot let CPU to go too [6:06] deep and too beyond GPUs. [6:10] So, in modern training framework, [6:12] we inject CPU GPU synchronizations [6:15] so that the CPU will block wait at those [6:18] synchronized position points to wait for [6:20] all the GPU kernels to finish. [6:23] Those sync point can be explicit, like [6:26] you just call a torch CUDA [6:27] synchronization, [6:29] or it can be implicit, like any tensor [6:32] copy from CPU to GPU or vice versa. [6:37] Last but not least, it's the GPU [6:39] inter-GPU [6:41] synchronization, [6:42] which is the NCCL collectives. [6:45] So, the NCCL collectives requires [6:49] to join this collective at finish. [6:51] Until then, the GPU will proceed to the [6:54] next GPU kernel. [6:56] And since the GPU kernels that GPU NCCL [6:59] collectives [7:00] communication kernels [7:02] are scheduled from CPU, [7:04] whenever there is a mismatch in the [7:06] scheduling, say some ranks didn't [7:08] schedule this collective, or some ranks [7:11] scheduled different collective, [7:12] or with different data type tensor type, [7:15] it's going to cause those collective on [7:18] GPU to mismatch, [7:20] causing all ranks to not finish this [7:22] collective. [7:24] Eventually, the CPU will block wait at [7:27] the first CPU GPU synchronization next [7:29] to this copper collective, [7:31] and [7:32] the NCCL watchdog timeout will kick in [7:35] and fire exception to kill this job. [7:39] Now that we know the concepts of nickel [7:42] collectives, nickel collective timeout, [7:44] and we know that it is caused by [7:46] mismatching scheduling. [7:48] Let's dig into how we could debug a [7:51] nickel collective timeout. [7:54] First, we have to acknowledge the [7:56] telemetry gap. [7:58] In modern debugging, we analyze and [8:01] reasoning from the stack traces, the [8:04] logs, and error message, [8:05] which is not viable in this case [8:08] because all the error message are [8:09] not containing any collective [8:11] information. [8:12] So, let's take a step back and review [8:14] what do we need from the telemetry. [8:18] Firstly, as we know that the [8:20] the nickel collective timeouts are [8:22] caused by a mismatch scheduling, [8:25] we know that we need the telemetry to [8:26] gather all the collective information [8:29] for more ranks. [8:32] Secondly, due to the nature of CPU GPU [8:34] asynchronously execution, [8:37] there could be multiple pending [8:39] collectives [8:40] when the timeout happen. [8:42] So, that we need the telemetry to to [8:44] progressively collect all the collective [8:46] information instead of just looking at [8:48] the latest scheduled collective. [8:52] Thirdly, it has to be performant [8:55] and lightweighted, and ideally it has to [8:57] be [8:58] the all the computation intense work can [9:02] happen after the timeout happen [9:04] because otherwise, [9:05] if it introduce performance regression, [9:09] we cannot enable it across the job's [9:11] duration. [9:13] And the value we are seeking from this [9:15] telemetry is all the collective [9:17] information. We need the stack traces, [9:20] the collective states at the time of the [9:22] timeout happen. We need a data type on [9:25] this tensor. [9:27] We need the tensor shape. We need [9:29] arguments size. [9:32] And with all those requirements, we need [9:34] this new telemetry, and fortunately, [9:37] PyTorch has has already implemented this [9:39] for us. [9:40] And that is the PyTorch flight recorder. [9:44] The PyTorch flight recorder is a ring [9:46] queue. [9:47] And [9:48] PyTorch will collect the information on [9:50] each rank [9:52] upon a schedule to collect it. [9:55] And this ring queue will get dumped upon [9:57] the collective timeout for all ranks. [9:59] It is performant because it instantiates [10:02] all the stack traces and information [10:04] after the timeout happened. [10:06] And as long as you're using PyTorch, [10:09] you have this out of the box. [10:12] So now that we have this powerful nickel [10:14] flight recorder, [10:16] we can build a quick visualization so [10:18] that we can identify the culprit [10:20] collective at a just simple glance. [10:24] Here is an example of meta internal [10:26] visualization that we built. [10:29] So, this is a distributed stack trace [10:31] visualization that every single cell, [10:34] every single rectangle, represents one [10:36] scheduled collective, and they are [10:37] colored by its collective information, [10:40] including stack traces, data type, [10:43] tensor shape, whatever it has. And so [10:45] that every single row represents the [10:47] sequential order [10:49] that the collective got scheduled on the [10:51] rank. So, the first row represent [10:53] all the collective scheduled on rank [10:55] zero. [10:56] So, due to the nature of the nickel [10:58] collective has to be scheduled the same [11:01] across ranks, we can easily identify the [11:04] first mismatching [11:06] collective [11:07] by just checking the first column that [11:09] having different colors. [11:11] So, for example, in this visualization, [11:13] the column with 3353 [11:16] having a green cell, which represent a [11:18] different collective scheduled from rank [11:20] zero compared to others. [11:22] Now that we have this visualization and [11:24] we can easily identify the culprit rank [11:27] and culprit collective, [11:29] the next step is [11:31] what causes the mismatch? [11:33] What causes certain ranks to schedule a [11:35] different collective than others? [11:37] And as you may already figured out, [11:40] actually any potential CPU execution [11:42] divergence will cause make your job [11:45] vulnerable to this mismatch scheduling. [11:49] It can be a branching point. Let's say a [11:51] simple if statement that got evaluated [11:54] differently across ranks. [11:57] It can be data imbalance that cause [11:59] certain ranks to have a few less [12:02] iteration than others. [12:05] It can be some asymmetric errors that [12:08] happen on a subset of the ranks. [12:11] And it can be asymmetric PTX compilation [12:15] that result in a different compiled code [12:17] across ranks. [12:20] The root causes for nickel watcher [12:22] timeout are [12:24] so many and [12:26] for the sake of time, we're not going to [12:28] deep dig into all the root causes. [12:32] But if you're interested, [12:34] you can check out the fly recorder [12:36] PyTorch blog, fly recorder and new lens [12:39] for understanding nickel watcher [12:40] timeout, [12:41] which dig [12:43] deeply [12:44] and investigate deeply inside all the [12:47] root causes. [12:48] Inside Meta, we built a debugging [12:51] runbook so that our on-call expert can [12:54] follow. [12:56] It involves telemetry cross-verification [13:00] so that you go to some [13:03] database tables, you go to check the [13:05] logs and configs, and cross-verify, and [13:08] eventually root cause what what causes [13:11] the mismatch. [13:12] We also built some static code [13:14] that runs automatically after the job [13:17] got terminated by nickel watcher timeout [13:19] and generate some insights to tell you [13:22] what is causing the mismatch. [13:25] And what is the mismatch? [13:27] However, those two flows we had have [13:30] quite a few limitations. [13:33] Firstly, the debugging workflow requires [13:35] a lot of human effort. Cross-verifying [13:38] with different telemetry is [13:39] time-consuming. Usually, our on-call [13:41] expert takes hours [13:44] from getting the telemetry, getting the [13:46] flight recorder, doing the analysis, and [13:49] eventually triage the problem to a [13:50] certain components. [13:52] Secondly, [13:53] the static analysis is not flexible [13:56] enough. It can only tell you that what [13:59] is the mismatch. It can tell you the [14:02] culprit rank and culprit collective, but [14:05] it cannot tell you what causes the [14:08] mismatch. [14:09] So, human is still in the loop of [14:12] root cause, what causes the mismatch, [14:14] and that takes time. [14:17] And here is the fun part, [14:18] the agentic debugging. [14:20] As the agent develops, [14:22] we realize that it can do many [14:25] mechanical work fluently if guided [14:27] correctly. [14:29] It can follow the debugging process [14:31] and add its own flexibility into the [14:34] process. [14:36] So, [14:37] the first attempt we made was to convert [14:39] the static debugging workflow [14:41] into agentic ones. [14:44] Since the agent can load the telemetry [14:47] into its context and reason it [14:49] comprehensively, [14:51] it can handle edge cases [14:53] that static analysis cannot. [14:55] And on the other hand, it can read the [14:57] telemetry much faster than human beings. [15:01] So, it's going to save hours of human [15:03] effort in triaging the nickel watch [15:05] attempt. [15:08] Also, agent can save hours on even days [15:12] of human effort when it comes to [15:14] code-based understanding. [15:16] Let's look at the example. [15:18] When we identified that some ranks [15:20] schedule a different collective from [15:22] others. [15:23] Let's say rank zero schedule in auto [15:25] all, while all other ranks schedule in [15:28] all reduce. [15:29] As a human, we usually just look at the [15:32] bottom of the stack trace and up trace [15:34] for the first [15:36] if statement and branching point [15:38] because that is the only problem that [15:41] causing a rank to schedule differently. [15:44] So, let's say this is the code [15:46] implementation that has this auto all [15:49] schedule. [15:50] As a human being, we usually just look [15:53] at the first or up trace the if [15:55] statements above it. [15:58] Let's say we are looking at this if [16:00] statement and this is a function call. [16:02] And how are we going to cross verify [16:04] whether this is valued differently [16:06] across ranks? [16:08] We're lucky enough that there's a [16:10] logging statement inside this if [16:11] statement. So, if we can check from the [16:14] raw log whether this log has been [16:17] called, whether this log is present in [16:20] the log, [16:21] we know that this is evaluated [16:23] differently [16:24] if some some ranks didn't have it. So, [16:27] as a on-call expert, I will download the [16:29] log and do some search inside the log. [16:32] This can take hours. And for this [16:34] debugging example, we don't see anything [16:37] of this logging statement. So, we rule [16:40] out the possibility that this branching [16:42] point causing the entire nickel watch [16:45] attack. [16:47] So, we The next step is to look at the [16:49] next branching point. And we see that [16:52] there is a if statement here causing a [16:53] conditional break for this for [16:55] statement. And we can see that this if [16:58] statement is evaluating a config. And as [17:01] an on-call expert, I have to pull the [17:03] config file and do some search. [17:06] And as you can see here, maybe we didn't [17:09] find anything for this config and so [17:12] that this if statement will be evaluated [17:16] samely across ranks. [17:19] And [17:21] if we roll out all the statements inside [17:23] this function, [17:24] the next step will be checking other if [17:26] statements. Let's say this is the [17:28] function that calls the auto wall. [17:31] The next step will be looking at the [17:33] next branching point inside this [17:35] function and cross-verify there. [17:38] And after jumping into the [17:39] implementation, we see four if [17:41] statements well, and then we just do the [17:43] same thing again and again. [17:45] This is really time-consuming. In [17:47] reality, in modern training frameworks, [17:50] the code base are complexity can be tens [17:52] or hundreds more. And the if statements [17:55] are evaluated more differently, and the [17:58] logging statement and the method that we [18:00] can cross-verify can be buried more [18:02] deeply. [18:03] So, it usually takes our agents hours or [18:06] even days [18:08] to finish this problem [18:09] finish the cross-verify to pinpoint the [18:12] branching point. But for agent, it can [18:14] usually takes [18:15] at the scale of minutes. [18:17] Given my usability of code base [18:19] understanding [18:20] and loading telemetries much faster than [18:23] human. [18:26] So, for the takeaways, [18:27] I want to first talk about [18:29] that [18:30] PyTorch as the leader in the AI training [18:33] area [18:34] focuses on rely AI reliability. [18:38] It provides the fly recorder [18:40] for debugging the code watch out time [18:42] out. And also, if you're interested in [18:45] Monarch, which is a new PyTorch [18:47] framework that got released last year, [18:50] it provides a centralized controller to [18:53] schedule all the collectives [18:55] so that they would [18:57] it eliminates the nickel watch out time [18:59] out [19:00] fundamentally because there will be no [19:02] mismatch scheduled. [19:04] Secondly, [19:05] I want to emphasize that agent debugging [19:07] is the future. [19:09] Well, the aim is that we eliminate [19:12] the humans out of the debugging loop and [19:15] we just use agent. [19:17] Because agent can do [19:18] the mechanical work much faster and [19:20] saves hours of and days of human effort [19:24] so that our uncle expert can do much [19:26] more impactful work. [19:29] Also, we were talking about debugging [19:32] using agent after the job got terminated [19:35] by Nico Watch a Time out. [19:37] But [19:38] actually we can expand the agent [19:40] debugging ability [19:42] even before the Nico Watch a Time out [19:44] happen. [19:45] Given the mismatch is given the Nico [19:48] Watch a Time out are happening due to [19:50] mismatch scheduling [19:53] and we can expand the skill to let the [19:55] agent understand [19:57] the code base and identify those Nico [20:00] collected schedules in different branch [20:03] points. [20:04] That will eliminate the Nico Watch a [20:06] Time out fundamentally from the code [20:08] base and prevent it from happening. [20:11] My name's Philip. [20:13] This is a journey inside Meta that [20:15] debugging Nico Watch a Time out and [20:16] using agent [20:18] and happy debugging.