Nvidia Overhauls Slurm Scheduler to Maximize GB200 Cluster Output

英伟达重构Slurm调度器以最大化GB200集群算力产出 英伟达通过将开源Slurm拓扑块调度与预测性工作负载模拟相结合，使损失的集群占用率恢复至距离最优容量仅差一个百分点以内。 NVIDIA集群团队详细介绍了其部署和扩展大规模HPC基础设施的运营策略，即从旧硬件过渡到GB200 NVL72和GB300架构。为解决多节点NVLink域造成的严重作业碎片化问题，NVIDIA与SchedMD合作开发了开源Slurm拓扑块插件，并利用针对20,000个GPU的加速模拟，使集群占用率重新优化至接近理论最大值的水平。此外，演讲者还概述了自适应测试策略、掉队节点隔离以及异步检查点等软件协同优化措施，这些方案支持了1200亿参数Nemotron-3模型的无缝训练。最终，演讲强调了随着AI工作负载扩展至数万个GPU，向持续多机架验证转变的必要性。 NVIDIA GB200 NVL72 架构通过 NVSwitch 将 18 个节点（72 颗 GPU）整合至单机架 NVLink 域，提供 130 TB/s 聚合带宽，实现高达 30 倍的推理加速与 25 倍能效提升。 (2:46) 为避免多节点 NVLink 产生任务碎片，NVIDIA 与 SchedMD 联合开发了开源 Slurm 拓扑块插件，为高带宽作业强制设定严格的调度边界。 (4:16) 团队构建了加速 Slurm 模拟器，在 7 小时内重放 7 天的生产负载，通过优化小任务分段分配，将多租户集群占用率恢复至理论上限的 1% 以内并保持 95% 以上的利用率。 (7:15) 针对超万卡训练场景，系统通过 NCCL all-to-all 基准与工作负载测试动态隔离慢节点（stragglers），并利用异步 Checkpoint 与优化垃圾回收保障 1200 亿参数 Nemotron-3 的训练有效算力（goodput）。 (14:06) 面对超 50,000 节点的集群规模，运营范式已从单次上线测试转向贯穿全生命周期的持续多机架验证与模拟动态调优，相关经验正直接反哺至 GB300 等下一代平台。 (19:17)

Engineers from NVIDIA's cluster operations team outline the architectural and operational frameworks required to run GB200 and GB300 systems at scale, focusing on topology-aware scheduling, pre-deployment simulation, and proactive multi-rack validation. NVL72 Architecture and Slurm Topology Scheduling (2:46) The transition to the GB200 NVL72 architecture shifts interconnects from single-node constraints to multi-node domains. Using NVSwitches, 18 nodes housing 72 GPUs are linked into a unified NVLink domain delivering 130 TB/s aggregate bandwidth, which the team notes provides up to a 30x inference speedup and a 25x improvement in energy efficiency over prior generations. Because traditional schedulers handle network topology on a best-effort basis, standard Slurm deployments risked severe job fragmentation on NVL72 fabrics. In response, NVIDIA worked with SchedMD to create an open-source topology block plugin that models each multi-node NVLink domain as a non-negotiable scheduling block. High-throughput workloads, such as Mixture of Experts (MoE), are divided into structured segments—such as splitting a 32-node job into two equal 16-node segments across two blocks—to protect input/output bandwidth while accommodating varied job sizes. Simulation-Driven Recovery of Cluster Occupancy (7:15) Maintaining the team's operational target of at least 95% GPU occupancy across more than 4,000 internal users proved difficult under rigid block placement rules. Initial configurations produced stranded capacity, dropping occupancy by several percentage points compared to unconstrained scheduling. To evaluate solutions before physical hardware arrived, the team built an accelerated Slurm simulator capable of replaying 15,000 production jobs across a 5,000-node cluster (20,000 GPUs) over a 7-day period in 7 hours—a 24x speedup. The simulation revealed that the block plugin naturally packed smaller jobs into the trailing nodes of a block, leaving contiguous segments available for massive runs with minimal self-fragmentation. By allowing low-demand jobs to spread across blocks, the team recovered lost capacity, bringing operational occupancy to within 1% of unconstrained scheduling benchmarks. These segment rules were then implemented transparently via Slurm CLI filters and hooks. Continuous Validation and Straggler Mitigation (14:06) With frontier clusters exceeding 50,000 nodes and single jobs spanning tens of thousands of GPUs, hardware degradation becomes a statistical certainty. To safeguard researcher time, the team abandoned static, one-time bring-up testing in favor of continuous, multi-rack validation throughout the cluster's operational lifecycle. Testing spans microbenchmarks for compute, storage, CPU, and network performance up to full multi-rack workload validation. Slow GPUs or degrading racks—termed stragglers—can bottleneck entire training runs. NVIDIA detects these nodes using NCCL all-to-all benchmarks, workload verifications, and set intersection analysis comparing slow and healthy jobs. Isolated units are automatically routed into a poor performer reservation for root-cause analysis. Testing frequency adapts dynamically, maintaining a light footprint during stable runs and triggering scale-wide parallel sweeps if interruptions emerge. Software Optimization for Nemotron-3 Production (17:43) The operational framework supported the end-to-end training of Nemotron-3, a 120-billion parameter hybrid Mixture-of-Experts model trained on tens of thousands of GPUs processing hundreds of billions of tokens daily. Achieving high goodput—the ratio of useful training computation to total elapsed time—required co-designing infrastructure and application software. Software enhancements focused on eliminating runtime interruptions. The team deployed asynchronous checkpointing and optimized garbage collection routines, which sharply reduced run-to-run performance variability and prevented training pauses during extended runs. NVIDIA released the resulting model weights, training recipes, and datasets to the broader ecosystem. NVIDIA's operators emphasize that the scheduling mechanisms, simulation methodologies, and continuous validation processes developed for GB200 clusters are directly informing the deployment of next-generation GB300 systems, as exponential model scaling turns rack-scale architecture into an operational baseline.

NVIDIA集群团队阐述了GB200及下一代机架级集群的落地与运维经验，指出随着大语言模型推动集群规模年增2.0至2.5倍，运营必须依靠软硬件协同设计、拓扑感知调度、离线模拟及持续自适应诊断，在维持高利用率的同时应对规模化故障。 GB200 NVL72架构与Slurm拓扑块调度 (2:46) GB200 NVL72打破了过去NVLink仅限单服务器节点内的限制，借助NVSwitch将18个节点（共72颗GPU）整合成单一NVLink域，总聚合带宽达130 TB/s。NVIDIA表示，Blackwell架构借此实现比以往架构高出30倍的推理加速和25倍的能源效率。然而，传统Slurm调度器对网络拓扑采用“尽力而为（best effort）”分配，在GB200环境会导致严重的作业碎片化与互连性能降级。 为此，NVIDIA与SchedMD联合开发了开源的拓扑块插件（topology block plugin），将每个多节点NVLink域定义为严格的不可分割调度块。针对混合专家模型（MoE）等高带宽需求任务，系统采用切分策略（例如将32节点作业均分为两个16节点段分别置于两个NVLink块），以保障I/O吞吐；对低I/O作业则分配较小段大小（如1或2），在保持通信性能的同时减轻集群织构的碎片化。 借助仿真恢复集群占用率与解决碎片化 (7:15) NVIDIA内部有超过4000名用户运行从单卡到数万张GPU的多元工作负载，团队设立了95%以上的GPU占用率目标（即闲置率低于5%）。但在引入刚性的拓扑块插件后，初期集群占用率相比无拓扑约束调度下降了数个百分点，造成运力滞留。由于GB200物理硬件当时尚未完全铺开，团队构建了加速的Slurm模拟器，以7小时运行完包含节点故障场景的5000节点（20000张GPU）集群与15000个作业在7天内的重放，实现24倍仿真加速。 仿真分析发现，拓扑块插件会自动将小作业聚集到每个块最后的两个节点，从而为超大作业留出大片连续空间，几乎没有引发自碎片化。通过允许较小作业跨块分布，团队将集群总体占用率恢复至距离无拓扑约束理论上限不到1%的水平，并利用Slurm命令行过滤器和钩子将段配置自动化，对上层研究人员完全透明。 慢节点自动诊断与全生命周期自适应验证 (14:06) 在数万张GPU协同训练的场景下，单颗低性能GPU（straggler）或劣化机架便会拖慢整体进度。针对这一问题，团队建立了包含微基准测试（网络、计算、存储与MoE CPU性能）及单节点、单机架到跨机架端到端负载的测试体系，并开发统一命令行工具实现自动化执行。通过针对MoE工作负载的NCCL All-to-all基准测试，并对比慢速作业与正常作业的GPU交集分析，系统可动态识别并隔离慢节点至专用故障排查队列。 团队随之调整了测试强度策略：如果数千卡任务运行平稳，日常测试即保持最低限度；一旦出现异常，系统自动触发大规模并行测试或跨机架隔离验证。随着前沿实验室集群规模超过50000个节点，硬件故障已成为必然现象，运维流程已彻底从最初的一次性上线验收，转向贯穿运行周期的全天候多机架自适应验证。 Nemotron-3万卡训练的软硬件协同优化 (17:43) 上述运维架构在Nemotron-3模型的实际训练中得到了验证。该模型具有1200亿参数，采用混合专家架构并支持长上下文窗口，在数万张GPU上每日处理数千亿token。为提升有效计算时间与实际消耗时间之比（即goodput），团队对底层软硬件进行了联合调优。 在软件层面，团队引入了异步检查点保存机制，并优化了底层垃圾回收逻辑，消除了训练中断与停顿，显著压低了不同批次运行之间的性能波动。NVIDIA指出，Nemotron-3作为开源生态项目，开放了模型权重、数据集和训练配方，其实践积累的运行数据也成为后续机架级系统优化的基准。 NVIDIA团队强调，从GB200跨越至GB300等更大规模机架计算系统，规模化带来的故障常态化要求基础设施在硬件部署、模拟仿真、动态拓扑调度与持续诊断之间紧密闭环；但对于极端超大规模下如何彻底消除慢节点与非计划停机，仍依赖测试与运维机制的进一步演进。

## Background 该内容由 @Scale 频道于 2026 年 7 月 2 日发布在 YouTube 平台上。页面简介显示，演讲由 NVIDIA 团队的 Ankur Srivastava、Douglas Wightman 和 Sachin Kumar Lakharia 共同呈现，重点探讨了将具备单机架 Exascale 级算力的 GB200 NVL72 系统进行规模化投产与运营的实战经验，以及如何通过合理的调度策略充分释放 NVLink 的超高网络带宽。 ## TL;DR NVIDIA 核心工程团队系统性分享了下一代 GB200 NVL72 超大规模集群的落地全流程。面对机架内 72 颗 GPU 统一 NVLink 域带来的极高带宽与硬边界限制，团队与社区合作开发了 Slurm 拓扑块插件，并通过超高加速比的 Slurm 仿真器进行参数调优，将刚性调度下的集群占用率差距压缩至 1% 以内。此外，团队构建了覆盖多尺度的自适应测试体系与掉队节点差集定位算法，配合异步 Checkpointing 等全栈优化，成功护航数万卡集群完成 1200 亿参数 Nemotron 大模型的端到端持续训练。最后，团队总结了向 GB300 等机架级计算演进的工程沉淀，并倡导以持续验证与生态开源赋能业界。 ## Atmosphere & style 整场演讲呈现出典型的高水准大厂工业级技术复盘氛围，沉稳严谨且干货密度极高。三位主讲人分工明确、环环相扣，从硬件互联、调度器改造逐步深入到仿真实验与万卡训练实战，层层拆解“峰值性能”与“数据中心整体利用率”之间的根本冲突。观感上不仅充满硬核的系统级工程智慧与第一视角下的踩坑沉淀，更在收尾时展现出赋能开源生态、迈向机架级计算新纪元的澎湃技术雄心。 ## Bulletin - [02:46](https://www.youtube.com/watch?v=1o4emwSid78&t=166s) NVIDIA GB200 NVL72 首次将 NVLink 拓展为多节点互联，单机架 18 个计算节点串联出拥有 130 TB/s 聚合带宽的 72 卡统一域，推理性能与能效比分别提升高达 30 倍与 25 倍。 - [04:08](https://www.youtube.com/watch?v=1o4emwSid78&t=248s) 针对 NVL72 硬拓扑边界导致的调度碎片化，NVIDIA 联合 SchedMD 研发并开源了 Slurm 拓扑块（Topology Block）插件，配合大小作业分段策略，将受拓扑约束的集群占有率与理论无拓扑极限差距收窄至 1% 以内。 - [09:42](https://www.youtube.com/watch?v=1o4emwSid78&t=582s) 团队开发出支持 24 倍时钟加速的原生 Slurm 仿真器，在硬件实装前仅用 7 小时即可重放 5,000 节点、20,000 颗 GPU 规模下的 7 天 15,000 个生产作业调度轨迹，完成了关键策略验证。 - [16:52](https://www.youtube.com/watch?v=1o4emwSid78&t=1012s) 依托自适应多级自动化测试平台与差集排查法，团队成功隔离掉队节点（Straggler），并在数万卡集群上完成了 120B 参数 Nemotron 模型的端到端全量训练，实现每日数千亿 Tokens 吞吐。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=1o4emwSid78&t=0s) 伴随大模型算力需求爆发，NVIDIA 内部集群规模以每年 2 至 2.5 倍的速度连年扩张，算力底座横跨 A100、H100、GB200 到 GB300 等多代异构架构。 - [02:46](https://www.youtube.com/watch?v=1o4emwSid78&t=166s) GB200 NVL72 通过 NVSwitch 突破了单机限制，将 18 个计算节点整合为拥有 130 TB/s 聚合双向带宽的 72 颗 GPU 统一 NVLink 域。 - [04:08](https://www.youtube.com/watch?v=1o4emwSid78&t=248s) 传统 Slurm 的尽力而为（Best-effort）调度会导致严重的拓扑碎片化和跨交换机带宽骤降，无法满足多节点全互联体系与 MoE 负载的硬通信需求。 - [04:08](https://www.youtube.com/watch?v=1o4emwSid78&t=248s) NVIDIA 与 SchedMD 联合研发的 Slurm 拓扑块插件（已开源入主干）将多节点 NVLink 域显式建模为刚性调度单元，允许大作业规整拆分为多段分配。 - [07:15](https://www.youtube.com/watch?v=1o4emwSid78&t=435s) 在支撑超 4,000 名研发人员且坚持 95% 以上集群 GPU 占有率红线的前提下，刚性拓扑约束原本会导致集群占有率出现多个单百分点的断崖式下跌。 - [09:42](https://www.youtube.com/watch?v=1o4emwSid78&t=582s) 团队构建了支持 24 倍加速的原生 Slurm 仿真器，通过在 20,000 颗 GPU 规模上重放 15,000 个真实任务，证实小作业的尾部填充效应能有效避免自我碎片化。 - [12:01](https://www.youtube.com/watch?v=1o4emwSid78&t=721s) 生产调度采用分段差异化策略：大型作业强制 Segment Size 为 16 以独占高速通信域，小型作业允许 Segment Size 设为 1 或 2 进行跨域填充以回收闲置容量。 - [12:01](https://www.youtube.com/watch?v=1o4emwSid78&t=721s) 团队通过修改 Slurm CLI 过滤器的底层机制，实现了根据作业规模透明改写调度参数，避免增加算法研究员的运维调试心智负担。 - [14:06](https://www.youtube.com/watch?v=1o4emwSid78&t=846s) 在数万卡超大规模集群中，单颗 GPU 或单个机架变慢形成的“掉队节点”（Straggler）足以拖垮整个分布式训练作业。 - [14:06](https://www.youtube.com/watch?v=1o4emwSid78&t=846s) 团队采用差集分析法（慢作业节点集合与正常作业节点集合求相对补集）快速收敛病灶，并将故障硬件隔离至专用的劣质节点保留池中排查。 - [14:06](https://www.youtube.com/watch?v=1o4emwSid78&t=846s) 测试体系告别固定节奏，转为自适应检测机制：正常运行时保持轻量监控，一旦出现异常则动态拉起数千节点乃至跨机架尺度的基准测试。 - [16:52](https://www.youtube.com/watch?v=1o4emwSid78&t=1012s) 自研的自动化测试平台支持一键式 CLI 排查，并结合异步检查点与深度 GC 控制，成功保障了数万卡集群端到端训练 120B Nemotron 模型。 - [16:52](https://www.youtube.com/watch?v=1o4emwSid78&t=1012s) 生产运营中采用有效产出率（Goodput）指标，精确审计物理运行时间中实际转化为模型梯度计算的比例，持续消除无谓开销。 - [18:41](https://www.youtube.com/watch?v=1o4emwSid78&t=1121s) AI 基础设施演进正在加速迈入以 GB200 和 GB300 为代表的机架级计算（Rack-scale Computing）范式，硬件故障已成为万卡集群常态。 - [18:41](https://www.youtube.com/watch?v=1o4emwSid78&t=1121s) 面向 GB300 演进，NVIDIA 确立了全生命周期持续多级验证、基于 Slurm 模拟器的自适应动态微调，以及机架级运营经验复用的三大演进方向。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=1o4emwSid78&t=0s) NVIDIA 集群规模演进与团队定位 在演讲开化之初，Sachin Lakharia 首先向与会者介绍了自己以及同台分享的同事 Douglas Wightman 和 Ankur Srivastava。本次分享的主题聚焦于 NVIDIA 团队如何将下一代 GB200 集群进行规模化投产与运营。在深入硬核的技术挑战之前，Sachin 阐述了团队在 NVIDIA 整体生态中所承担的四个核心战略角色：首先是“部署与扩展”（Deploy and Scale），负责将最尖端的计算、存储与网络硬件引入并转化为超大规模 HPC 集群；其次是“运营专家”（Operational Experts），针对前沿 AI 研发负载对 HPC 集群进行极致调优以释放峰值性能；第三是“构建与改进”（Build and Improve），研发先进的监控体系与研究员生产力工具；最后，所有这些工程沉淀共同反哺“生态创新”（Ecosystem Innovation），推动软硬件全栈技术的进化。 近几年来，伴随大语言模型（LLM）研究对算力需求的爆炸式激增，NVIDIA 内部集群规模以每年 2 到 2.5 倍的速度连年扩张。其算力底座横跨数代架构，从早期的 A100、H100，一直演进到如今最新的 GB200 和 GB300。Sachin 回顾道，GB200 的工程探索始于一年多以前的软硬件协同设计阶段，历经早期实验性的 Pre-cluster 验证、生产就绪（Production Readiness）阶段的大规模仿真开发，再到最终的全球范围铺开。整个技术分享也正是遵循这一落地链路层层展开。 ```mermaid flowchart LR accTitle: NVIDIA Cluster Scaling and Lifecycle accDescr: 展现团队的四大核心角色与 GB200 从协同设计到全球落地的生命周期。 subgraph Roles["团队四大核心职责"] R1["部署与扩展"] --> R2["运营优化"] R2 --> R3["工具构建"] R3 --> R4["生态创新"] end subgraph Lifecycle["GB200 落地演进历程"] L1["软硬件协同设计"] --> L2["Pre-cluster 验证"] L2 --> L3["生产就绪与仿真"] L3 --> L4["全球集群铺开"] end ``` - **业务年复合增长率**：集群算力规模实现每年 2x 到 2.5x 的年对年（YoY）复合增长。 - **架构支持跨度**：涵盖 A100、H100、GB200 至 GB300 等多代异构 GPU 架构。 - **技术落地分工**：Douglas 负责拓扑与调度演进，Sachin 负责前置仿真与投产准备，Ankur 负责测试验证与全量铺开。 --- ### [02:46](https://www.youtube.com/watch?v=1o4emwSid78&t=166s) GB200 NVL72 架构：从单机内部互联到多节点 NVLink 随后，Douglas Wightman 登台详细剖析了 GB200 带来的颠覆性硬件架构升级。与前代架构仅在单一服务器节点内通过 NVLink 互联不同，GB200 首次将 NVLink 拓展为多节点（Multi-Node）互联系统。通过核心枢纽 NVSwitch，系统将一个机架内多达 18 个计算节点直接串联在一起，构筑成拥有 72 颗全互联 GPU 的巨型单机超级计算机（NVL72），所有 GPU 均能以无损全速跨节点通信。 这一颠覆性变革的驱动力在于 AI 模型的爆发式演进——前沿模型的参数复杂度与通信需求已彻底超越了传统网络 Fabric 的承载能力。全新的 Blackwell 架构由此带来了巨大的飞跃：推理性能相比前代提升高达 30 倍，能效比改善 25 倍。其底层最关键的支撑，正是 NVSwitch 织网所构建出的跨 72 颗 GPU 统一 NVLink 域，可提供高达 130 TB/s 的惊人聚合互联带宽。 - **多节点互联形态**：单机架集成 18 个节点，通过 NVSwitch 互联形成 72 颗 GPU 的单一 NVLink 域。 - **聚合互联带宽**：NVL72 内部聚合双向带宽达到 130 TB/s。 - **性能与能效提升**：相比前代架构提供 30 倍的推理性能加速与 25 倍的能效优化。 --- ### [04:08](https://www.youtube.com/watch?v=1o4emwSid78&t=248s) 破解调度碎片化：Slurm 拓扑块（Topology Block）插件 拥有了 130 TB/s 的超级 Fabric 之后，系统面临的最严峻挑战转移到了底层的作业调度引擎——Slurm。Douglas 指出，虽然传统的 Slurm 早就具备网络拓扑感知调度能力，但以往的算法本质上属于“尽力而为”（Best-effort）。在面对 GB200 这种要求极其严苛的多节点全互联体系时，这种粗粒度机制完全捉襟见肘。GB200 集群必须将 NVLink 域视为不可打破的“硬边界”（Hard boundaries）。为此，NVIDIA 团队与 SchedMD 展开深度合作，联合打造了全新的 Slurm 拓扑块插件（Topology Block Plugin）。 ```mermaid flowchart TD accTitle: Slurm Topology Block Plugin Concept accDescr: 对比朴素调度器的碎片化分配与拓扑块插件基于块的规整分配。 subgraph Naive["传统朴素调度 (Best-effort)"] N1["32 节点作业请求"] --> N2["离散空闲节点放置"] N2 --> N3["跨 Leaf 交换机碎片化 / NVLink 带宽断崖"] end subgraph Block["拓扑块调度 (Topology Block Plugin)"] B1["32 节点作业请求"] --> B2["硬边界分段 (例如 2x16 分段)"] B2 --> B3["精准锁定规整 NVLink Block / 满足 MoE 高带宽"] end ``` 在传统的调度模式下，一个需要 32 个节点的作业可能被调度器随机塞进任意 32 个零散空闲节点中，即便调度器尝试将任务限制在叶子交换机（Leaf Switch）内，常规的拓扑匹配也会迅速妥协放弃，从而导致严重的拓扑碎片化。而借助拓扑块插件，每个多节点 NVLink 域被显式建模为一个刚性调度单元（Block），管理员可在配置文件中精确定义拓扑边界。在面对 32 节点的大作业时，调度器会智能地将其拆分为两个规整的 16 节点分段（Segments），精准分配到对应的 Block 中。这对于通信极度密集、极易受网络瓶颈制约的混合专家模型（MoE）训练负载至关重要。 Douglas 特别强调，为了榨干系统的通信潜力，针对不同应用需要精细化微调“分段大小”（Segment Size）：对于 MoE 等高通信开销负载，必须指定大分段以霸占高带宽 NVLink；而对于通信 IO 需求较低的较小任务，则可放宽并采用较小的分段尺寸以换取集群整体的高利用率。目前该功能已完全合并进入 Slurm 开源主干供全社区使用。 --- ### [07:15](https://www.youtube.com/watch?v=1o4emwSid78&t=435s) 生产环境的高标准与硬边界引发的利用率冲突 重新接棒的 Sachin 将话题引向落地过程中的深水区。一方面，NVIDIA 内部承载的计算负载类型异构且跨度极大：涵盖 LLM、生成式 AI 直至数字人（Digital Humans）等前沿方向；作业粒度小至单张 GPU，大至数万张 GPU 的超大集群分布式训练。另一方面，集群服务的要求近乎苛刻：超过 4,000 名内部研发用户共享这套多租户环境，需要严格的公平调度（Fair Sharing），同时团队设定了集群占有率（Cluster Occupancy）必须达到 95% 以上的刚性运营红线。 Sachin 对此澄清了“占有率”的量化定义：即集群中处于正在运行状态的 GPU 数量除以集群总可用 GPU 数量。95% 的占有率意味着在持续高需求的常态下，任何时刻闲置的 GPU 比例都不得超过 5%。然而，拓扑块插件带来的“硬约束”虽然保障了作业内部能吃满 NVL72 的吞吐，却给作业放置的灵活性戴上了沉重的枷锁。在没有专项策略调优的基线情况下，由于刚性块的限制，集群占有率与完全不考虑拓扑的理想调度（No-topology）相比，会出现“多个单百分点”（Multiple single digits）的肉眼可见的断崖式暴跌。 - **多租户规模**：服务全公司超过 4,000 名 AI 算法研究员与工程师。 - **任务规模跨度**：单机单卡（1 GPU）至万卡规模（Tens of thousands of GPUs）。 - **运营硬指标**：全集群运行状态 GPU 占比（Occupancy）长期坚守 95% 以上高位，闲置率压制在 5% 以内。 --- ### [09:42](https://www.youtube.com/watch?v=1o4emwSid78&t=582s) 构建 Slurm 仿真器与去碎片化关键发现 为了化解“作业峰值性能”与“数据中心整体占有率”之间的尖锐对立，团队在尚未拿到真实 GB200 硬件的阶段，选择依托仿真系统先行破局。他们打造了一套一等公民（First-class）级别的超逼真 Slurm 仿真器。该仿真平台直接运行原生 Slurm 代码库，能够无缝加载不同版本的调度内核、重现生产环境的真实拓扑与调度配置，并能直接抓取并重放生产环境的历史作业 Trace。不仅如此，仿真器支持极高的时间加速比——原本需要 7 天的生产轨迹，在仿真环境中仅需 7 个小时即可跑完，从而允许团队以高频迭代测试海量的调度参数组合。 针对 GB200，团队建立了一个由 5,000 个节点（等效于 20,000 颗 GPU）组成的超大规模虚拟集群，在为期 7 天的加速窗口中灌入 15,000 个真实作业，同时模拟了部分节点进入故障排查（Triage）的不稳定工况。这场跨越数月的仿真实验带来了令人惊喜的核心突破： 在碎片化方面，直觉往往认为小作业会随机侵占规整的 Block，将其切割得支离破碎并导致后续大作业无法入驻。但仿真数据显示，拓扑块插件表现出惊人的打包能力，它能够巧妙地把小作业压缩进最后剩余的两三个边角节点中，从而为后续的大规模作业完整保留下大颗粒度的规整分段，系统几乎“完全没有出现自我碎片化”（Very minimal to no self-fragmentation）。而在占有率指标上，尽管完全不限制拓扑在理论上能获得最高的调度灵活性，但只要集群中存在足够基数的小作业来扮演“填充物”（Fillers），调度器就能成功将刚性块调度造成的空隙重新填满，最终使得受拓扑严格限制的集群占有率与理想无拓扑基准之间的差距成功收窄至 1% 以内。 - **仿真器加速效能**：支持 24 倍时钟加速，7 天规模的大型集群作业运行仅耗时 7 小时即可回放完毕。 - **基准测试规模**：模拟 5,000 个节点（20,000 颗 GPU），7 天时间线上高保真重放 15,000 个生产作业。 - **关键调度结论**：小作业具备优秀的尾部对齐填充效应，未产生预想中的严重内部碎片化，最终集群占有率与理论极限值的差距被压减至 1% 以内。 ### [12:01](https://www.youtube.com/watch?v=1o4emwSid78&t=721s) 拓扑调度策略与从模拟到生产的跨越 在深入分析机架拓扑与集群占用率的关系时，Sachin 指出无拓扑限制（No Topology）理论上能提供最高的设备占用率，也就是所谓的“光速占用率”（speed-of-light occupancy）。然而，一旦强制要求所有作业都在完整的 NVL72 域内以严格拓扑对齐的方式运行，整体算力利用率就会出现显著下滑。造成这一现象的核心原因在于调度器失去了灵活摆放作业的空间，从而导致大量碎片化的算力容量被“搁浅”（stranded capacity）。为了破解这一难题，团队采取了灵活放开的策略：允许小型作业自由跨域分散调度，从而成功挽回了被闲置的资源。 ```mermaid flowchart TD accTitle: 拓扑调度与资源分配策略 accDescr: 展示作业规模与切片大小分配策略，以及通过 Slurm CLI 过滤器的透明化下发流程。 subgraph JobTypes["作业分流策略"] Large["超大作业 (大型训练)"] Small["小型作业 (测试/微调)"] end subgraph Allocation["分段与拓扑配置"] Seg16["Segment Size = 16<br/>严格 NVL72 拓扑对齐"] Seg2["Segment Size = 1 或 2<br/>允许跨域分散排布"] end subgraph Implementation["调度系统实现"] SlurmFilter["Slurm CLI 动态过滤机制"] ExecEnv["生产环境透明下发执行"] end Large --> Seg16 Small --> Seg2 Seg16 --> SlurmFilter Seg2 --> SlurmFilter SlurmFilter --> ExecEnv ``` 与此同时，分段大小（Segment Size）的设定对于性能表现至关重要。不均衡的资源分配会直接引发作业吞吐量的严重滑坡。针对这一特性，团队为集群用户制定了明确的推荐标准：占用绝大部分算力的大型作业必须采用分段大小为 16 的配置，以跑出极限性能；而小型作业则应将分段大小限制在 2 甚至是 1。为了不增加用户的认知负担，团队通过修改 Slurm CLI 过滤器等底层拦截机制将这套规则完全内嵌，使整个调度优化对研究员透明无感。Sachin 强调，模拟环境的理想数据只是起点，真实的生产环境对稳定性和可预测性的要求苛刻得多：研究人员的时间应当百分之百投入在模型架构迭代上，绝不应该被迫把精力消耗在底层基础设施的调试排错中。 - **大型作业分配准则**：设定 Segment Size 为 16，独占连续高速域以释放最高通信性能。 - **小型作业调度策略**：设定 Segment Size 为 1 或 2，允许散布在非连续节点以回收碎片化资源。 - **调度透明化封装**：通过 Slurm CLI 过滤器实现对用户透明的自动化参数重写。 --- ### [14:06](https://www.youtube.com/watch?v=1o4emwSid78&t=846s) 自适应测试与掉队节点（Straggler）定位 接过话茬的 Ankur Srivastava 阐述了构建高性能、高可靠集群背后的严苛验证体系。要交付真正可用的大规模系统，不能仅依赖单一维度的基准测试，而是必须构建覆盖网络、计算和存储的多层次微基准测试（microbenchmarks）与真实工作负载测试。在现代混合专家架构（MoE）负载中，CPU 的角色同样不容轻视，团队为此专门开发了针对 CPU 的微基准测试工具。更为关键的是，这些验证必须跨越单节点、单机架以及多机架集群等不同尺度，因为在数万颗 GPU 协同作业的超大集群中，任何单点迟钝都会演变为灾难。正如 Ankur 警示的那样：“请记住，哪怕只有一颗 GPU 或一个机架出现变慢，就足以拖慢甚至彻底毁掉横跨成千上万颗 GPU 的整个训练作业。” ```mermaid flowchart LR accTitle: 掉队节点检测与隔离流程 accDescr: 从集群运行异常中排查掉队节点，并通过布尔交集分析将其移入受限测试池。 subgraph Detection["异常监测机制"] A["NCCL All-to-All 基准测试"] B["真实工作负载探针"] C["运行数据交集对比分析"] end subgraph Triage["根因隔离流程"] Target["锁定掉队节点 / 机架"] Pool["移入劣质节点保留池<br/>(Poor Performer Reservation)"] Diagnosis["深度排查与根因分析"] end A --> Target B --> Target C --> Target Target --> Pool Pool --> Diagnosis ``` 针对这种“掉队节点”（Straggler）问题，团队摸索出了一整套组合检测拳。除了面向 MoE 算子的高压力 NCCL All-to-All 集合通信测试之外，团队还常用一种极其实用且高效的即席差集分析法（Ad-hoc Analysis）：把所有表现恶劣或变慢的作业所用到的 GPU 节点取交集，并从中剔除所有在健康作业中正常运行的节点，剩下的节点就大概率是隐藏的病灶。一旦揪出问题节点或迟钝机架，系统会立刻将其划入专用的“劣质节点保留池”（Poor Performer Reservation）进行离线深挖诊断，以彻底查清故障根因。 在测试执行时机上，团队摈弃了固定节奏的机械模式，转向因地制宜的自适应策略。如果当天数千颗 GPU 的训练流程平稳顺畅、无中断无减速，测试系统就会保持静默或仅执行极其轻量的监测；可一旦某天训练遭遇异常，系统就会立刻升级响应级别，动态划拨算力并在数千个节点上并行拉起大规模测试，甚至锁定跨机架容量来专门排查跨域通信隐患。 - **木桶效应危害**：单颗 GPU 或单个机架的亚健康足以将数万卡集群的整体迭代吞吐拉垮。 - **掉队排查算法**：提取慢作业节点集合与正常作业节点集合求相对补集，快速收敛嫌疑硬件。 - **硬件降级保留池**：隔离确诊的异常硬件至独立资源池，防止其污染生产作业队列。 --- ### [16:52](https://www.youtube.com/watch?v=1o4emwSid78&t=1012s) 自动化测试平台与百亿级模型生产实战 为了将自适应测试策略真正落地，自动化成为了不可逾越的技术门槛。过去由少数几名核心工程师频繁手写批处理提交脚本的作法完全无法应对动态变化的高频测试需求。为此，团队投入大量精力打造了一套通用的自动化测试框架。该工具不仅适配各种不同类型的 GPU 硬件，而且内建了极为丰富的微基准测试与真实负载测试套件库。运维与算法团队只需敲下一行极其简洁的 CLI 命令，就可以按需在任意节点尺度上发起对应测试并迅速获取结论，使针对硬件波动的决策响应变得敏捷精准。 在这套软硬件协同方案的护航下，团队与 NeMo 团队紧密协作，完成了拥有 1200 亿参数（120B）的 Nemotron 大模型的端到端全量训练。该任务调动了数万颗 GPU 规模的集群，创下了每天吞吐数千亿 Tokens 的惊人处理速度。 这一突破来自于软硬件全栈层面的深度调优。在软件层面，团队引入了异步检查点保存机制（Async Checkpointing）和深度垃圾回收（GC）控制，大幅减轻了因 IO 停顿和内存回收引起的训练中断与各轮次间的运行波动（run-to-run variability）。同时，系统还在训练代码内部深度埋点，用于持续监控并量化运行期“有效产出率”（Goodput）——即在流逝的物理时钟时间（Wall-clock time）中，到底有多少比例被切实转化为了生产性的模型梯度计算，从而倒逼系统不断剔除隐藏的无效开销。 - **一键式自动化测试**：告别手工调度脚本，通过统合 CLI 命令在任意规模的集群节点上瞬时展开基准排查。 - **120B 规模实战验证**：数万卡超大规模集群协同，支撑了日处理数千亿 Tokens 的生产级持续训练。 - **训练有效算力（Goodput）监控**：穿透单纯的 GPU 占用表象，精确审计有效前向与反向传播在物理运行时间中的实际占比。 --- ### [18:41](https://www.youtube.com/watch?v=1o4emwSid78&t=1121s) 开源生态、机架级计算演进与 GB300 展望 重新接过麦克风的 Sachin 对 Nemotron-3 模型的发布背景进行了补充。他介绍道，Nemotron-3 采用了先进的混合专家（Hybrid MoE）网络结构并具备长上下文处理能力。更重要的是，它秉承了全开放的生态定位，无论是模型权重、训练数据集还是调优配方（recipes）均已完整开源，全球开发者与研究人员都可以自由下载、评估与微调。 随后，Sachin 将视线投向了 AI 基础设施演进的宏大趋势。随着模型复杂度的呈指数级暴增，传统的单节点或多节点堆叠正在加速向以 GB200 和 GB300 为代表的机架级计算（Rack-scale Computing）范式迁移。如今最前沿的 AI 实验室管理的集群节点规模已经突破了 50,000 大关，主流单次训练作业更是动辄跨越数千个节点、调度数万颗 GPU。在如此巨大的物理规模下，硬件故障不再是偶然事件，而是成了不可避免的日常客观规律。 为了直面这一物理现实，团队确立了面向未来的三大演进路径： 1. **持续多级验证（Continuous Validation）**：彻底告别过去仅在开荒交付时执行的一次性测试，将自动化健康检测转变为全生命周期的持续行为，且测试颗粒度从单节点、多节点全面延伸至多机架维度。 2. **自适应调度闭环（Adaptive Efficiency）**：深度应用 Slurm 模拟器技术，根据动态工作负载与拓扑特征实时对物理集群实施参数微调，将硬件算力利用率推向理论极限。 3. **经验沉淀与机架级工程演进**：将生产运营 GB200 集群所积累的一线硬核经验，全量迁移并复用到下一代以 GB300 为代表的超大规模机架级系统中，实现更早期的系统稳定性收敛。 在演讲的尾声，Sachin 诚挚呼吁业界去切身体验 NVIDIA GB300 等机架级计算平台所带来的极致能效，并强调 NVIDIA 绝不仅是优化自身的内部基建，更致力于通过开放前沿 AI 基础设施研究与开源模型生态赋能整个 AI 工业界。他满怀激情地总结道：“激动人心的未来就在眼前，欢迎大家加入我们的使命，共同打造下一代通用智能的前沿阵地。” ## Notable quotes > [00:00](https://www.youtube.com/watch?v=1o4emwSid78&t=0s) 部署与扩展 — Sachin Lakharia > [00:00](https://www.youtube.com/watch?v=1o4emwSid78&t=0s) 运营专家 — Sachin Lakharia > [00:00](https://www.youtube.com/watch?v=1o4emwSid78&t=0s) 构建与改进 — Sachin Lakharia > [00:00](https://www.youtube.com/watch?v=1o4emwSid78&t=0s) 生态创新 — Sachin Lakharia > [09:42](https://www.youtube.com/watch?v=1o4emwSid78&t=582s) 完全没有出现自我碎片化 — Sachin Lakharia > [12:01](https://www.youtube.com/watch?v=1o4emwSid78&t=721s) 光速占用率 — Sachin Lakharia > [14:06](https://www.youtube.com/watch?v=1o4emwSid78&t=846s) 请记住，哪怕只有一颗 GPU 或一个机架出现变慢，就足以拖慢甚至彻底毁掉横跨成千上万颗 GPU 的整个训练作业。 — Ankur Srivastava > [14:06](https://www.youtube.com/watch?v=1o4emwSid78&t=846s) 劣质节点保留池 — Ankur Srivastava > [16:52](https://www.youtube.com/watch?v=1o4emwSid78&t=1012s) 有效产出率 — Ankur Srivastava > [18:41](https://www.youtube.com/watch?v=1o4emwSid78&t=1121s) 持续多级验证 — Sachin Lakharia > [18:41](https://www.youtube.com/watch?v=1o4emwSid78&t=1121s) 自适应调度闭环 — Sachin Lakharia > [18:41](https://www.youtube.com/watch?v=1o4emwSid78&t=1121s) 激动人心的未来就在眼前，欢迎大家加入我们的使命，共同打造下一代通用智能的前沿阵地。 — Sachin Lakharia ## Mentioned NVIDIA, Sachin Kumar Lakharia, Douglas Wightman, Ankur Srivastava, GB200, GB200 NVL72, GB300, A100, H100, Blackwell, NVLink, NVSwitch, Slurm, SchedMD, Topology Block Plugin, MoE, NCCL, Nemotron, Nemotron-3, NeMo, @Scale ## Video description Speakers: Ankur Srivastava, Douglas Wightman, and Sachin Kumar Lakharia The NVIDIA GB200 NVL72 functions as an exascale computer within a single rack. Maximizing the utilization of its NVLink fabrics through proper scheduling allows AI training jobs to significantly leverage the substantial networking bandwidth this system offers. Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=1o4emwSid78)

[0:00] Sachin Lakharia: 部署与扩展 [0:00] Sachin Lakharia: 运营专家 [0:00] Sachin Lakharia: 构建与改进 [0:00] Sachin Lakharia: 生态创新 [9:42] Sachin Lakharia: 完全没有出现自我碎片化 [12:01] Sachin Lakharia: 光速占用率 [14:06] Ankur Srivastava: 请记住，哪怕只有一颗 GPU 或一个机架出现变慢，就足以拖慢甚至彻底毁掉横跨成千上万颗 GPU 的整个训练作业。 [14:06] Ankur Srivastava: 劣质节点保留池 [16:52] Ankur Srivastava: 有效产出率 [18:41] Sachin Lakharia: 持续多级验证 [18:41] Sachin Lakharia: 自适应调度闭环 [18:41] Sachin Lakharia: 激动人心的未来就在眼前，欢迎大家加入我们的使命，共同打造下一代通用智能的前沿阵地。

[20:48] Exciting future is ahead of us. Please join us in our mission to build the next frontier of intelligence.

[0:00] Good afternoon everyone. [0:02] My name is Sachin Lakharia and with me I [0:05] have my colleague Douglas Wittman and [0:08] Ankur Srivastava. [0:10] Today we'll be sharing our experience [0:12] productionizing and operating GB200 [0:15] clusters at scale. [0:19] Before we go into the technical [0:21] challenges, let me quickly introduce our [0:24] team and the strategic role that it [0:26] plays in the NVIDIA ecosystem. [0:29] Firstly, with deploy and scale, our team [0:32] is responsible for bringing up the next [0:35] generation of compute, storage, and [0:38] networking and converting them into HPC [0:42] clusters. [0:43] Secondly, we come in as operational [0:46] experts where we optimize the HPC [0:49] clusters to deliver peak performance for [0:52] our AI research workloads. [0:55] Thirdly, with build and improve, we are [0:57] responsible for developing advanced [1:00] monitoring tools and researcher [1:02] productivity tools. [1:04] And lastly, all of this goes into the [1:06] ecosystem innovation where we help [1:09] advance the hardware and the software [1:12] stack, not just for NVIDIA, but for the [1:15] entire industry. [1:18] Since this talk is about scale, let me [1:20] share our scale journey with you. As you [1:23] can see, we have grown tremendously in [1:26] the last few years. [1:28] We have grown more than two to 2.5x year [1:31] over year in these years. [1:34] Most of this growth has been primarily [1:36] driven by the demand coming from large [1:39] language model research. [1:42] Our fleet comprises of diverse NVIDIA [1:45] architecture ranging from A100 to H100 [1:49] and now we have GB200 and GB300 [1:52] clusters. [1:54] Today we'll be sharing our experience [1:56] and our GB200 journey with you from [1:59] inception to the global rollout. This [2:02] journey that started more than a year [2:04] ago with hardware and software co-design [2:08] and then into Nvidia pre-clusters. [2:11] My colleague Douglas will be talking [2:12] about some of the work that goes during [2:14] these phases, especially around like [2:17] scheduling. [2:18] Then we will talk about the production [2:20] readiness. Well, I'll be talking about [2:23] the simulation and development work that [2:26] goes before the production rollout. And [2:28] eventually Ankur will be talking about [2:30] the testing and validation that enables [2:34] the global rollout of GB200. [2:37] With that, let me pass it on to Douglas [2:39] who will talk about the NVL GB200 [2:42] system. [2:46] >> Thank you, Sachin. [2:48] The GB200 was designed with several [2:50] major enhancements and advancements. And [2:53] unlike previous generations that used [2:56] NVLink within a single machine, the [2:58] GB200 introduced multi-node NVLink. [3:02] So this system interconnects GPUs on one [3:05] node with direct links to the GPUs on [3:08] many other nodes [3:09] via NVSwitches. [3:12] This turns 18 nodes into a giant [3:15] supercomputer, like it was mentioned [3:17] earlier. So we now have 72 fully [3:20] connected GPUs communicating at full [3:24] speed. [3:28] These architectural changes [3:32] were driven by a clear trend in the [3:34] industry. Model complexity is rapidly [3:38] outgrowing existing fabrics. So the new [3:41] Blackwell architecture provides huge [3:43] improvements in this area. A 30x [3:46] inference speed up compared [3:50] and a 25x improvement to energy [3:52] efficiency. [3:53] And the key enabler to this new [3:55] technology is the NV switch system, [3:57] which like I said, connects those 72 [3:59] GPUs into a single NV link domain with [4:03] 130 terabytes per second of aggregate [4:06] bandwidth. [4:08] However, [4:09] to take advantage of these advancements, [4:13] a big change was required to our [4:15] underlying workload scheduling engine, [4:18] Slurm. [4:22] Now, Slurm has always supported [4:25] scheduling within network topologies, [4:27] but these were always best effort and [4:30] were insufficient for this new [4:32] architecture. [4:33] GB200 clusters require that the job [4:36] scheduler treat NV link domains as very [4:39] hard boundaries. So, we collaborated [4:42] with SchedMD to build the topology block [4:46] plugin. [4:48] Each multi-node NV link domain is [4:50] modeled as a rigid scheduling unit or a [4:53] block. [4:55] And this topology can be defined by the [4:56] administrator in the Slurm [4:58] configuration, [4:59] making sure that sites can fine-tune [5:03] their bandwidth allocation for their [5:05] large jobs. [5:09] So, let's look at an example of this [5:12] kind of block scheduling. [5:15] A naive job scheduler trying to start a [5:17] 32-node job in many situations would [5:21] just place that on any 32 free nodes. [5:24] Maybe the scheduler would try to contain [5:27] it within a network leaf switch, but [5:29] with [5:30] the normal [5:32] leaf switch scheduling, that can give up [5:35] quickly, leading to fragmentation. [5:38] So, more intelligence was needed [5:40] to take advantage of the multi-node NV [5:43] link. [5:44] This slide demonstrates how Slurm's [5:47] topology block algorithm can efficiently [5:50] distribute the job across multiple [5:53] blocks, splitting them equally into two [5:56] segments of 16. [5:59] This distribution [6:01] is essential for utilizing the high [6:03] bandwidth needed by mixture of experts [6:07] type workloads. [6:11] Now, there can still be per application [6:13] tuning needed when it comes to segment [6:15] sizes to fully take advantage of Slurm's [6:18] topology block algorithm. [6:20] And we recommend application developers [6:23] play around, simulate to find what is [6:27] their optimal segment size. [6:29] Larger jobs, especially MOE workloads, [6:32] will require large segment sizes to take [6:35] advantage of that high bandwidth. [6:37] And conversely, smaller jobs with lower [6:39] IO bandwidth should use a smaller [6:41] segment size [6:43] to maximize cluster utilization. [6:45] And all of this is available open source [6:47] in Slurm, and we encourage you to try it [6:51] out on your systems with your workloads. [6:54] There can be tradeoffs [6:56] depending on the various workloads, and [6:58] strict topology scheduling [7:00] has the potential to impact overall data [7:03] center utilization, [7:05] but these can be mitigated. [7:07] Sachin is going to go over the workloads [7:09] and the simulations and the [7:10] methodologies that we use at Nvidia. [7:13] Sachin. [7:15] >> Thank you, Douglas, [7:17] for walking us through the block [7:18] topology plugin and some of the ways [7:21] that it works. [7:22] Next, I'll go into some of the [7:24] challenges that are imposed by block [7:26] topology plugin. But before that, let's [7:29] take a quick look at the scale and [7:30] variety of our workload that amplifies [7:32] these challenges even further. [7:35] So, we support diverse types of workload [7:37] ranging from LLMs, generative AI, to [7:40] digital humans. [7:41] In terms of the job shapes and sizes, we [7:43] have jobs which require just one GPU all [7:46] the way to jobs that needs like tens of [7:49] thousands of GPUs when they are running. [7:51] And above all, the service expectations [7:54] are really high. We are users uh expect [7:58] really high reliability from our [8:00] platform. We operate our clusters at 95% [8:03] or more occupancy, and I'll talk about [8:06] what what do I mean by occupancy in the [8:08] next slide. [8:09] And above all, because it's a [8:11] multi-tenant uh shared across all the [8:13] NVIDIA teams, they expect fair sharing [8:16] of the resources among more than 4,000 [8:19] of our of our users. [8:21] So, let's take a look at what are some [8:23] of the challenges with with the block [8:25] scheduling. [8:26] Uh first, let's define what do we mean [8:28] by occupancy. [8:30] Occupancy is defined as the total number [8:33] of GPUs running any workload divided by [8:36] the total number of GPUs available in a [8:38] cluster. And we define this at a cluster [8:40] level, and we, as I've said previously, [8:43] we strive for more than 95% cluster [8:46] occupancy. That means that at any [8:49] moment, no more than 5% of our GPUs [8:52] should be sitting idle. Of course, there [8:54] has to be enough demand, which is always [8:56] there. [8:57] Uh let's see what happens like, you [8:59] know, uh with block scheduling, the hard [9:02] constraint that the the block sched- uh [9:05] scheduler places, it guarantees [9:07] performance to the job because job will [9:09] get scheduled on the same NVL72 domain. [9:13] But at the same time, it actually limits [9:15] the scheduler flexibility [9:17] uh in placing the jobs. [9:19] And without any optimization, what we [9:21] observed that, you know, your occupancy [9:24] may fell by multiple single digits [9:26] compared to something like a no topology [9:28] where scheduler doesn't have to look at [9:30] topology at all. [9:32] Uh hence, uh strategic tuning was [9:34] required to bridge this gap between like [9:37] performance that we are delivering to [9:39] the jobs versus the overall cluster [9:41] utilization. [9:42] So, how did we bridge this gap? We [9:45] turned to simulation. [9:46] We did not had the GB200 environment to [9:49] play with because we didn't had any [9:51] GB200 cluster. [9:52] So, we built our first class Slurm [9:55] simulator. [9:57] The Slurm simulator runs the exact Slurm [9:59] code base. It is equipped with running [10:02] different versions of Slurm. It is also [10:04] equipped with trying different [10:06] configurations of scheduling that we [10:08] could play around whether it's topology [10:10] and other scheduler configuration. [10:13] And it's also equipped with fetching [10:15] data from production environment. And [10:18] once we we are able to replay this [10:21] with different scheduler configurations [10:24] and even compare these matrices metrics [10:27] to production environment. [10:29] It is also equipped with accelerated [10:31] runtime. [10:32] That means that you know, a 7-day [10:35] simulation could be done in 7 hours. It [10:37] allows us to iterate faster and try [10:39] different types of simulations. So, for [10:42] GB200, we simulated a cluster of 5,000 [10:45] nodes equivalent to 20,000 GPUs with a [10:48] workload of 15,000 jobs over a period of [10:50] 7 days and we modeled some reliability [10:54] by assuming some nodes are being in the [10:56] triage. [10:58] So, what are some of the the key [10:59] findings from our simulations? We ran [11:02] like you know, [11:03] simulations across like different [11:05] configuration for multiple months in [11:08] fact and some of the key findings are on [11:10] the fragmentation side. So, one is like [11:12] it's natural to believe that smaller [11:14] jobs can come in and fragment these [11:16] blocks and make them unavailable for the [11:18] larger jobs. However, during our [11:20] simulation what we observed was that the [11:23] block topology plugin was able to fit in [11:26] these smaller jobs to the last two node [11:28] and leaving [11:29] larger segments [11:31] for for the larger job. So hence from [11:33] like very minimal to no self [11:35] fragmentation was observed. [11:37] Uh next on the occupancy, which is one [11:39] of our key metrics, is that you know, [11:42] firstly we know that no topology of [11:44] course represents the best theoretical [11:46] occupancy that scheduler can give with [11:49] because of the maximum freedom that is [11:50] had. But given enough small jobs, we are [11:54] able to kind of you know, recover this [11:55] occupancy loss and we observed that we [11:58] were able to achieve occupancy within 1% [12:01] of the no topology, which is the speed [12:03] of light occupancy that we can get. [12:06] So [12:07] based on this, we came up with some [12:09] recommendations, but before that a quick [12:12] look into the workload and site. As I [12:15] shared, no topology gives us the best [12:18] occupancy. If we allow all jobs to [12:21] utilize the NVL-72 domain, we we see a [12:25] drop in the overall utilization. And [12:27] this is primarily coming from [12:29] like you know, scheduler not having that [12:31] flexibility to place the jobs and it [12:33] leaves some of the capacity stranded. [12:36] If we allow small jobs to spread, then [12:40] we are able to recover some of this [12:42] occupancy. And the other thing is the [12:44] segment size, which Douglas also talked [12:47] about that is very critical. We don't [12:49] want disproportionate allocation because [12:51] disproportionate allocation leads to a [12:53] performance drop in the jobs. [12:56] So based on this, we came up with some [12:57] recommendations for our users that you [13:00] know, the larger job should be done with [13:02] segment size of 16 for maximum [13:06] performance and you know, larger jobs [13:08] often use most of the compute. They [13:10] should be running with more performance [13:12] and smaller jobs should be done with a [13:14] segment size of two or even even one. [13:17] And we changed our defaults in the [13:20] cluster user make it user agnostic. We [13:22] changes some of this through you know, [13:25] the slum CLI filter and other mechanism [13:28] so that this is like transparent to the [13:29] users and we were able to like get good [13:32] occupancy results. [13:35] However, like you know, these are just [13:37] simulation [13:38] environment and the production [13:40] environment expectations are much much [13:42] different than this. Our users are [13:44] looking for like stable and fast [13:46] infrastructure. They want reliable [13:49] execution for predictable training [13:51] results and we do not want our [13:54] researchers to be spending time on [13:56] infrastructure debugging. They should be [13:58] focused on model building. [14:00] So, how did we meet these expectations? [14:02] That's something that Ankur will talk [14:03] about. [14:05] All right. Over to you, Ankur. [14:06] >> Thank you, Sachin. [14:08] Now to deliver performant, optimal and [14:12] reliable clusters, we need to do a lot [14:13] of testing. These tests could be [14:15] microbenchmarks to isolate [14:18] issues with network, compute or storage. [14:22] But these could be workload issues as [14:23] well, which could be a catchall for [14:25] different types of issues. And these [14:26] workload tests need to run at different [14:29] scales to identify problems at single [14:31] node level, a single rack level and also [14:34] multi-rack scale as well. [14:37] Now, especially for the [14:39] MOE style of workloads, we cannot ignore [14:41] the CPU. We need to develop [14:43] microbenchmarks to identify problems [14:45] with with CPUs as well. [14:49] Now, we are developing these tests to [14:52] not only minimize our training [14:54] interruptions, but also identify when [14:56] does the training and why does the [14:58] training slow down. [14:59] Remember, one slow GPU or slow rack can [15:03] bring down or slow down your training [15:05] across thousands or tens of thousands of [15:08] GPUs. [15:09] So, when we identify a single a [15:11] straggling node or a straggling rack, we [15:13] move to it to a poor performer [15:15] reservation to do further [15:18] testing to find the root cause of it. [15:20] And we use different methods for [15:23] detecting stragglers. These could be [15:25] like nickel all-to-all benchmarks for MU [15:27] seller workloads, or it this could be a [15:30] workload based testing, or it could be [15:33] some more simpler ad hoc analysis where [15:36] we say, "Hey, let's take intersection of [15:38] all the GPUs that were present in the [15:39] bad or the slow jobs and remove any [15:42] nodes that were not present [15:46] that were present in the good jobs, [15:47] right? So, just simple ad hoc analysis [15:50] will also give you [15:53] stragglers as well. [15:56] Now, when do we run these tests? What [15:58] test do we run? At what scale? How do we [16:00] develop a testing strategy when we are [16:03] working with so many teams with the AI [16:05] researchers themselves, with the [16:07] scheduling team, with the cloud service [16:08] provider, with other infrastructure [16:10] teams? So, we need to be really [16:14] adaptive in our testing strategy. And [16:16] practically speaking, it should be [16:19] adaptive on a day-to-day basis. So, if [16:21] your training run across thousands of [16:23] GPUs when well as expected without any [16:27] interruptions or slowdowns, then your [16:29] testing could be very very minimal or no [16:31] testing at all. [16:33] But, if you had a bad training day, then [16:35] you could decide, "Hey, I want to run [16:37] test in parallel at scale across [16:39] thousands of nodes, or I want to block [16:41] capacity and test for cross rack issues [16:44] as well." So, so we really need to [16:47] develop adaptive testing strategy. [16:52] How do we develop an adaptive testing [16:54] strategy? We can only get there with [16:55] automation. [16:57] We cannot have a testing strategy where [16:59] we are writing submission scripts every [17:02] time, and like a very few folks in the [17:05] teams are are responsible for for [17:07] writing those submission scripts. [17:09] We spent a lot of time in building [17:11] automation tools for our testing [17:13] strategy, which could work for a whole [17:16] bunch of different GPUs, had a whole [17:18] library of micro benchmark and workload [17:20] test, and it allowed us to submit these [17:23] tests with a single CLI run command, [17:26] where we can decide to run micro [17:28] benchmarks or workload tests across any [17:30] scale that that we wanted with a single [17:32] CLI command. And that way, we can be [17:35] really adaptive and really quick in in [17:38] getting results from our tests and [17:40] making decisions on top of it. [17:43] Okay, now putting everything together. [17:45] So, we worked very closely with the NeMo [17:46] Tron team to train their very large 120 [17:49] billion parameter model. Uh [17:51] and this training ran across tens of [17:54] thousands of GPUs, and we trained over [17:57] hundreds of billions of tokens per day. [17:59] Now, how did we get there? We got there [18:01] by making changes both on the software [18:04] stack and the hardware stack. On the [18:06] software side of things, we implemented [18:08] things like async checkpointing, garbage [18:10] collection to reduce training [18:12] interruptions, but also reduce [18:14] run-to-run variability as well. [18:16] And like I talked about, straggler [18:18] detection, automation in our testing [18:20] strategy, uh we developed those changes [18:22] as well. And we instrumented the [18:24] training code as well to identify [18:27] uh run time good put, which is basically [18:30] how productive is your is is your [18:33] training. [18:34] Uh as as opposed to the total wall clock [18:37] time. [18:38] Okay, next I will give it back to Sachin [18:41] to talk about the success of NeMo Tron 3 [18:44] and future work. [18:47] >> Thank you, Ankur, for walking us through [18:48] the testing and validation that led to [18:51] the successful release of NeMo Tron 3. [18:54] A quick word on NeMo Tron 3, it's a [18:56] hybrid MoE architecture model with a [18:59] long context window, but above all, it's [19:01] a fully open ecosystem. Developers and [19:04] researchers can download the open [19:05] weights, data sets, recipes, fine-tune [19:09] it, play with it. So, please give it a [19:10] try. [19:12] Uh I would like to talk a bit about our [19:15] future work that we are doing. Before [19:17] that, a quick look into some of the [19:19] growth trajectory that we discussed. [19:22] Model complexity is demanding a shift to [19:25] rack-scale computing like GB200 and [19:28] GB300. [19:29] If you look at, everything is [19:31] exponentially growing. Like some of [19:33] these frontier labs are now managing [19:35] nodes more than 50,000 in in of [19:39] clusters. [19:40] And not only that, uh as the job sizes [19:43] are increasing, as I spoke about, some [19:44] of our jobs are more than tens of [19:46] thousands of GPUs are running across [19:48] like, you know, thousands of nodes. Now [19:50] failures are bound to happen. [19:52] So what we are doing, like some of the [19:54] work to address some of these [19:56] challenges. One, as Ankur also spoke [19:58] about, we are moving away from one-time [20:00] testing. We are moving towards like [20:02] continuous testing and validation, not [20:04] just to bring up test, but continuous [20:07] test not just single node, but [20:10] multi-node, but and now we are moving to [20:12] like even multi-rack testing. [20:15] Uh adaptive efficiency. I spoke about [20:17] the Slurm simulator. We are leveraging [20:19] some of those simulation technologies to [20:21] dynamically fine-tune our clusters, so [20:24] that based on the workload and the [20:26] configurations, it can dynamically adapt [20:29] and utilize the GPU resources to the [20:31] maximum possible [20:33] uh uh [20:34] uh [20:35] way. And not only that, the we are [20:38] applying all of these [20:39] uh hard-learned lessons to like the [20:42] upcoming rack-scale systems and also [20:44] stabilizing systems like GB300. [20:48] I would like to wrap the call today uh [20:50] this talk today with few quick points. [20:53] Firstly, please experience the enormous [20:55] efficiency that can be gained by [20:58] applying NVIDIA GB300 rack-scale [21:00] computing. [21:02] As we noted, any NVIDIA is not just [21:05] enabling itself, but also the entire AI [21:08] industry by doing its research on the on [21:11] the [21:12] AI infrastructure and the open models. [21:15] And I would like to thank you all today [21:16] for listening to us. [21:18] Exciting future is ahead of us. Please [21:20] join us in our mission to build the next [21:22] frontier of intelligence. [21:24] Thank you.