NVIDIA Introduces Open-Source Cluster Readiness Engine to Validate Kubernetes GPU Clusters

NVIDIA Introduces Open-Source Cluster Readiness Engine to Validate Kubernetes GPU Clusters NVIDIA has released the NVIDIA Cluster Readiness Engine (NVCRE), an open-source Kubernetes controller designed to identify slow or failing GPU nodes by running distributed test workloads before production AI training jobs begin. GPU clusters can pass individual health checks yet fail or degrade during distributed workloads due to hidden hardware or network issues. NVCRE organizes testing through a layered Kubernetes CRD architecture (Certification, Workflow, Job) and evaluates performance via Common Expression Language (CEL) expressions. The tool includes adaptive fault isolation to automatically bisect failing node groups down to the specific problematic nodes. The standalone WorkloadRun CRD simplifies orchestrating multi-node GPU frameworks like PyTorch or MPI, optionally leveraging gang schedulers to avoid placement deadlocks. NVCRE acts as the active testing layer in the NVIDIA DSX OS stack, complementing AICR's configuration management and NVSentinel's passive telemetry monitoring. The GPU Cluster Readiness Challenge A multi-node GPU cluster can report healthy status across all GPUs and network links while still degrading or failing under distributed AI workloads. Standard health checks and smoke tests fail to mimic the inter-node stress of large training jobs, leaving teams to manually bisect clusters using custom scripts or discover degradation only after customer jobs fail. Unlike Slurm, Kubernetes lacks native single-command tooling to orchestrate distributed tests with matching NCCL parameters, RDMA configurations, and shared memory volumes. NVCRE fills this gap by actively executing distributed workloads on Kubernetes to pinpoint exact node failures before workloads land. Layered API and Pass Criteria NVCRE relies on Kubernetes Custom Resource Definitions arranged hierarchically. A Certification resource defines target nodes and test categories, initiating a Workflow per category, which then launches a child Job to execute the run and monitor node health. NVCRE provides a catalog covering five NCCL variants, DCGM level-4 diagnostics, and NeMo pretraining benchmarks with Nemotron 5 models (8B and 56B). Test pass/fail thresholds are configured via Common Expression Language (CEL) evaluated against runtime measurements; missing a performance target flags a ValidationFailed condition even if the workload completed without a crash. Test Scales and Adaptive Fault Isolation Validation can run at varying scales: intra-node, intra-rack using GPU clique topology labels, full-scale, or diagnose mode. Because multi-node failures often mask the single problematic node across an entire group, setting the scale to diagnose triggers adaptive fault isolation. The engine performs topology-aware hierarchical group testing by recursively splitting failing node groups and rerunning them down to a minimum group size, isolating suspect nodes while limiting concurrent runs to avoid fabric saturation. The WorkloadRun API and Ecosystem Integration The standalone WorkloadRun CRD simplifies running arbitrary multi-node GPU tasks on Kubernetes across torch, mpi, or exec frameworks. It automatically sets up Kubeflow TrainingRuntime, shared memory, and network environments, while supporting gang-aware schedulers like KAI Scheduler to prevent rendezvous deadlocks. Within the broader NVIDIA DSX OS stack, NVCRE serves as active workload validation, pairing with NVIDIA AI Cluster Runtime for baseline configuration management and NVSentinel for passive, continuous telemetry monitoring and remediation. Prerequisites and Availability NVCRE is open-sourced under the Apache 2.0 license. Running the engine requires Kubernetes 1.29 or newer, kubectl, Helm 3.x, and the NVIDIA GPU Operator. Specialized catalogs such as NVIDIA GB200 NVL72 and GB300 NVL72 also require the NVIDIA DRA Driver for GPUs, while DCGM level-4 testing requires the standalone DCGM service. Installation and execution can be managed using the nvcrectl CLI.

英伟达开源 NVCRE：在生产负载上线前自动化验证 GPU 集群就绪性 英伟达推出开源 Kubernetes 控制器 NVIDIA Cluster Readiness Engine（NVCRE），通过在拓扑感知的节点组上运行实际分布式负载，主动暴露硬件与网络性能缺陷，精确锁定故障节点，避免大规模训练任务上线后因隐藏隐患导致降级或停滞。 传统健康检查无法暴露负载下的性能退化，NVCRE 通过主动运行真实分布式工作负载来验证集群 提供 Certification、Workflow、Job 三层 API 体系，并使用 CEL 表达式根据实测指标判断是否通过 具备自适应故障隔离功能，可通过层次化分组二分测试在多节点异常中精准锁定故障节点 提供独立的 WorkloadRun CRD 抽象框架，支持 torch、mpi、exec 运行时并集成 Gang 调度 集群就绪性验证与分层 API GPU 集群在通过基础诊断后仍可能在分布式任务中因单卡降级或网络变慢而受损。NVCRE 采用分层 CRD 设计：Certification 声明测试节点与分类；Workflow 管理单项测试并处理平台配置覆盖；Job 执行实际负载并记录节点状态与度量。内置目录涵盖五种 NCCL 通信测试、DCGM 4 级诊断及 NeMo 预训练基准。指标判定采用通用表达式语言（CEL），即使任务运行结束，未达到带宽或算力阈值依然判定为验证失败。 多尺度测试与自适应故障隔离 NVCRE 支持单节点、机架内（基于 NVLink 拓扑）和全规模等多尺度测试。针对 64 节点 All-Reduce 等任务中难以归因的群体性能低下问题，其提供自适应故障隔离模式（testScale: diagnose）。该模式运用拓扑感知的分层组测试策略，将故障组自动拆半复测，直至缩小到设定的最小组规模，结合并发限制避免网络饱和，快速将排查范围收敛到极少数嫌疑节点。 WorkloadRun 运行时与生态协同 WorkloadRun CRD 可独立运行，支持 Torch、MPI 及通用 Exec 执行，自动生成 Kubeflow TrainingRuntime 并配置共享内存与网络变量；同时支持配置 KAI Scheduler 等 Gang 调度器，防止节点部分就绪引发死锁。在 NVIDIA DSX OS 架构中，AICR 负责静态配置漂移防护，NVCRE 作为主动负载层捕获无遥测故障，NVSentinel 则被动监控运行指标并基于 NVCRE 暴露的故障执行节点隔离与排空。

A GPU cluster can pass every health check and still fail to run an AI workload. Even when every GPU, network link, and pod reports healthy, a 512-GPU training job can underperform or fail. The cause may be one slow GPU, a link that degrades under load, or a configuration that quietly routes traffic over a slower path. Operators may not discover the problem until hours into the run or until a customer files a ticket. Teams can then spend days bisecting the cluster to find the root cause while the capacity sits idle. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that narrows the search to the specific nodes involved before production workloads land. It runs real distributed workloads across topology-aware node groups, measures the results, and reports which nodes failed each test. Operators no longer need to write NVIDIA Collective Communications Library (NCCL) manifests by hand, bisect racks manually, or learn about degraded hardware from customer tickets. Readiness becomes a proven property of the cluster rather than an assumption. What does proving readiness require? A GPU cluster becomes ready in stages. It moves through bring-up, burn-in, preproduction, and production, with each stage setting a different bar. A node that passes a smoke test is not necessarily ready to join a 512-GPU training run. Platform teams often encode that progression in a runbook, spreadsheet, or shell scripts wrapped around NCCL tests. It becomes another system they must build and maintain alongside node configuration, GPU sharing, and workload orchestration. A cluster can pass standard diagnostics and still fail under a real distributed job, so the best way to test readiness is to run a workload. On Slurm, that requires a single srun command. Kubernetes has no built-in equivalent, so the same test requires GPU and remote direct memory access (RDMA) resource requests, NCCL settings matched to the network fabric, a large enough shared-memory volume, and a way to ensure that all pods start together. NVCRE fills these gaps on Kubernetes. It runs workloads that expose real hardware problems and names exactly which node caused each failure. communication/nccl-all-reduce Status: Failed Runtime: 42m 18s Scale: full-scale Nodes/Job: 8 Failed Nodes: gpu-node-07 ThresholdViolation gpu-node-12 HardwareFailureDetected Summary Categories: 1/2 passed Failed Nodes: 2 Result: FAILED How NVCRE works The API is the product surface. Custom resource definitions (CRDs) define each resource, so you can inspect it with kubectl and manage it through GitOps workflows. A layered API The API has three resources arranged in a hierarchy. Certification : The resource you create. It names the nodes to test and the categories to run. Workflow : Manages one category. It applies catalog, platform, and GPU overrides; manages iteration count; sets the orchestration target; and creates the child job. Job : Runs the workload for the target node group, monitors node health, and records measurements and failures. A certification creates one workflow per category, and each workflow creates its child job. Results then propagate upward. The job records which nodes failed and why, the workflow reports the test result, and the certification groups results by category. That hierarchy attributes each failure to a specific node and category. For example, a run reports that gpu -01 hit a hardware fault during NCCL and that gpu -02 missed its bandwidth target. Validating a cluster The following example shows how a certification names its targets and the categories to run. apiVersion: nvcre.nvidia.com/v1alpha1 kind: Certification metadata: name: gpu-cluster-cert spec: target: nodeSelector: nvidia.com/gpu.present: "true" categories: - domain: communication variant: nccl-all-reduce - domain: training variant: nemotron5-8b $ kubectl apply -f certification.yaml $ kubectl get certifications.nvcre.nvidia.com -w The built-in catalog currently covers three domains: five NCCL communication variants (all-reduce, all-gather, all-to-all, loopback, and loopback across NVIDIA NVSwitch), the NVIDIA Data Center GPU Manager (DCGM) level-4 diagnostic suite, and NVIDIA NeMo pretraining with NVIDIA Nemotron 5 models at 8B and 56B parameters. Each entry includes platform-aware defaults. NVCRE detects the GPU architecture and cloud platform from the target nodes and derives the rest of the configuration, including GPUs per node, the NCCL environment, and platform-specific networking. Pass criteria as expressions Pass and fail criteria use Common Expression Language (CEL) and are evaluated against measured metrics. No thresholds ship by default. The values below are illustrative examples for NVIDIA GB200 NVL72-class systems. categories: - domain: communication variant: nccl-all-reduce options: thresholds: busBandwidthGBps: "value >= 900" - domain: training variant: nemotron5-8b options: thresholds: goodputRatio: "value >= 0.9" avgTFLOPsPerGPU: "value >= 800" When a measured metric misses its target, NVCRE sets a ValidationFailed condition, recorded separately from whether the run itself succeeded. A workload that finishes but misses its target is still reported as a failure. Testing at the scale where failures appear Some failures are visible only at a particular scale, so the grouping strategy is explicit. The testScale field selects the strategy. Intra-node tests each node independently. Intra-rack partitions nodes by topology domain using the nvidia.com/gpu.clique label. Full-scale puts every node in a single group. Diagnose runs adaptive fault isolation (covered next). The strategy you select dictates what gets measured. An NCCL test inside one NVIDIA NVLink domain measures NVLink bandwidth, while the same test across three racks measures the scale-out fabric. The two measure different things. Adaptive fault isolation The hardest case in multi-node validation is a failure that cannot be attributed to any single node. A 64-node all-reduce returns low bandwidth, and every node in the group is equally implicated. Isolating the cause by hand can take days of engineering time. NVCRE automates that isolation. Setting testScale: diagnose runs topology-aware hierarchical group testing. The engine splits each failing group, reruns the halves, and continues until it reaches minGroupSize . Groups that still fail at that size are flagged as suspects. maxConcurrent limits the number of jobs that run concurrently so they do not saturate the fabric being measured. The output names a small number of suspect nodes instead of implicating the entire group and gives the reason each node failed. Running any workload: the WorkloadRun API Running a multi-node GPU workload on Kubernetes requires platform detection, framework-specific runtime configuration, GPU and network resource requests, and cleanup after a failed run. That setup is repetitive and error-prone. WorkloadRun handles the setup: provide a container image, select a framework, and specify the number of nodes. apiVersion: nvcre.nvidia.com/v1alpha1 kind: WorkloadRun metadata: name: nccl-all-reduce spec: image: nvcr.io/nvidia/pytorch:26.01-py3 framework: mpi: binary: /usr/local/bin/all_reduce_perf_mpi args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"] mpirunPath: /usr/local/mpi/bin/mpirun numNodes: 4 bandwidthMeasurement: logProfileRef: nccl-bandwidth testType: all_reduce The framework field supports exactly one of torch (distributed training through torchrun ), mpi (NCCL tests and other MPI workloads), or exec (an arbitrary command). NVCRE generates the matching Kubeflow TrainingRuntime , injects the shared-memory volume, sets the NCCL and platform environment variables, and enables NVIDIA NVLink scale-up networking where the hardware supports it. Without a gang scheduler, the default Kubernetes scheduler places pods independently, and ranks wait for their peers at the framework rendezvous. On a busy cluster, that can deadlock: partially placed pods hold GPUs while waiting for peers that never arrive. Setting spec.gangScheduler opts every workload pod into a gang-aware scheduler, such as KAI Scheduler, which holds all pods until the entire gang can be placed at once. Because WorkloadRun is a plain CRD, external tools can use it to run workloads without adopting the rest of the NVCRE. NVCRE provides the execution path, and the calling tool supplies the test. Configure, validate, and monitor AI clusters at scale A cluster must answer three questions on the way to production: Is it configured correctly? Is it ready to run a real AI workload? Is it healthy right now? A different layer of NVIDIA DSX OS addresses each question. NVIDIA AI Cluster Runtime (AICR) establishes and maintains a validated cluster configuration. AICR captures validated combinations of drivers, operators, kernels, and system settings as version-locked recipes. Teams can reproduce the same optimized configuration across clusters, validate live state, and detect drift. This reduces performance variation and avoids days of tuning while costly GPUs sit idle. NVCRE verifies that a cluster is ready to run real AI workloads. This active, workload-driven layer generates load, so it can find failures that produce no telemetry. A single degraded GPU slows a synchronous training job to the speed of its worst rank, and an NCCL bandwidth test can identify the problem in minutes. NVIDIA NVSentinel continuously monitors cluster health. As the passive, telemetry-driven layer, it watches signals the cluster already produces, including DCGM metrics, Xid errors, system logs, and cloud provider maintenance events. It detects runtime faults and can drive quarantine, drain, and remediation workflows. Because it consumes no GPU time, it can run continuously in production, where active testing would take GPUs away from workloads. Each project delivers value independently and integrates with the others for teams running the full stack. Figure 1. NVSentinel passively receives cluster telemetry, while NVCRE actively generates load to find failures that produce no telemetry NVCRE records failed nodes and reasons. It does not cordon, taint, or patch node conditions, avoiding conflicting actions and desynchronized cleanup. The NVSentinel NVCRE Certification Monitor can translate failed certification results into health events. Configured NVSentinel policies can then quarantine and drain nodes or trigger external remediation. A later successful certification can clear the failure signal and release the taint. Get started NVCRE requires Kubernetes 1.29 or later, kubectl , Helm 3.x, and NVIDIA GPU Operator on the target cluster. NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 catalog entries also require the NVIDIA DRA Driver for GPUs because those entries create ComputeDomain resources. The DCGM level-4 category requires the standalone DCGM service. A gang-aware scheduler such as KAI Scheduler is optional but recommended on busy, shared clusters. Install the CLI and set up the cluster. The nvcrectl setup init command installs the CRDs, controller, Kubeflow Trainer, and default log profiles. $ curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash $ nvcrectl setup init Then run an end-to-end validation. $ nvcrectl certification run --cert-file certification.yaml --wait $ nvcrectl certification report gpu-cluster-cert -n <namespace> The following report lists each category’s status, runtime, measured bandwidth, and any failed nodes, with the reason for each failure. Certification Report Name: gpu-cluster-cert Platform: aws GPU: gb300 Nodes: 16 communication/nccl-all-reduce Status: Succeeded Runtime: 3m 56s Scale: full-scale Nodes/Job: 16 MNNVL: Enabled Bandwidth: Size AlgBW BusBW Samples 16 GB 473.44 GB/s 932.09 GB/s 9 Summary Categories: 1/1 passed Failed Nodes: none Result: PASSED Get involved NVCRE is licensed under Apache 2.0 and developed in the open. Report a bug, request a feature, or propose a change before sending a pull request on GitHub . You can also contribute catalog entries, workload adapters, tests, and documentation. Use the engine to validate GPU clusters before production. Its shared workflow and test catalog run consistently across Kubernetes clusters, with roadmap support for new NVIDIA architectures, inference, and automated lifecycle validation. GitHub repository Documentation Validate your GPU cluster before your next production workload lands. The project is part of NVIDIA DSX OS , the operating layer of the NVIDIA DSX AI Factory Platform .