NVIDIA Details Infrastructure Diagnostic Fixes to Close Cloud AI Performance Gaps

NVIDIA Details Infrastructure Diagnostic Fixes to Close Cloud AI Performance Gaps AI clusters with identical hardware often trail NVIDIA reference architectures by 8% to 12% due to compounding configuration issues across the kernel, hypervisor, CPU, container runtimes, and collective communication libraries. Enabling Arm SMMU Virtual Command Queue support in the host and guest hypervisor recovered a 12% throughput gap on virtualized GB200 NVL72 systems running DeepSeek-V3 MoE. Correcting BIOS C-states to allow C6 sleep and enforcing NUMA cpuset isolation recovered 9% of lost performance on an H100 training cluster. Increasing NCCL queue pairs per connection from 1 to 4 improved collective communication efficiency by up to 31% on GB300 NVL72 clusters with ConnectX-8 fabrics. Bind-mounting missing NCCL topology files directly into workload containers resolved silent auto-detection fallbacks that caused performance drops between 13% and 53%. Arm SMMU Virtualization and Invalidation Bottlenecks In virtualized GB200 NVL72 environments running FP8 pre-training for DeepSeek-V3 MoE, CPU overhead in small-kernel regions caused iteration times to lag bare metal by 12% to 14%. Profiling revealed that 24% of host CPU cycles were consumed by arm_smmu_cmdq_issue_cmdlist due to memory mapping invalidations trapping to the host and serializing through a single queue. Enabling the Command Queue Virtualization extension (CMDQV/VCMDQ) via the host tegra241-cmdqv driver and exposing it through hypervisor attributes allowed the guest to issue invalidations directly to hardware, bringing performance within reference tolerance. CPU Frequency C-States and NUMA Misplacement An H100 SXM5 cluster trailed reference performance by 12% on Llama 3 70B workloads due to configuration issues in user space and the BIOS. Cores were locked at 3.0 GHz instead of their 3.8 GHz turbo rating because idle cores were restricted to the C1 state, consuming package power headroom. Additionally, hypervisor background threads shared physical cores with data loader workers, and 18% of memory traffic crossed to remote NUMA nodes. Enabling C6 idle states regained roughly 4% in turbo headroom, while isolating hypervisor threads and training processes via cpuset recovered another 5%. Fabric Collective Tuning on ConnectX-8 SuperNICs A GB300 NVL72 cluster with 1.6 Tbps ConnectX-8 SuperNICs showed a 31% gap during 512-GPU Nemotron-4 15B training due to communication exposure during AllGather and ReduceScatter phases. Profiling demonstrated that adjusting NCCL_IB_QPS_PER_CONNECTION from the default of 1 to 4 reduced iteration times from 1.09 seconds to 0.83 seconds by improving large-message collective throughput. However, the setting is fabric- and message-dependent, meaning tuning should be verified against target message sizes to avoid unnecessary CPU overhead. Container Topology Visibility and Silent Failures On a virtualized B200 deployment, containerized training ran 13% to 53% slower than reference baselines even though host-level collective tests passed. The NCCL_TOPO_FILE environment variable and its underlying XML file were configured on the VM host but were neither mounted nor passed into the enroot container environment. NCCL silently defaulted to auto-detection, causing AllGather and ReduceScatter routines to run 2 to 4 times slower. Bind-mounting the XML topology file and confirming its visibility from inside the job container resolved the issue. Preflight Diagnostic Strategies Performance gaps typically accumulate from multiple small discrepancies rather than a single component failure. Recommended preflight checks before full-scale training include validating hardware health via DCGM, verifying virtualization and page-size settings, tuning CPU power governors and affinity, verifying topology visibility within job containers, and benchmarking collective communication with nccl-tests.

英伟达解析AI基础设施性能落差排查：四个调优实战案例 相同硬件规格的AI计算集群在实际训练中常出现8%至12%的吞吐量落差。英伟达指出，此类差距往往由内核、虚拟化、CPU功耗与网络配置等多层隐性问题累加所致，并分享了四个涵盖SMMU、NUMA、NCCL与容器配置的实战调优案例。 即便采用相同的H100、GB200或GB300系统，合作伙伴集群因软硬件配置层层损耗，常无法达到英伟达Exemplar Cloud要求的95%性能阈值。 GB200虚拟机训练MoE模型落后12%，原因为缺少Arm SMMU的CMDQV/VCMDQ虚拟化支持导致命令队列自旋锁竞争，启用该功能后恢复基准水平。 H100集群因BIOS限制C-state至C1及NUMA远程访问过高损失12%性能，通过允许空闲核降至C6和cpuset隔离追回9%差距。 GB300 NVL72在1.6 Tbps网络上利用率不足，将NCCL_IB_QPS_PER_CONNECTION从1调至4后，集合通信性能提升并追回31%迭代时间。 虚拟化与SMMU：解除Arm指令队列瓶颈 在GB200 NVL72虚拟机环境下进行DeepSeek-V3 MoE FP8预训练时，小内核频繁调用导致迭代时间落后裸机12%至14%。perf性能分析显示主机在arm_smmu_cmdq_issue_cmdlist上消耗了24%的CPU周期。通过在主机内核启用CMDQV/VCMDQ并暴露给客户机，允许客户机直接向硬件发送SMMU失效命令而无需退出虚拟机，成功将迭代时间差距拉回容限之内。 CPU电源策略与NUMA绑核冲突 在H100集群的Llama 3 70B训练中，BIOS将C-states限制在C1以维持“低延迟”，导致空闲核心持续消耗功耗预算，使忙碌核心无法提升至3.8 GHz睿频；同时虚拟化管理线程与训练数据加载线程竞争同一物理核心，且存在约18%的跨NUMA内存访问。将空闲核心允许降入C6以释放功耗空间，配合cpuset进行进程隔离与NUMA绑定后，找回了9%的性能损失。 高带宽网络并发与容器运行拓扑配置 针对搭载ConnectX-8 SuperNIC（1.6 Tbps）的GB300集群，将NCCL队列对并发参数NCCL_IB_QPS_PER_CONNECTION由默认的1调整至4，大幅加速了大报文的AllGather与ReduceScatter，缩短了31%的迭代时间。而在虚拟化B200部署中，由于enroot容器未正确挂载NCCL_TOPO_FILE与topo.xml，导致NCCL静默回退至自动探测，引发多达53%的落差；排查需在作业容器内部执行验证以杜绝此类静默失效。

Two AI computing clusters built from identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems can deliver materially different training throughput. We routinely see 8% to 12% gaps between partner deployments and the corresponding NVIDIA reference architecture (RA) on the same workload, same model, same global batch size. The cause is often a stack of configuration choices in the kernel, hypervisor, BIOS, and NVIDIA Collective Communications Library (NCCL) settings, each costing a few percent, that compound into a gap large enough to miss the 95% threshold required for [**NVIDIA Exemplar Cloud**](https://www.nvidia.com/en-us/data-center/ai-cloud-performance/) validation. This post walks through four debugging investigations from real partner clusters. Each diagnostic isolates a distinct layer of the stack: system memory management unit (SMMU) and page-table behavior on [NVIDIA Grace CPU](https://www.nvidia.com/en-us/data-center/grace-cpu/); power management and non-uniform memory access (NUMA) placement on x86-based CPU; [NVIDIA NCCL](https://developer.nvidia.com/nccl) queue-pair concurrency on 1.6 Tbps fabrics; and silent hardware-installation defects. The post also shows the specific signal in `perf`, [NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems), or NVIDIA [NCCL tests](https://github.com/nvidia/nccl-tests) that pointed to the root cause, alongside the tuning change that closed the gap. Infrastructure engineers and performance architects who already run these benchmarks can benefit from these diagnostic patterns we use internally to run against their own clusters before formal RA validation. ### **Prerequisites** To reproduce the diagnostics in this post, you will need: - An NVIDIA HGX H100, HGX H200, HGX B200, GB200 NVL72, or GB300 NVL72 systems cluster with [NVIDIA Quantum InfiniBand](https://www.nvidia.com/en-us/networking/products/infiniband/) or RoCE interconnect. - A distributed training workload with stable iteration timing—[NVIDIA NeMo](https://github.com/nvidia-nemo) on Llama 3 model, [NVIDIA Nemotron](https://developer.nvidia.com/topics/ai/nemotron), or DeepSeek configuration is a reasonable reference. - Root access on at least one node for `perf`, BIOS/UEFI changes, and kernel parameter changes. - `nccl-tests` built against the same NCCL version your training stack uses, NVIDIA Nsight Systems, and Linux perf with kernel symbols available. ## Common patterns behind training performance gaps Recent Exemplar training engagements show that performance gaps rarely come from a single obvious failure. More often, they come from configuration details that become visible only under workload pressure. Some recurring patterns include: 1. **Grace and virtualization readiness:** Missing platform capabilities, SMMU overhead, IOMMU behavior, or page-size settings that don’t match the expected configuration. 2. **CPU power and process placement:** Cores running below expected turbo frequency, ranks or helper threads placed on the wrong cores, or NUMA/PCT bindings that don’t match the platform topology. 3. **Runtime topology:** Host topology files or NCCL settings that are correct on the node but missing inside the workload container or launcher environment. 4. **Fabric and collective behavior:** NCCL settings that don’t match the target fabric, message size, or scale of the training workload. 5. **Application-to-platform binding:** Training processes binding by core ID or rank order instead of topology-aware affinity. These aren’t the only causes of training performance gaps, and checking them doesn’t replace validation with real applications. Four case studies below show how these patterns appeared in recent training work: what signal exposed the issue, what changed, and how the fix was verified. The order isn’t a universal triage sequence; the right starting point depends on the workload, platform, and first profiler signal. ## Case study 1: NVIDIA GB200 NVL72 FP8 pre-training, 12% slower in a virtual machine (VM) than on bare metal **Layer: Virtualization and SMMU** A GB200 NVL72 partner deployment running DeepSeek-V3 Mixture-of-Experts (MoE) FP8 pre-training inside a VM was producing iteration times 12% to 14% longer than the bare-metal RA. Pre-training recipes for dense models like Llama 3 70B ran within 3% of RA performance, while DeepSeek-V3 MoE, which issues many small kernels per iteration, was the outlier. Nsight Systems traces captured on the partner cluster showed significantly higher CPU overhead for tiny kernel regions of the workload. Microbenchmarks targeting just the CPU single thread performance demonstrated near identical performance on the partner and RA cluster nodes. This indicated that a 30-second `perf record -a -g` capture on the host, viewed with `perf report`, surfaced an unexpected top frame: 24% of CPU cycles spent on `arm_smmu_cmdq_issue_cmdlist`. `arm_smmu_cmdq_issue_cmdlist` is the function that submits invalidation commands to the Arm SMMU’s command queue. Under virtualization, every map/unmap resulting in guest invalidation traps the host and serializes through a single command queue, producing the spinlock contention visible in the profile. Virtual Command Queue (VCMDQ) is a feature available through the Command Queue Virtualization extension to the standard Arm SMMUv3 and allows the guest to issue SMMU invalidation commands directly to hardware without VM exits. **The fix:** Enable CMDQV/VCMDQ in the host kernel on the partner cluster and expose it to the guest. This requires a kernel built with the `tegra241-cmdqv` driver and the corresponding hypervisor support; recent QEMU/libvirt versions have added a `cmdqv` IOMMU attribute to expose it to guests. After this change, `linux perf` showed `arm_smmu_cmdq_issue_cmdlist` falling out of the top frames and dTLB miss rates returning to bare-metal parity. The MoE iteration-time gap narrowed to within RA tolerance from 12%. The takeaway is that Grace-based virtualized deployments need the VM stack to expose the right SMMU capabilities for memory-mapping-heavy workloads. With CMDQV/VCMDQ enabled in the host kernel and exposed to the guest, the platform can avoid unnecessary SMMU serialization and return MoE training performance to within RA tolerance. The next layer down is the CPU itself, where the failure mode looks completely different. ## Case study 2: H100 cluster losing 12% to CPU contention and NUMA misbinding **Layer: CPU power and process placement.** A partner’s H100 SXM5 cluster, running the same NCCL version and NeMo container as NVIDIA’s HGX RA, was running Llama 3 70B pre-training 12% slower than reference. Unlike the GB200 NVL72 case, this wasn’t a kernel-level issue; everything happened in user space and BIOS. **Two things stood out:** 1. **CPU frequency:** `turbostat -i 1` during training showed busy cores pegged at 3.0 GHz, despite the SKU being rated for 3.8 GHz turbo. Idle cores were also at 3.0 GHz, with C-states sitting in C1 rather than dropping to C6. 2. **NUMA-remote traffic:** `numastat -p <python_pid>` showed roughly 18% of the training process’s memory accesses going to the remote NUMA node **Root cause:** - The CPU on the partner cluster was configured with C-states limited to C1 in BIOS. This is a common “low-latency” default that is actively wrong for AI training workloads. With idle cores held in C1, they continued to draw package power; the busy cores feeding the GPU with kernels couldn’t claim enough of the package power budget to hit turbo. Allowing the idle cores to drop to C6 freed power headroom, enabling the busy cores to climb to 3.8 GHz and recover roughly 4% on this workload. - The hypervisor housekeeping threads were pinned to the same physical cores as the training process’s data loader workers. Inside the VM this looked like sporadic 50–100 ms stalls in the python threads, which then propagated as the long tail in step time. The fix was a `cpuset` separation: hypervisor and host services on cores 0–7 and 56–63, training processes on the remainder. **Result:** The 12% gap shrank to 3%, with the residual traced to a different NCCL tuning issue covered in the next case study. The pattern here is that no single fix recovered the whole gap. The C-state change was the largest single contributor at \~4%, and the rest came from process isolation through NUMA binding. With CPU and virtualization addressed, the next ceiling is the network. ## Case study 3: GB300 NVL72 with NVIDIA ConnectX-8 SuperNIC under-utilizing 1.6 Tbps fabric **Focus: ConnectX-8 SuperNIC collective tuning** A GB300 NVL72 deployment with [NVIDIA ConnectX-8 SuperNICs](https://www.nvidia.com/en-us/networking/infiniband-adapters/) (1.6 Tbps per node) showed a 31% training performance gap on Nemotron-4 15B pre-training. Single-node throughput looked healthy; the gap appeared at 512 GPUs, where the profiler showed exposed AllGather and ReduceScatter time. That pointed to the collective path on the ConnectX-8 fabric rather than compute. The investigation tested several variables with NCCL Tests (`nccl-tests`), including iteration count, UCX/UCC behavior, NUMA mapping, NVLS, and NCCL versions. For the workload’s networking performance, the relevant tuning change was narrower: increasing `NCCL_IB_QPS_PER_CONNECTION` to 4 from the default value of 1. Nsight Systems trace showing communication overhead exposed at lower QPS values, contributing to longer training iteration time Signal was visible in both the workload and the `nccl-tests` collective measurements. On the NVIDIA reference cluster, the default configuration ran at about 1.09s per iteration. With `QPS=4`, the same reference workload improved to about 0.83s. In the profile, AllGather time dropped from about 375ms to 262ms, and ReduceScatter dropped from about 389ms to 273ms. The comparison run was about 0.76s and used a different NCCL version. The remaining difference was therefore partly attributable to an NCCL version mismatch between the comparison and reference environments; aligning the versions narrowed the residual gap further. Because NCCL version changes are outside the normal Exemplar tuning scope, the recommended tuning keeps the deployed NCCL version unchanged. **Lesson**: Don’t increase QPS everywhere. QPS is fabric- and workload-dependent. On this GB300 ConnectX-8 workload, QPS=4 improved large-message AllGather and ReduceScatter behavior. On other fabrics or message-size profiles, the same setting may add CPU overhead without improving training throughput. The right approach is to test the collective at the workload’s real message sizes, sweep the setting on the target fabric, and verify the result in the training workload. ## Case study 4: The environment variable that never made it inside In a virtualized B200 deployment, training throughput was 13%–53% below the NVIDIA reference even though `nccl-tests` run on the host showed expected performance. Inside the `enroot` workload container, AllGather and ReduceScatter were 2–4× slower, shifting the investigation from fabric health to a direct comparison of the NCCL topology configuration visible on the VM and inside the training job. ``` Host (VM) Container (enroot) ───────── ───────── NCCL_TOPO_FILE=/etc/nccl/topo.xml →NCCL_TOPO_FILE (not propagated) /etc/nccl/topo.xml present →/etc/nccl/topo.xml(not mounted) ↓ NCCL falls back to auto-detection → 13–53% below reference ``` | **Platform** | B200, virtualized stack | | **Symptom** | 13–53% below reference; AllGather/ReduceScatter 2–4x slower; NCCL tests on host pass fine | | **Root cause** | `NCCL_TOPO_FILE` set on the VM but neither the variable nor the topology file was mounted into the enroot container | | **Fix** | `--mount type=bind,source=/etc/nccl/topo.xml,target=/etc/nccl/topo.xml` | *Table 1. Diagnosis and remediation of missing NCCL topology configuration inside a virtualized B200 workload container* **Lesson:** Run checks from inside the same container, launcher, and Slurm allocation that will run the benchmark not from the host. Running `echo $NCCL_TOPO_FILE && cat $NCCL_TOPO_FILE` inside the job container is the fastest sanity check. If the path doesn’t resolve, NCCL fails silently with no error making this one of the harder gaps to diagnose without knowing where to look. **Summary of fixes** | **Case** | **Platform** | **Layer** | **Diagnostic signal** | **Fix** | **Recovered** | |---|---|---|---|---|---| | 1 | GB200 NVL72 (VM) | SMMU | arm\_smmu\_cmdq\_issue\_cmdlist dominant in perf; multi-fold dTLB miss increase | Enable VMDQV | \~12% | | 2 | H100 (VM) | CPU + NUMA | Cores stuck at 3.0 GHz; bimodal step time; 18% NUMA-remote | C-state tuning, cpuset isolation, numactl binding, SMT/mitigations off | 9% (12-3) | | 3 | GB300 NVL72 | NCCL concurrency | AllGather busbw at \~28 GB/s vs \~61 GB/s with QPS=4 | NCCL\_IB\_QPS\_PER\_CONNECTION update from 1 to 4 for CX8 | 31% iter time | | 4 | B200 (VM) | Runtime-visible topology | Host NCCL topology looked correct, but inside the enroot container NCCL\_TOPO\_FILE was not propagated and /etc/nccl/topo.xml was not mounted; AllGather/ReduceScatter were 2-4x slower | Bind-mount the topology file into the container and verify NCCL\_TOPO\_FILE from inside the job container | Closed 13-53% reference gaps | *Table 2. Diagnostic signals, corrective actions, and recovered performance across the four Exemplar Cloud case studies* ## Preflight checks before full-scale training debug When a cluster underperforms relative to its NVIDIA reference architecture specifications, these checks help rule out common platform issues before full-scale workload tuning. | **Area** | **What to check** | **Useful tools** | | **GPU and hardware health** | Clock, power, thermal, and NVLink bandwidth consistency under sustained load | nvidia-smi, DCGM, dcgm-exporter | | **Grace and VM readiness** | CMDQV support, guest page size, IOMMU passthrough behavior, and large-page availability | perf, dmesg, kernel config, boot parameters | | **CPU power and placement** | Busy-core turbo, cpuset isolation, and NUMA / PCT binding near GPUs | turbostat, lscpu, numactl, nvidia-smi topo -m | | **Runtime topology** | Topology files, NCCL environment variables, and HCA visibility inside the job container | env, cat $NCCL\_TOPO\_FILE, NCCL\_DEBUG=INFO | | **Fabric collectives** | AllGather and ReduceScatter behavior at workload message sizes | nccl-tests, workload traces | | **Workload tuning** | Pipeline parallelism, microbatch sizing, and communication overlap — only after platform issues are ruled out | Nsight Systems, workload logs | *Table 3. Recommended preflight checks and diagnostic tools for evaluating GPU health, VM readiness, CPU placement, runtime topology, fabric collectives, and workload configuration before full-scale training debugging* ## Debug early, debug less  Performance gaps between a cloud training deployment and the corresponding NVIDIA reference architecture are often cumulative with a few percent from CPU power settings, another from NUMA or PCT binding, more from a missing kernel capability, container-visible topology, or fabric configuration. These issues are worth checking before validation because they can turn into expensive full-scale debug sessions. At the same time, preflight diagnostics don’t guarantee an Exemplar Cloud pass. Some issues only appear in the validation workloads themselves, under the exact model, precision, topology, container, launcher, and network conditions used for the run. The practical goal is to remove known platform risks early, then use the training workload traces to debug the gaps that only appear at scale. - Go deeper **GTC session on demand:** [S81845 — *Drive Optimal Tokens per Watt on any AI Infrastructure Using Benchmarking Recipes*](https://www.nvidia.com/gtc/session-catalog/sessions/gtc26-s81845/). - [NVIDIA Grace CPU Performance Tuning Guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/index.html) - [NCCL Environment Variables](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html) — particularly NCCL\_IB\_QPS\_PER\_CONNECTION, NCCL\_IB\_SPLIT\_DATA\_ON\_QPS, NCCL\_IB\_HCA, and NCCL\_DEBUG - [NVIDIA NeMo Framework](https://www.nvidia.com/en-us/ai-data-science/products/nemo/) — pull updated containers from [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo) for the latest optimizations on Llama 3, DeepSeek, and Nemotron - [NVIDIA Performance Explorer](https://docs.nvidia.com/dgx-cloud/benchmarking/performance-explorer-user-guide/latest/index.html) — TCO Calculator for evaluating infrastructure on perf-per-TCO - [NVIDIA Performance Benchmarking Recipes](https://github.com/NVIDIA/dgxc-benchmarking) — ready-to-use templates for evaluating performance of AI use cases across hardware and software combinations