Full-Stack Observability Framework for NVIDIA AI Factories

Full-Stack Observability Framework for NVIDIA AI Factories AI infrastructure is susceptible to cascading and gray failures where degraded performance at one layer, such as network bit error drift, can stall entire synchronous training workloads. To mitigate multi-hour compute waste without causing alert fatigue, teams must implement a structured observability framework that connects platform, GPU, fabric, and cluster telemetry into an actionable triage pipeline. AI training workloads rely on synchronous parallel models where single-rank link degradations cause cascading job-wide throughput drops. Observability coverage must systematically span platform health, GPU metrics, fabric integrity, cluster scheduling, and inference services. A minimal toolset mapped to components reduces metric clutter and avoids 'watermelon metrics' where dashboards mask underlying service failures. Telemetry should funnel into a two-layer triage approach featuring a unified Grafana top-k dashboard backed by deep-dive vendor interfaces. AI Factory Failure Domains and Gray Failures AI infrastructure spans compute, networking, storage, orchestration, and application layers. Because distributed training jobs use the bulk synchronous parallel model, a degradation in one component, such as an InfiniBand link experiencing an elevated bit error rate, stalls synchronous collective operations like NCCL all-reduce. In these gray failures, hardware is degraded but not marked down, causing overall throughput to collapse to the speed of the slowest rank. Addressing these issues requires enumerating failure domains early, specifically platform health, GPU performance, network fabrics, cluster jobs, and inference services, before silent failures consume GPU hours. Mapping Infrastructure Components to Observability Tools Selecting the minimum toolset that eliminates coverage gaps prevents redundant exporters and alert fatigue. For GPUs, DCGM covers utilization, power, temperature, and XID/ECC errors, while NVSM aggregates DGX-class node and drive health. Fabric observability depends on architecture: UFM monitors InfiniBand, NetQ monitors Spectrum Ethernet/RoCE, and NMX manages rack-scale NVLink. Base Command Manager acts as the cluster and job aggregator rather than a source for low-level counters, and Run:ai or NIM enter the stack only when workload scheduling fairness or inference service-level objectives become operational priorities. Deploying the Framework on an InfiniBand Cluster For a standard DGX InfiniBand cluster running training jobs, the baseline telemetry stack combines Redfish/IPMI for platform hardware, DCGM for GPU performance, NVSM for node health, UFM for port and bit error telemetry, and BCM for job scheduling context. Excluded initially are NetQ, NMX, and inference metrics until their respective hardware or service configurations exist. Integrating these distinct sources is necessary because neither DCGM nor UFM alone can catch both network bit error regressions and GPU-level faults simultaneously. Actionable Alerting and Two-Layer Triage Operational monitoring prioritizes a concise top-k alert set tied to service-level indicators rather than raw counter dumps. Key baseline metrics include IPMI fan and power telemetry, DCGM GPU utilization and XID errors, and UFM bit error rates and chip temperatures. Architecture implementations scrape these exporters into Prometheus and visualize them through Grafana. Operations adopt a two-layer structure: Layer 1 provides a unified Grafana dashboard to identify whether an issue sits in the node, GPU, or fabric, while Layer 2 utilizes component-specific vendor interfaces for deep-dive root-cause analysis.

NVIDIA 详解 AI 工厂全栈可观测性选型与落地框架 在 AI 基础设施中，底层硬件与网络的微小降级（如灰色故障）极易在紧密耦合的大规模分布式训练中引发级联阻塞。为避免“外表全绿、内部停滞”的西瓜指标现象，NVIDIA 提出了全栈可观测性决策框架，指导运维团队以精简的工具集与 Top-k 告警快速定位并隔离故障。 分布式训练采用 BSP 同步模式，单个链路误码率上升引发的灰色故障会拖慢整个 NCCL all-reduce 通信并造成全局阻塞。 将 AI 工厂划分为平台健康、GPU 性能、网络交换 Fabric、集群与作业调度、推理服务五大故障域并建立工具覆盖矩阵。 采用最小工具集原则：如 InfiniBand 集群采用 IPMI、DCGM、NVSM、UFM 与 BCM 组合，通过 Prometheus 和 Grafana 统一汇聚。 推行两层监控架构：Layer 1 统一部署高维分诊看板，Layer 2 结合各组件专用 UI（如 UFM、BCM）进行底层根因分析。 可观测性成熟度取决于能否在浪费算力前明确故障组件与处置动作，告警必须绑定负责人与排障 Runbook。 AI 训练中的灰色故障与级联瓶颈 在 AI 基础设施中，一个层级出现的性能下降往往根源于其他技术栈。例如在大规模分布式训练任务中，GPU 利用率和排队时间看似正常，但单个 InfiniBand 链路的误码率（BER）上升会导致链路级重传。由于大模型训练遵循大步同步并行（BSP）模型，NCCL all-reduce 等同步集合通信对掉队者（straggler）极为敏感，单个 Rank 的延迟会导致整个作业的吞吐量降至最慢节点水平，形成硬件未被标记为“停机”但全局算力受损的灰色故障与级联失效。 AI 工厂故障域与工具选型决策矩阵 构建可观测性前必须明确覆盖五大领域：平台健康（风扇、PSU、BMC、CPU、内存等）、GPU 状态与性能（利用率、温度、功耗、XID/ECC、NVLink）、网络 Fabric（IB/以太网链路完整性、拥塞及机架级 NVLink）、集群作业（调度、排队、空闲分配）以及推理服务。针对这些领域，应遵循“用最少工具完全覆盖核心领域”的原则。GPU 指标首选 DCGM 接入 Prometheus，DGX 节点健康选用 NVSM，IB 网络采用 UFM，以太网采用 NetQ，机架级 NVLink 采用 NMX，集群调度聚合采用 BCM。避免引入无排障路径的多余采集器造成告警疲劳和“西瓜指标”。 以 InfiniBand 集群为例的最小监控栈落地 以基于 DGX、InfiniBand、BCM 和 Slurm 的训练集群为例，在初始阶段排除 NetQ、NMX、Run:ai 及 NIM。通过每台节点的 Redfish/IPMI 监控平台物理状态，DCGM 监控 GPU 算力与 NVLink，NVSM 聚合整机健康，UFM 跟踪 IB 端口误码率与拥塞，BCM 作为集群层聚合作业与硬件告警。该组合可覆盖 DCGM 单独使用时遗漏的跨节点网络长尾延迟，以及 UFM 单独使用时遗漏的 GPU 故障，最终将各项采集端统一接入 Prometheus。 建立精简的 Top-k 告警与两层分诊看板 运维不应倾泻所有底层计数器，而应建立与 SLI/SLO 绑定的 Top-k 告警集。指标精简聚焦于关键项：平台级的风扇、电源与温度；GPU 级的利用率、功耗与 XID 报错；IB 网络的传输数据量、误码计数（Effective_BER 等）与芯片温度；以及 BCM/Slurm 的在运行任务与等待时间。架构上采用两层监控体系：Layer 1 为 Grafana 统揽看板，统一展示 GPU、节点与网络的时间序列，用于毫秒级判断故障所属层级；Layer 2 保留 UFM、BCM 等原生界面，供确认故障域后深入定位根因。 验收准则与三步分阶段扩展路径 可观测性落地的验收标准并非仪表板数量，而是覆盖域是否完整、告警是否绑定明确操作、各信号是否统一时间线呈现。实施推广分为三步：一是确立覆盖，按矩阵挑选全功能支持工具；二是集成导出，打通 IPMI、DCGM、NVSM、UFM、BCM 至 Prometheus 与 Grafana；三是落实归属，在添加新采集前必须明确告警负责人与处置 Runbook。后续当引入以太网、机架级 NVLink 或推理服务时，再按序引入 NetQ、NMX 及 NIM 算子监控。

AI infrastructure spans multiple layers, from compute and networking to storage, orchestration, and applications. When performance degrades, identifying the source can be difficult because a symptom observed at one layer may originate elsewhere in the stack. A full-stack observability strategy connects telemetry across these layers, helping infrastructure and operations teams detect problems, isolate their causes, and maintain reliable AI workloads. This post presents a practical observability framework for NVIDIA AI infrastructure and shows how to apply it to common monitoring and troubleshooting scenarios. Consider a distributed training job that is three days into execution. GPU utilization and queue wait times remain normal. After six hours of reduced throughput, the team traces the cause to a single InfiniBand link drifting into an elevated bit error rate. This is a classic gray failure . The hardware is degraded, but the system does not report it as “down.” AI training follows the bulk synchronous parallel (BSP) model. These tightly coupled systems are sensitive to stragglers: one slow rank holds back the job. Link-level retransmissions stall a single rank during synchronous collective operations such as NVIDIA Collective Communications Library (NCCL) all-reduce. Throughput then falls to the slowest rank, and the other ranks block. That is a cascading failure. You see this failure mode often in AI factories. The required telemetry typically already exists; the challenge is selecting the right signals from the right tools early enough to act. Operators do not need every metric from every product. They need a decision path that maps components to tools, tools to a concise alert set, and that alert set into a single triage dashboard. This post shows that path. You will learn how to: Enumerate the failure domains that must be observable before selecting software. Map AI infrastructure components to telemetry tools sources using a decision framework derived from NVIDIA DGX deployments. Apply the observability framework to an InfiniBand cluster. Reduce telemetry to a top-k alert set and correlate signals in a single triage dashboard. Keep detailed catalogs, protocol matrices, and per-tool enablement guides in product documentation. Here, you’ll focus on how to choose an observability stack. NVIDIA DGX and NVIDIA HGX deployments share the same observability surface, even when hardware configurations differ (Figure 1): Identify AI factory failure domains Figure 1. AI factory architecture Before selecting monitoring software, enumerate the domains in which silent failure consumes GPU hours: Platform health : Fans, PSUs, BMC, chassis, CPU, memory, local storage. GPU health and performance : Utilization, temperature, power, XID/ECC, and NVIDIA NVLink throughput. Fabric : InfiniBand or Ethernet link integrity, congestion, and switch/cable health; rack-scale NVLink where present. Cluster and jobs : Scheduling, reservations, idle allocated GPUs, queue wait. Inference services : Latency, success rate, and cache behavior when NVIDIA NIM microservices or similar services are in production. In complex systems, coverage gaps are rarely closed in one pass. They appear later, when latent failure modes show up under load . Analyzing those modes early shortens discovery. This analysis can help prevent failures from recurring under load. Map AI infrastructure components to telemetry tools Operations teams need a clear mapping from component to telemetry source. Table 1 maps NVIDIA Data Center GPU Manager (DCGM), NVIDIA System Management (NVSM), NVIDIA Unified Fabric Manager (UFM), NVIDIA NetQ, NVIDIA NMX, NVIDIA Base Command Manager (BCM), and NVIDIA Run:ai to those components. Green indicates full support for the domain; yellow indicates partial or indirect coverage. Use the framework to select the minimum tool set that eliminates coverage gaps. Component Redfish / IPMI DCGM NVSM UFM NetQ NMX BCM Run:ai NIM Base infrastructure 🟢 ⚪ ⚪ ⚪ ⚪ ⚪ 🟢 ⚪ ⚪ Compute node 🟢 🟡 🟢 ⚪ ⚪ 🟡 🟢 ⚪ ⚪ GPU ⚪ 🟢 🟢 ⚪ ⚪ ⚪ 🟢 🟡 ⚪ Node interconnects ⚪ 🟢 🟡 ⚪ ⚪ ⚪ 🟢 ⚪ ⚪ Rack-scale NVLink ⚪ ⚪ ⚪ ⚪ ⚪ 🟢 ⚪ ⚪ ⚪ Ethernet network ⚪ ⚪ ⚪ ⚪ 🟢 ⚪ 🟢 ⚪ ⚪ InfiniBand network ⚪ 🟡 ⚪ 🟢 ⚪ ⚪ 🟢 ⚪ ⚪ Cluster management ⚪ ⚪ ⚪ ⚪ ⚪ ⚪ 🟢 ⚪ ⚪ Jobs and workloads ⚪ ⚪ ⚪ ⚪ ⚪ ⚪ 🟢 🟢 ⚪ AI inference ⚪ ⚪ ⚪ ⚪ ⚪ ⚪ ⚪ 🟡 🟢 Legend: 🟢 = Full support · 🟡 = Partial/indirect support Table 1. Observability decision framework Key tradeoffs include: DCGM compared to NVSM for GPUs : Prefer DCGM for utilization, power, temperature, NVLink, and XID/ECC export into Prometheus. Retain NVSM for system health on DGX-class nodes (drives, power, overall health). The tools overlap on GPU metrics; neither substitutes for platform BMC data. UFM compared to NetQ : Select by fabric type. InfiniBand uses UFM. Spectrum Ethernet/RoCE uses NetQ. Deploy both only when both fabrics are present. NMX : Required for rack-scale NVLink. Omit on classic multi-node NVLink topologies where DCGM already covers node interconnects. BCM : Treat BCM as the aggregator and cluster/job plane, not as the source of low-level counters. Specialized tools remain responsible for deep telemetry. Run:ai and NIM : Introduce when workload scheduling fairness or inference SLOs are first-class operational requirements. Neither replaces DCGM or fabric monitoring. A useful rule: cover every required green cell with the fewest tools. Extra exporters without a clear triage path add noise, not observability. That noise leads to alert fatigue. Teams keep adding metrics and dashboards, yet still cannot answer what is broken and why. The result is “watermelon metrics”: dashboards that look green outside while services fail inside. The corrective principle is the one stated in “ As Simple as Possible, No Simpler ”: keep monitoring simple, and remove unused signals rather than accumulate them. Apply the observability framework to an InfiniBand cluster Consider a DGX cluster with InfiniBand, BCM, and Slurm. Most jobs are training; inference is not yet in production. The operational requirement is a single triage dashboard and alerts that detect fabric and GPU health regressions before multi-hour job waste accumulates. Here’s the decision process: Domains in scope : platform, GPU, InfiniBand fabric, cluster/jobs. Out of scope for the initial deployment: NetQ, NMX, Run:ai, NIM. Tool selection from the framework Redfish/IPMI on every node for fans, PSU, chassis, and BMC state. DCGM on every GPU node for utilization, power, temperature, XID/ECC, and NVLink. NVSM on DGX nodes for system health aggregation. UFM for InfiniBand port health, BER, congestion, and routing. BCM as the cluster aggregator for jobs, reservations, and consolidated hardware alerts. Rationale : DCGM alone would miss the BER regression described in the introduction. This is a tail-at-scale problem . When work is synchronized across many ranks, end-to-end throughput follows the slowest rank, not average component health. UFM alone would miss GPU XID storms and node power faults. BCM alone would not generate the low-level counters. Combined, they cover the domains that waste GPU hours in this environment. Exclusions : Skip NetQ, NMX, and inference metrics until Ethernet, rack-scale NVLink, or inference services are introduced. That gives you an initial stack of IPMI, DCGM, NVSM, UFM, and BCM, unified in Prometheus/Grafana. Build an actionable AI infrastructure alert set Most tools expose hundreds of metrics. Prefer a short top-k set tied to service-level indicators (SLIs) and service-level objectives (SLOs) , not a dump of every hardware counter. Each alert should map to a clear remediation action. UFM Telemetry exposes hundreds of fields; start with the documented high-frequency telemetry fields . For the apply the observability framework to an InfiniBand cluster, begin with: Platform : fan speed, PSU status, key temperatures ( SPD_FAN_* , PWR_* , TEMP_* via Redfish/IPMI). GPU : DCGM_FI_DEV_GPU_UTIL , DCGM_FI_DEV_MEM_COPY_UTIL , DCGM_FI_DEV_POWER_USAGE , DCGM_FI_DEV_XID_ERRORS , plus NVSM GPU/system health. InfiniBand : PortXmitDataExtended , SymbolErrorCounterExtended , Effective_BER , Total_Raw_BER , Chip_Temp . Jobs : BCM/Slurm signals for running jobs, GPU reservations, and wait time so fabric or GPU alerts correlate to workload impact. Expand the set only when an incident demonstrates a coverage gap. Consistent with Symptoms Versus Causes , alert on symptoms that map to a defined action (drain node, replace cable, open fabric case), not on every counter a collector can emit. Prefer Prometheus exporters where available: DCGM and NVSM both expose Prometheus endpoints; UFM and BCM can feed the same scrape model via exporters or APIs. This keeps protocol selection simple for the initial deployment and avoids introducing a second control plane before gNMI or SNMP integration is required. Build a unified AI infrastructure triage dashboard With tools and top-k metrics selected, build a unified AI infrastructure triage dashboard (Figure 2): Figure 2. Unified observability architecture A practical architecture looks like this: Install IPMI and DCGM exporters on every GPU node. Run UFM Telemetry where the fabric is reachable; scrape or export into Prometheus. Retain BCM as the cluster management and aggregation plane. Point Grafana at Prometheus for dashboards and alerts across GPU, node, and fabric signals. Add a community Slurm dashboard if job-level context is not already available in BCM. Use a two-layer monitoring approach. Layer 1 gives a high-level view for triage; Layer 2 keeps the detail needed to inspect individual components. Layer 1 is the Grafana dashboard used first: is the fault in the GPU, node, or fabric? Layer 2 is the vendor UI deep dive (UFM web UI, BCM Base View, and similar tools) once the failing domain is known. Day 2 operations is faster when Layer 1 answers the triage question from one board. Layer 2 stays available for root-cause analysis. Define observability acceptance criteria You’re ready to move on when: Every failure domain in scope has at least one full-support tool from the framework. Alerts are bound to a short top-k metric list with owners and actions. GPU, node, and fabric signals share a common timeline in one view. Further tools (Ethernet, rack-scale NVLink, Run:ai, NIM) are added only when needed, not by default. Don’t measure observability maturity by the number of dashboards. Measure it by whether your signals show the failing component and the next action before significant compute capacity is wasted. Expand the NVIDIA AI infrastructure observability stack After your initial deployment meets the define observability acceptance criteria , expand coverage in this order: Enable GPU telemetry with the DCGM User Guide and GPU Telemetry . Validate DGX system health paths in the NVSM User Guide . Configure fabric visibility with UFM Telemetry or NetQ , depending on the network fabric. For cluster aggregation and operations, begin with Base Command Manager and, where applicable, NVIDIA Mission Control . For inference services, add NVIDIA NIM Operator observability. Use the decision framework to justify each addition. Keep detailed metric dictionaries and protocol matrices in runbooks or product docs. Keep the production alert set short enough for on-call use. A decision framework beats a metric catalog. It’s what gets you to a few well-chosen signals and one triage board instead of fifty dashboards nobody reads. 3-step rollout checklist: Establish coverage: Pick one full-support tool for each in-scope domain (see Table 1). Integrate exporters: Wire Redfish/IPMI, DCGM, NVSM, UFM, and BCM into Prometheus, and point Grafana at the unified telemetry endpoint. Enforce ownership: Bind every alert to an owner and a playbook action before adding the next exporter. Don’t measure observability maturity by the number of dashboards. Measure it by whether your signals name the failing component and the next action before significant compute capacity is wasted.