NVIDIA Introduces Topograph for Dynamic, Topology-Aware Workload Scheduling Across Clusters

NVIDIA Introduces Topograph for Dynamic, Topology-Aware Workload Scheduling Across Clusters NVIDIA has detailed Topograph, an open-source toolkit designed to map cluster network topologies and dynamically publish them to workload managers like Kubernetes, Slurm, and Slinky. By continuously discovering interconnect hierarchies across cloud APIs and on-premises fabrics, Topograph helps schedulers minimize network contention and optimize GPU placement in AI factories. Topograph translates cloud and on-premises physical network topologies into canonical models consumed by Kubernetes labels, NFD resources, Slurm configs, and Slinky ConfigMaps. The system uses five internal components—an API Server, Node Observer, Node Data Broker, Provider, and Engine—to update topology views automatically as clusters change. Kubernetes integration maps switch and accelerator tiers into variable-depth labels usable by standard affinity, Kueue, and the KAI Scheduler. For Slurm and Slinky, Topograph generates cluster-wide tree, block, or per-partition YAML configurations, supporting both bare-metal and containerized environments. The Topology Bottleneck in AI Factories Distributed AI training and inference rely heavily on high-speed interconnects, including NVIDIA NVLink (up to 1.8 TB/s per GPU in Blackwell and 3.6 TB/s in Vera Rubin) and Quantum InfiniBand (up to 800 Gb/s). Suboptimal workload placement forces traffic across shared links and switches, increasing contention and causing GPUs to waste provisioned power waiting on data. While schedulers like Slurm and Kubernetes support topology-aware placement, they often fail when operating on outdated, manually maintained network maps. Architecture: Providers and Engines Topograph resolves this limitation through a modular architecture based on providers and engines. Providers query cloud APIs (such as Google Cloud, Lambda, Nebius, Nscale, and OCI) or on-premises networks (like InfiniBand and Spectrum-X/NetQ fabrics) and normalize data into a canonical model. Engines then convert this model into target-specific formats. Topograph stays updated via an API Server, a Node Observer watching cluster changes, a Node Data Broker gathering node attributes, and dynamic generation routines featuring built-in aggregation delays (typically 15 seconds) to avoid redundant computations during burst events. Topology Scheduling on Kubernetes Deployed via Helm, Topograph publishes fabric hierarchy as hierarchical node labels, where tier-0 represents the closest leaf switch and tier numbers increase outward, alongside two-level accelerator domain labels. Schedulers consume these labels to keep related workloads clustered. Native Kubernetes can evaluate them using weighted podAffinity terms, while advanced schedulers like KAI Scheduler and Kueue use them for topology-aware gang scheduling. For environments utilizing Node Feature Discovery, an alternate NFD engine publishes NodeFeature and NodeFeatureGroup objects. Deployments with Slurm and Slinky On bare-metal or virtualized Slurm environments, Topograph installs via Debian or RPM packages and generates cluster-wide tree, block, or per-partition YAML configurations (introduced in Slurm 25.05), with optional automated triggers and scontrol reconfigure execution. For Slinky—the SchedMD platform running Slurm within Kubernetes—Topograph translates node data to slurmd Pods and writes topology specifications directly into ConfigMaps or node annotations, enabling automated topology mapping in hybrid container environments.

NVIDIA推出开源工具Topograph以实现拓扑感知工作负载调度 NVIDIA介绍了开源工具Topograph，旨在解决AI工厂中GPU工作负载调度时的网络拓扑碎片化和跨交换机通信延迟问题。该工具能够从云API或本地网络架构中自动发现物理网络拓扑，将其归一化并输出给Kubernetes、Slurm及Slinky等调度器，从而实现动态、自动化的拓扑感知调度。 不良的GPU工作负载布局会导致网络拓扑碎片化，增加延迟并空耗电力；NVLink与Quantum InfiniBand等技术需要调度器掌握精准的互连拓扑以发挥最大效能。 Topograph通过Provider从云平台（如GCP、OCI、Lambda等）或本地网络（Spectrum-X、InfiniBand等）发现拓扑，并通过Engine将其转化为调度器所需的配置。 在Kubernetes上，Topograph生成织物层级标签，可配合KAI Scheduler或Kueue实现拓扑感知Gang调度，也支持NFD资源输出。 在Slurm及基于Kubernetes运行Slurm的Slinky环境中，Topograph支持生成Tree、Block模式或按分区的YAML拓扑配置，并能通过API或节点事件触发自动重新配置。 拓扑感知调度的核心问题与Topograph架构 分布式AI工作负载在训练和推理过程中需要高频通信，跨远距离交换机传输会导致链路争用和吞吐量下降。Topograph是一个开源工具集，通过Provider从云API或本地系统获取物理网络拓扑并进行标准化，再通过Engine生成调度系统可识别的格式。其核心包含五个组件：API Server（处理请求与合并防抖）、Node Observer（监听节点/Pod变更触发更新）、Node Data Broker（收集单节点属性）、Provider（数据归一化）和Engine（格式转换输出）。 Kubernetes与NFD支持模式 针对Kubernetes环境，Topograph将织物层级（如fabric.topograph.run/tier-0最近交换机层）和加速器域写入节点标签。原生调度器可通过Pod亲和性偏好这些标签；KAI Scheduler或Kueue TAS等调度器则可基于这些层级定义拓扑树，实现跨节点的拓扑感知Gang调度。此外，Topograph还提供nfd引擎，将拓扑作为NodeFeature和NodeFeatureGroup对象发布至Node Feature Discovery。 在Slurm环境中的部署与配置生成 在裸机或虚拟机Slurm环境中，Topograph以Linux系统服务运行。它可以通过/v1/generate端点生成传统集群级别的tree和block格式topology.conf文件，或者Slurm 25.05引入的分分区YAML配置。管理员可配置reconfigure参数使Topograph在生成配置后自动执行scontrol reconfigure，或通过strigger脚本根据节点状态变化自动刷新拓扑。 Slinky环境下的Slurm容器化整合 对于在Kubernetes上运行Slurm的Slinky方案（由2025年12月被NVIDIA收购的SchedMD开发），Topograph以Helm Chart部署并把Kubernetes节点映射至slurmd Pod，随后将Slurm拓扑写入ConfigMap。针对MNNVL系统，Topograph还提供基于DRA的Provider，直接读取GPU clique标签构建Block拓扑，同时支持为动态Slurm节点自动添加拓扑注解。

AI factories are power-limited systems that deliver maximum value when fully optimized. GPU workload placement is a key optimization. Poor workload placement fragments topology domains and forces traffic across shared links, reducing throughput, raising job costs, and leaving GPUs consuming provisioned power while waiting on data without advancing the workload. GPUs exchange data continuously during training and inference, so distributed workloads benefit from communication locality. NVIDIA NVLink and NVLink Switch provide high-bandwidth, all-to-all scale-up connectivity within rack-scale GPU domains, while NVIDIA Spectrum-X Ethernet provides predictable, low-latency scale-out networking across systems and racks. A scheduler can place workloads efficiently only with a current, accurate view of those GPU and fabric relationships, and keeping that view current as the cluster changes is where placement breaks down in practice. NVIDIA Topograph solves that problem. It discovers cluster topology from cloud APIs or on-premises fabric systems, normalizes it into a common model, and publishes it in the format each workload manager expects: Kubernetes node labels, Slurm topology configuration, or Slinky ConfigMaps. Within the NVIDIA DSX OS cluster orchestration layer, Topograph works alongside Dynamic Resource Allocation (DRA) and KAI Scheduler to enable topology-aware gang scheduling across AI factory infrastructure. This post walks through deploying Topograph and using it to schedule topology-aware workloads on Kubernetes, Slurm, and Slinky. The core topology problem Topograph maps how cluster hardware is connected so schedulers can favor nearby resources. Think of the network as a road system: GPUs within the same locality domain have short, high-bandwidth paths, while traffic between domains crosses more shared links and switches. Spreading a tightly coupled workload across distant domains can increase contention and latency, so Topograph helps place workloads in the most efficient locations and avoid these bottlenecks. Modern NVIDIA Quantum InfiniBand ports can achieve up to 800 Gb/s , while NVIDIA NVLink provides 1.8 TB/s of bidirectional bandwidth per GPU in its fifth generation (NVIDIA Blackwell, such as GB200/GB300) and 3.6 TB/s per GPU in its sixth generation (Vera Rubin), through a dedicated NVIDIA NVLink Switch fabric. That non-blocking, all-to-all design gives each GPU its own lane rather than sharing bandwidth under load. Schedulers with a current view can favor GPUs in the closest topology domain. Slurm and Kubernetes both support topology-aware allocation, but a scheduler can only act on the topology it observes. Topograph regenerates that view on request and upon watched cluster changes, so the scheduler works from current data rather than a manually maintained snapshot. A common model across environments Topograph is an open source toolkit that identifies a cluster’s network topology, enabling workload managers to make topology-aware scheduling decisions. It has two concepts: providers and engines. A provider discovers topology from cloud APIs or on-premises systems and normalizes it into a canonical model. An engine translates that model into Slurm configuration, Kubernetes labels, Slinky ConfigMaps, Node Feature Discovery (NFD) resources, or instance-oriented topology JSON. Cloud providers that have a working integration with Topograph include Google Cloud , Lambda , Nebius , Nscale , and OCI , with more cloud and colocation providers in development. Environment and Engine Support Environment or provider Kubernetes Slurm Graph Node labels (k8s) NFD resources (nfd) Slinky ConfigMap (slinky) Cloud and hosted providers Crusoe Yes Yes Yes Yes Yes Google Cloud Yes Yes Yes Yes Yes Lambda Yes Yes Yes Yes Yes Nebius Yes Yes Yes Yes Yes Nscale Yes Yes Yes Yes Yes Oracle Cloud Infrastructure (OCI) Yes Yes Yes Yes Yes On-premises deployment models InfiniBand in Kubernetes Yes Yes Yes Yes Yes InfiniBand on bare metal or VMs No No No Yes Yes On-premises networking and topology Spectrum-X or NetQ-managed fabric Yes Yes Yes Yes Yes MNNVL NVLink partitions (DRA block topology only) No No Yes No No Table 1. Supported topology providers by engine Scope and interpretation . This matrix reflects current upstream main as of September 16, 2026. It shows supported provider-to-engine output combinations; requirements can vary by Topograph version, environment, and provider configuration. The Crusoe provider reads fabric and accelerator-domain labels from Crusoe Managed Kubernetes nodes; Topograph therefore runs in Kubernetes for this provider. The Slurm engine can run in Kubernetes, but it requires a writable volume for its configured topology.conf output path. The NFD engine requires the alpha NodeFeatureGroupAPI feature gate. The Kubernetes engine publishes Node labels instead. Staying current as the cluster changes Five components keep that view current: API Server: Validates requests, aggregates duplicates, and dispatches discovery Node Observer: Watches configured Kubernetes node or Pod changes and API readiness, then requests regeneration with retries Node Data Broker: Collects per-node attributes and stores them as node annotations Provider: Converts cloud or fabric data into the canonical representation Engine: Writes the representation in a format the scheduler understands Figure 1. Topograph accepts a generation request, discovers and normalizes topology from a selected cloud or network fabric provider, and publishes scheduler-ready outputs for Slurm and Kubernetes or instance-oriented topology JSON. Kubernetes deployments can also use runtime helpers to react to cluster changes and collect per-node data How clients query topology The API server exposes five service endpoints: POST /v1/generate – submits an asynchronous request and returns its ID with HTTP 202. GET /v1/topology?uid=<request-id> – returns HTTP 202 while processing and HTTP 200 with the result when complete. POST /v1/lookup – returns the cached status or result for the same request body without submitting it again. GET /healthz – is the liveness endpoint. GET /metrics – exposes Prometheus metrics. The aggregation delay is required; 15 seconds is typical. Repeated identical requests reset a trailing timer and are processed once, reducing redundant work during bursts of cluster events. For testing without production hardware, simulation models describe node and switch hierarchies. The kwok-nodes utility and Kind/KWOK helpers turn those models into virtual Kubernetes nodes. Solving it on Kubernetes (engine: k8s) The default Kubernetes scheduler doesn’t discover physical interconnect hierarchy. Topograph addresses that gap by publishing provider-reported topology as node labels, which native affinity and topology-aware schedulers can consume. Prerequisites are Kubernetes 1.27 or later, Helm 3.10+ or 4.x, kubectl permissions, and a supported provider . KAI Scheduler or Kueue TAS is optional for topology-aware gang scheduling. Deploying Topograph with Helm Topograph is distributed as a Helm chart : helm repo add topograph https://dsx-ai-factory.github.io/topograph helm repo update helm install topograph topograph/topograph \ --namespace topograph \ --create-namespace \ --set engine.name=k8s \ --set provider.name=<provider> Replace <provider> with the value that matches your environment. The repository includes example Helm values files in charts/topograph , named with a values.k8s prefix and a short scenario description. Each carries inline configuration comments. After installation, verify that the deployment completed successfully: helm test topograph --namespace topograph The bundled test hooks query /healthz and /metrics in-cluster and confirm the responses include the topograph_version metric. Confirm the Pods are running: kubectl get pods -n topograph Verifying topology labels on nodes Topograph represents fabric locality with a variable-depth label family and accelerator locality with a two-level hierarchy: fabric.topograph.run/tier-0 # switch closest to the node fabric.topograph.run/tier-1 # next fabric tier outward fabric.topograph.run/tier-<N> # additional discovered tiers accelerator.topograph.run/domain # accelerator domain accelerator.topograph.run/sub-domain # optional nested sub-domain Fabric tier 0 is the leaf switch closest to the compute node, and tier numbers increase outward. Topograph writes only the tiers present in the discovered topology, with no fixed maximum depth. Operators can set the Kubernetes engine’s fabricLabels array and acceleratorLabel parameter to use custom keys; tiers beyond that array are not labeled. The sub-domain key is fixed. To verify that the labels have been applied, run: kubectl get nodes --show-labels | grep -E 'fabric\.topograph\.run|accelerator\.topograph\.run' If labels are missing, inspect the Topograph logs: kubectl logs -n topograph -l app.kubernetes.io/name=topograph NOTE: Topograph reflects reported rather than intended topology. Labels refresh when generation runs, for example, after a watched node or pod change. Visibility of a fabric change depends on the provider and its triggering events. Exposing the API The API is a ClusterIP service by default. With the release and namespace above, its address is: topograph.topograph.svc.cluster.local:49021 . For local debugging: kubectl -n topograph port-forward svc/topograph 49021:49021 curl http://localhost:49021/healthz Using standard Kubernetes scheduling Topology labels can be used as topologyKey values in preferred Pod affinity: affinity: podAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 90 podAffinityTerm: labelSelector: matchLabels: app: myapp topologyKey: fabric.topograph.run/tier-0 - weight: 70 podAffinityTerm: labelSelector: matchLabels: app: myapp topologyKey: fabric.topograph.run/tier-1 Each matching term contributes to a candidate node’s score, strongly favoring the tier-0 domain of existing app=myapp Pods while also rewarding tier-1 locality. Because the default scheduler places Pods individually, this is a preference rather than globally optimal gang placement. KAI Scheduler and Kueue can use the same node labels for topology-aware gang placement. Kubernetes 1.36 also introduced alpha topology-aware workload scheduling through KEP-5732 . Upstream beta work is ongoing; consult the enhancement tracker rather than depending on a specific future release. Using KAI Scheduler for Topology-Aware Gang Scheduling KAI Scheduler (a CNCF Sandbox project donated by NVIDIA) organizes node labels into a hierarchy : apiVersion: kai.scheduler/v1alpha1 kind: Topology metadata: name: cluster-topology spec: levels: - nodeLabel: topology.kubernetes.io/zone - nodeLabel: fabric.topograph.run/tier-1 - nodeLabel: fabric.topograph.run/tier-0 - nodeLabel: kubernetes.io/hostname Apply it with kubectl apply -f cluster-topology.yaml , then annotate a multi-Pod Job: apiVersion: batch/v1 kind: Job metadata: name: topology-aware-workers annotations: kai.scheduler/topology: cluster-topology kai.scheduler/topology-required-placement: fabric.topograph.run/tier-1 kai.scheduler/topology-preferred-placement: fabric.topograph.run/tier-0 spec: parallelism: 4 completions: 4 template: metadata: labels: app: inference-worker spec: schedulerName: kai-scheduler restartPolicy: Never containers: - name: worker image: nvcr.io/nvidia/nemo:latest resources: limits: nvidia.com/gpu: 1 The required annotation keeps the gang within a single tier-1 domain. The preferred annotation asks KAI to concentrate Pods in a tier-0 domain when feasible, but permits multiple tier-0 domains inside the required boundary. For more advanced topology-aware scheduling examples, see the documentation for Grove and NVIDIA Dynamo . Grove provides Kubernetes APIs and an operator for hierarchical gang scheduling, topology-aware placement, and coordinated scaling. Dynamo is an open source distributed inference serving framework that integrates with Grove for Kubernetes workload orchestration. Publishing topology through NFD (engine: nfd) Topograph also supports consumers already using Node Feature Discovery. The nfd engine publishes one NodeFeature per selected topology node and one NodeFeatureGroup for every distinct fabric-tier, XCLR-domain, and XCLR-sub-domain value. The NFD master evaluates those specifications and owns each group’s status.nodes membership. Install nfd first with its alpha NodeFeatureGroupAPI feature gate enabled; it is off by default. Then select the engine and the namespace where the NFD master runs: engine: name: nfd nfdNamespace: node-feature-discovery Use this output when a downstream component consumes NodeFeatureGroup objects; it is not a substitute for Kubernetes topologyKey labels. For native Pod affinity, KAI Scheduler, or Kueue TAS, continue to use engine: k8s . The chart scopes NFD permissions to the nfd namespace. The engine deletes stale Topograph-managed objects after reconciliation, but preserves the last published topology if a generation produces none. Solving it on Slurm (engine: slurm) Topograph generates cluster-wide configurations in the tree and block formats , shown in the top-center and bottom-center panels of Figure 2, below. Slurm 25.05 introduced per-partition configuration in YAML format , which Topograph also supports, as shown in the diagram. Figure 2. A representative three-tier cluster topology with Multi-Node NVLink domains and Topograph’s configuration-dependent Slurm output modes: cluster-wide tree, cluster-wide block, or per-partition topology YAML Installing Topograph Slurm clusters typically run on Linux bare-metal servers or virtual machines, where Topograph is installed via a native package manager. The repository includes Debian and RPM build targets: make deb # Debian / Ubuntu make rpm # RHEL / Rocky / SUSE The package installs the service without starting it, so you can review and edit the configuration file /etc/topograph/topograph-config.yaml http: port: 49021 provider: <provider> engine: slurm requestAggregationDelay: 15s Replace <provider> with the value that matches your environment. After updating the configuration, start the service and verify that it is healthy: sudo systemctl enable --now topograph.service curl http://localhost:49021/healthz Generating Slurm topology configuration To initiate discovery, POST to Topograph’s /v1/generate endpoint, which regenerates the Slurm topology configuration. Submit a request and poll its result: id=$(curl -s -X POST -H 'Content-Type: application/json' \ -d @payload.json http://localhost:49021/v1/generate) curl -s "http://localhost:49021/v1/topology?uid=$id" For cluster-wide tree output, use an absolute path: { "engine": { "name": "slurm", "params": { "plugin": "topology/tree", "topologyConfigPath": "/etc/slurm/topology.conf", "reconfigure": true } } } Use topology/block plus optional blockSizes for block output. The optional reconfigure parameter runs scontrol reconfigure after a file is written and defaults to false. If topologyConfigPath is omitted, Topograph returns the generated content from the result endpoint instead of writing a file. { "engine": { "name": "slurm", "params": { "topologies": { "gpu-block": { "partition": "gpu", "plugin": "topology/block", "blockSizes": [8, 16] }, "cpu-tree": { "partition": "cpu", "plugin": "topology/tree" }, "default": { "plugin": "topology/flat", "clusterDefault": true } }, "topologyConfigPath": "/etc/slurm/topology.yaml", "reconfigure": true } } } For node-state-driven refresh with a provider that auto-discovers Slurm node mappings, run the repository’s script as root: scripts/create-topology-update-script.sh -p <provider> -c /etc/slurm/topology.conf It registers a permanent strigger for node up and down transitions. It does not detect arbitrary switch rewiring or every inventory change. Solving It on Slinky (engine: slinky) Slinky, developed by SchedMD, runs Slurm on Kubernetes. NVIDIA acquired SchedMD in December 2025. The Topograph Slinky engine maps Kubernetes nodes to slurmd Pods and writes Slurm topology data to a ConfigMap. The Slinky engine supports cluster-wide topology/tree and topology/block output, as well as multiple-topology YAML for partition-specific configurations. Install Topograph as a Helm chart : helm repo add topograph https://dsx-ai-factory.github.io/topograph helm repo update helm install topograph topograph/topograph \ --namespace topograph \ --create-namespace \ --values my-values.yaml The repository provides ready-to-adapt Helm examples for tree , block , per-partition , and InfiniBand block deployments. Topograph regenerates and updates the ConfigMap when selected slurmd Pods change. The dra provider is a narrower Slinky block-topology option for MNNVL systems. It reads existing nvidia.com/gpu.clique labels when regenerating the topology configuration. For dynamic Slurm nodes, the optional useDynamicNodes mode also annotates selected Kubernetes nodes with the current Slurm topology specification. ConfigMap updates and dynamic-node reconciliation are distinct mechanisms, so choose the mode that matches the deployed Slinky configuration. Getting started Placement problems compound at scale and surface as network congestion. Topograph gives schedulers a current, provider-reported map of the physical network, so topology-aware decisions stay consistent across cloud and on-premises environments without manual maintenance. Through KAI Scheduler, Kueue, and native Kubernetes, the map improves AI factory efficiency, tokens per watt, and cost. Deploy Topograph from the dsx-ai-factory/topograph GitHub repo , or learn more about the DSX OS ecosystem .