NVIDIA Introduces NodeWright for Declarative Host OS Management in Kubernetes Fleets

NVIDIA Introduces NodeWright for Declarative Host OS Management in Kubernetes Fleets NVIDIA has open-sourced NodeWright (formerly known internally as Skyhook), a Kubernetes-native package manager designed to declaratively configure, update, and validate host operating systems across GPU node fleets without interrupting sensitive training workloads. NodeWright operates under NVIDIA DSX OS as an open-source tool (Apache 2.0) that brings declarative, package-based host management to Kubernetes nodes. Updates follow a safe lifecycle sequence—cordon, wait for labeled non-interruptible workloads, drain, apply/configure, interrupt if needed, and uncordon. Deployment policies manage fleet-wide rollouts across compartments using fixed, linear, or exponential batching strategies with success and failure thresholds. NodeWright integrates with NVIDIA AI Cluster Runtime (AICR), NVCRE, and NVSentinel, functioning beneath the NVIDIA GPU and Network Operators rather than replacing them. Challenges of Managing GPU Infrastructure Traditional configuration management tools like Ansible or Puppet treat nodes individually and lack awareness of Kubernetes scheduling, pod disruption budgets, or active workloads. In GPU clusters, recycling nodes or interrupting long-running training jobs for kernel updates, driver fixes, or CVE remediations is costly and slow, often forcing teams to rely on manual runbooks and off-hours maintenance windows. Architecture and Node Lifecycle Management NodeWright consists of an operator, Custom Resources, and containerized packages containing scripts, binaries, and verification steps. When applying updates, the operator cordons the node, waits for workloads labeled as non-interruptible to complete, drains remaining pods, executes system modifications, handles necessary reboots or restarts, and uncordons the node. It can also manage taints on newly provisioned nodes so they remain unschedulable until all packages and validation checks pass. Progressive Rollout Strategies To safely execute fleet-wide changes, NodeWright uses DeploymentPolicy resources to divide nodes into compartments based on labels. Operators can define rollout strategies—fixed batch sizes, linear increments, or exponential growth—paired with batch success thresholds and failure thresholds that halt the rollout if consecutive batches fail, exposing errors through the Kubernetes API. Ecosystem and Runtime Integration NodeWright acts beneath the NVIDIA GPU Operator and Network Operator, focusing strictly on the host OS layer. It works alongside the NVIDIA AI Cluster Runtime (AICR) to apply version-locked recipes, NVCRE for pre-workload readiness validation, and NVSentinel for fault detection and remediation. NVIDIA provides intent-based tuning packages supporting architectures like Hopper and Blackwell, as well as cloud-specific setups for Amazon EKS and Google Kubernetes Engine.

NVIDIA 开源 Kubernetes 主机配置管理工具 NodeWright NVIDIA 宣布将此前内部命名为 Skyhook 的项目正式以 NodeWright 之名开源。该工具作为开源 DSX OS 的一部分，专为大规模 Kubernetes GPU 集群设计，提供声明式、工作负载感知的主机底层操作系统与配置管理方案。 NodeWright 填补传统配置管理工具不感知 Kubernetes 状态的空白，支持在不中断长期训练作业的前提下变更底层节点系统。 采用声明式架构与严密的节点变更时序（Cordon、Wait、Drain、Apply、Interrupt、Uncordon），并内置状态校验与回滚熔断保护。 提供支持固定、线性和指数增长的三种渐进式发布策略，并引入分区与批次阈值实现细粒度安全部署。 与 NVIDIA AI Cluster Runtime (AICR)、NVCRE 和 NVSentinel 深度集成，专职管理底层系统，与 GPU/Network Operator 形成互补。 GPU 节点维护痛点与声明式管理 传统如 Ansible 或脚本化的运维工具缺乏对 Kubernetes 状态的感知，在需要修改内核参数、系统包或修复 CVE 时，往往无法协调正在运行的关键 AI 训练负载。由于 GPU 硬件稀缺且长时间训练成本高昂，随意废弃重建节点并不现实。NodeWright 作为 Kubernetes 原生包管理器，支持通过 CRD、Helm 或 GitOps 工具统一部署，在完整生命周期中充分遵循 PodDisruptionBudgets、节点标签、污点与容忍度等原生原语。 节点生命周期与安全更新流程 NodeWright 主要由 Operator、自定义资源和打包为容器镜像的 Packages 构成。在执行更新时，Operator 会严格按照六步时序执行：先将节点设为不可调度（Cordon），等待标记为不可中断的关键 Pod 自然结束（Wait），依规驱逐其余负载（Drain），应用配置与内核调整（Apply），按需重启或重载服务（Interrupt），最后恢复节点调度（Uncordon）。每个包均内置验证机制，若校验失败将立即终止发布并上报状态；对于扩容的新节点，还可利用污点机制确保只有配置并校验完成的节点才对外承接业务。 大规模分批发布与熔断策略 针对千卡级集群的变更风险，NodeWright 的 DeploymentPolicy 资源引入了基于节点标签的分区（Compartment）概念。用户可为不同分区选择固定步长（Fixed）、线性递增（Linear）或指数扩展（Exponential）的发布策略。系统在每个批次通过成功率阈值控制是否推进，一旦连续失败超标即触发熔断，避免故障在大规模集群中级联蔓延。 生态集成与开源定位 NodeWright 采用 Apache 2.0 许可证开源，定位为底层主机 OS 管理，不替代上层的 NVIDIA GPU Operator 或 Network Operator。该工具可与 NVIDIA AI Cluster Runtime (AICR) 的预验证配置模版协同工作，并与用于基础设施就绪性检查的 NVCRE 以及负责故障监控的 NVSentinel 配合，支持从单节点金丝雀验证到跨云、跨芯片架构（如 Hopper、Blackwell）环境的自动化调优与运维。

Kubernetes manages what runs on your nodes. Managing the nodes themselves is the challenge: kernel settings, system packages, storage layouts, security agents, and the host-level tuning that GPU workloads depend on. Many teams manage this with Ansible playbooks, custom scripts, and manual runbooks. That works until a new cluster comes up in a different region, a kernel upgrade breaks RDMA, or a CVE has to be remediated across the whole fleet this week. GPU infrastructure makes the challenge harder. You cannot simply discard a GPU node and spin up a fresh one. Hardware is scarce, replacements can take hours, and long-running training jobs cannot simply be rescheduled. So the operational question is not how to change a node; it is how to change all of them without killing the training run. Today, that answer is usually a spreadsheet, a maintenance window, and an engineer watching a terminal at 3 a.m. The NVIDIA DSX platform helps you design and operate AI factories , where an entire facility is treated as a single system rather than a collection of machines. Host configuration follows the same logic: the unit of change is the fleet, not the node. NVIDIA DSX OS is the open-source software layer , and NodeWright is the project within it that configures and updates the underlying host operating systems. From scripts to declarative node management DSX OS addresses this challenge through a modular portfolio of open-source projects spanning the AI-ready foundation, resource and workload orchestration, and production AI services. The AI-Ready Foundation defines, configures, validates, and assures accelerated Kubernetes infrastructure. NVIDIA GPU Operator, NVIDIA Network Operator, Topograph , DRA Driver for NVIDIA GPUs , NodeWright, NVIDIA Cluster Readiness Engine (NVCRE), and NVSentinel provide the supporting infrastructure capabilities. NodeWright declaratively configures and safely updates Kubernetes node operating systems without disrupting workloads. It is an open-source, Kubernetes-native package manager for modifying and maintaining host infrastructure at scale. Think of NodeWright as apt or yum for an entire cluster: aware of workloads, aware of disruption budgets, and able to roll changes out progressively across a fleet. NodeWright has run in production at NVIDIA as Skyhook and is available as open source. This post officially introduces the NodeWright name. Why Kubernetes needs its own package manager Excellent tools already exist for configuring machines. Ansible and Puppet have done it for years. They were designed for a world where machines are managed individually, not as part of a cluster that is actively running sensitive workloads. When you need to update a kernel parameter across 200 GPU nodes, the hard part is not running the script. It is doing so without disrupting the training jobs on those nodes. Traditional configuration management is not aware of Kubernetes. It will not cordon a node before making changes, wait for a critical pod to finish, or drain workloads before rebooting. It will not track success or failure inside the cluster, where the rest of your observability already lives. NodeWright closes that gap. It manages the full lifecycle of host-level changes, from installation through configuration, upgrade, and uninstallation. It does so while respecting the Kubernetes primitives your workloads already depend on: PodDisruptionBudgets , node selectors, taints, and tolerations. NodeWright packages are defined as Custom Resources, so they deploy the way everything else in your cluster does: through kubectl, Helm, Argo CD, Flux, or whatever GitOps tooling you already run. How NodeWright works NodeWright has three main components: the operator, custom resources, and packages. The operator is a Kubernetes controller that watches for NodeWright custom resources and manages the lifecycle of changes across your nodes. Packages are container images that carry the actual modifications: scripts, configurations, and binaries. Packages also include verification scripts that surface failures and stop the rollout when a modification is incorrect. apiVersion: nodewright.nvidia.com/v1alpha1 kind: NodeWright metadata: name: gpu-node-tuning spec: nodeSelectors: matchLabels: nodepool: gpu interruptionBudget: percent: 33 podNonInterruptLabels: matchLabels: workload: long-running-training packages: nvidia-tuned: version: 0.9.0 image: ghcr.io/nvidia/nodewright-packages/nvidia-tuned When you apply a NodeWright custom resource, the operator orchestrates a careful sequence on each targeted node. Figure 1 shows the full sequence and how a protected workload pauses it. Figure 1. The operator cordons a node before it makes any changes and uncordons it only after the change completes. A pod labeled as non-interruptible keeps the sequence in the wait stage, so a running training job finishes or moves on before the node is interrupted Cordon. Mark the node as unschedulable so no new workloads land on it. Wait. Let critical workloads finish gracefully. You declare which pods must never be interrupted by label. Drain. Evict remaining pods, honoring PodDisruptionBudgets by default, with configurable drain behavior when the defaults are insufficient. Apply and configure. Run the package: set kernel parameters, install agents, and configure system services. Interrupt. If the change requires it, restart a service or reboot the node. Uncordon. Return the node to the cluster, ready for workloads. This sequence is the difference between losing half a training run to a kernel update and landing the same update across a fleet without workload disruption. NodeWright packages can perform many host-level operations that would normally require root access without recycling nodes. They can set sysctl and GRUB parameters, configure crash dump collection, create logical volumes, install security agents, remediate CVEs, and perform other system-level configuration tasks. NodeWright tracks the state and semantic version of every package on every node, allowing it to distinguish between a fresh installation, an upgrade, and a downgrade. Packages can also declare dependencies, which NodeWright uses to determine the correct execution order. Validation is built into the package lifecycle. Apply, configuration, upgrade, uninstall, and post-interrupt work can each be paired with a check that verifies the expected node state and surfaces failures through Kubernetes. For example, a CVE remediation package can detect whether a vulnerable kernel module remains loaded and mark the package as failed, giving operators immediate visibility into affected nodes rather than leaving them to discover the problem later through workload failures. This same model extends to node readiness as clusters scale. Newly provisioned nodes can otherwise become schedulable before required configuration and tuning have been applied. NodeWright can require new nodes to join the cluster with a Kubernetes taint, complete the required package operations and validation checks, and remove the taint only after the node passes. This creates a controlled path from provisioned to configured to validated to schedulable , ensuring that new capacity does not accept production workloads until it is genuinely ready. Safe rollouts at scale Pushing a change to one node is straightforward. Pushing it to a thousand GPU nodes running production training jobs is a different problem. The NodeWright DeploymentPolicy resource provides progressive rollout strategies that control how changes propagate across a fleet. You define compartments, which are named groups of nodes selected by labels, each with its own disruption budget and rollout strategy. Figure 2 shows the three available strategies. Figure 2. Compartments partition the fleet, and each compartment carries its own strategy and budget. A batch threshold sets the success rate required for the rollout to advance, and a failure threshold stops a compartment when too many consecutive batches fail Fixed. Constant batch size. Update five nodes at a time, every time. Linear. Increase the batch by a fixed delta. Start with 1, then 2, then 3, building confidence as you go. Exponential. Multiply the batch by a growth factor. Start with 1, then 2, then 4, then 8. Fast once you trust it. Each strategy includes a batch threshold, the minimum success percentage required before advancing to the next batch. An optional failure threshold stops that compartment if too many consecutive batches fail. You set the risk tolerance. NodeWright enforces it. This means you can start a fleet-wide kernel update with a single canary node, verify that it is healthy, and let the rollout accelerate automatically. If something goes wrong, the update stops rather than cascading. NodeWright reports errors in its status, marks failed Jobs, and adds a label and condition to each affected node. You can quickly identify and triage the cause through the Kubernetes API. NodeWright packages and NVIDIA AI Cluster Runtime NodeWright functions as a versatile package management platform. Because packages execute operations that require root-level privileges, host modification relies directly on native Kubernetes primitives: fine-grained RBAC controls user permissions, admission controllers validate specifications, and integrated validation checks ensure state consistency at every step. The public package repository provides modular foundational components for running shell commands, managing bind mounts, and establishing kernel crash dump collectors. NVIDIA also publishes packages based on operational knowledge that NVIDIA teams have developed while running GPU clusters at scale. Tuning packages use an intent-based model. Rather than naming a profile, you declare your accelerator and what you are doing with it, such as NVIDIA Blackwell GPUs and multi-node training. The package automatically assembles the right profile: kernel parameters, power management, and system settings. Coverage spans NVIDIA Hopper and Blackwell GPUs, with a generic baseline profile for any NVIDIA GPU and variants for different cloud environments. A dedicated package covers Google Kubernetes Engine (GKE) nodes running Container-Optimized OS, where the usual tuning stack is not available. Node setup packages automate bootstrap steps for specific cloud and accelerator combinations, handling kernel version management and Elastic Fabric Adapter (EFA) driver installation for Amazon Elastic Kubernetes Service (Amazon EKS) clusters with NVIDIA Hopper or Blackwell GPUs. These packages are part of a broader effort. NodeWright integrates with NVIDIA AI Cluster Runtime (AICR). AICR captures known-good combinations of drivers, operators, kernels, and system configurations and publishes them as version-locked recipes. Its component catalog pins both the NodeWright operator and the NodeWright customizations that carry environment-specific tuning, then renders them into deployment-ready bundles for Helm, Argo CD, Flux, or Helmfile. NodeWright applies the host-level parts of those recipes to running nodes. Two companion NVIDIA open-source projects—NVCRE and NVSentinel—complete the ecosystem. NVCRE handles pre-workload validation to verify that accelerated infrastructure is production-ready, whereas NVSentinel monitors for runtime faults and facilitates cordon, drain, and remediation operations. Together with NodeWright, these tools support provisioning, maintenance, and self-healing for GPU-accelerated Kubernetes clusters. To be clear about scope: NodeWright does not replace the NVIDIA GPU Operator or NVIDIA Network Operator. It manages the host OS layer underneath them. Get started NodeWright installs with Helm into any Kubernetes cluster. The chart is distributed as an OCI artifact, so there is no repository to add: helm install nodewright oci://ghcr.io/nvidia/nodewright/charts/nodewright \ --version <chart-version> \ --namespace nodewright \ --create-namespace From there, define a NodeWright Custom Resource with the packages you want applied, target your nodes by label, and the operator handles the rest. NodeWright repository: source, issues, and discussions Packages repository: NVIDIA and community packages NodeWright documentation: architecture, CLI reference, and deployment policies NVIDIA AI Cluster Runtime: the broader validated configuration system Get involved NodeWright is licensed under Apache 2.0 and part of DSX OS, a modular portfolio of open-source projects spanning the AI-ready foundation, resource and workload orchestration, and production AI services. Adopt one project, integrate several, or compose them into a platform. The value lies in open interfaces, independent adoption, and a coherent lifecycle, not in a monolithic stack. Operating this stack at customer scale reveals failure modes early, and NVIDIA teams share those observations with the community. The package repository is where that happens for NodeWright. The NodeWright team is especially interested in packages for hardware and cloud combinations that the catalog does not yet cover and tuning profiles from operators running configurations the team has not yet characterized. Kubernetes transformed how teams manage workloads. The underlying nodes—especially GPU nodes running demanding AI workloads—still rely on scripts and manual runbooks. NodeWright brings the same declarative, automated, safe approach down to the host layer. Adopt what you need. Help shape what comes next. Project references: NodeWright repository and NodeWright packages repository .