NVIDIA Introduces ModelExpress to Accelerate Model Loading and Lifecycle Across GPU Clusters

NVIDIA Introduces ModelExpress to Accelerate Model Loading and Lifecycle Across GPU Clusters NVIDIA ModelExpress accelerates the distribution of large model weights and runtime artifacts by selecting the fastest data path, prioritizing direct GPU-to-GPU transfers over P2P RDMA to substantially reduce worker startup times. ModelExpress transfers DeepSeek-V4 Pro weights and JIT kernel cache between replicas in under 10 seconds, cutting overall startup time from 8 minutes to 1 minute 44 seconds. Runtime path selection prioritizes P2P RDMA, followed by ModelStreamer, GPUDirect Storage, and default host-staged loading, with safe fallback handling to prevent serving partial weights. NIXL memory registration overhead is mitigated through pool registration—reducing registrations by 80% to 99%—or single-call VMM arena registration. The system extends peer transfers to JIT-compiled kernel caches via CPU-to-CPU RDMA and introduces a receiver-driven refit architecture for RL post-training weight updates. Tiered Bootstrapping and GPU-to-GPU Transfers When a worker initializes without an active peer, ModelExpress bootstraps directly from storage. It can stream safetensors from cloud buckets straight to GPU memory via bounded CPU staging to avoid local disk landing, coordinate cluster-wide downloads so shared caches are populated only once, or leverage GPUDirect Storage for direct disk-to-GPU reads. Once an initial replica is running, subsequent replicas bypass storage entirely: ModelExpress identifies compatible peers using runtime metadata identifiers and transfers post-processed tensors directly between GPUs over P2P RDMA via the NVIDIA Inference Xfer Library. Registration Optimization and Safe Path Fallback To minimize the overhead of individually registering tens of thousands of model tensors for RDMA, ModelExpress offers pool registration, which registers underlying allocations to reduce registration calls by 80 to 99 percent. It also supports VMM arena registration, which directs load-time allocations into a single 16 TiB virtual-address arena registered as a single dmabuf-backed region. At runtime, the system automatically selects the highest available capability in order of priority—P2P RDMA, ModelStreamer, GDS, and the default loader—and reinitializes the model if a failure occurs after transfer begins so partially written weights are never served. Kernel Cache Sharing and RL Post-Training Refits ModelExpress addresses post-load warmup costs where inference engines spend minutes JIT-compiling kernels and capturing CUDA graphs. Through its Artifact Transfer API, compiled caches for tools like Triton, DeepGEMM, and FlashInfer transfer directly between host-memory buffers over CPU-to-CPU RDMA, allowing new replicas to skip warmup without requiring shared Kubernetes ReadWriteMany volumes. For reinforcement learning workflows where policy weights change continuously, ModelExpress provides a four-stage, receiver-driven refit mechanism—publish, discover, plan, and pull—allowing rollout actors to extract updated weights directly from trainer partitions.

英伟达推出 ModelExpress：通过 P2P RDMA 与工件复用加速大模型权重分发 英伟达推出了 NVIDIA ModelExpress（MX），旨在消除大模型在冷启动、自动扩缩容及强化学习后训练中反复搬运模型权重的开销。该方案通过优先采用 GPU 到 GPU 的 P2P RDMA 传输，并结合 JIT 内核缓存复用，将 DeepSeek-V4 Pro 的总启动时间从 8 分钟大幅缩减至 1 分 44 秒。 首个副本通过 Model Streamer 从对象存储流式载入显存或利用 GDS 直读本地存储，避免冗余的磁盘落地与主机内存暂存。 后续副本基于 NIXL 进行 GPU 间 P2P RDMA 传输，并通过池化注册和 16 TiB VMM 虚拟内存 Arena 技术最小化内存注册耗时。 系统引入 Artifact Transfer API 跨节点共享 JIT 编译及自动调优内核缓存，消除耗时数分钟的重复预热过程。 针对强化学习后训练设计了由接收端驱动的“发布-发现-规划-拉取”四阶段权重更新机制，并支持跨集群增量更新。 首节点引导与集群入口优化 在集群尚无运行副本时，首个工作节点必须从存储引导。MX 使用 Model Streamer 将 safetensors 分片并发拉入复用的 CPU 暂存区并流水线送入 GPU，完全绕过本地磁盘。在张量并行模式下，各 Rank 会拆分远程读取并通过 NCCL 共享结果。面对共享磁盘缓存，MX Model Cache Service 通过元数据原子声明将多个并发请求合并为单次外部下载。在具备 GPUDirect Storage（GDS）支持的系统中，NIXL 可将权重直接从本地存储读入 GPU 显存，无需主机内存中转。 副本间 P2P RDMA 扇出与内存注册优化 当首个副本上线后，后续兼容副本直接将其作为活跃源。MX 控制面依据模型与运行时配置计算 mx_source_id，通过 Redis 或 Kubernetes 元数据发现匹配节点，数据面则由 NIXL 驱动 P2P RDMA 传输。为解决大模型数万个张量逐一注册导致的耗时问题，MX 提供了将分配整合的池化注册（削减 80% 至 99% 的注册调用），以及借助 CUDAPluggableAllocator 汇聚至 16 TiB 虚拟地址空间的 VMM Arena 注册，将注册合并为单次 dmabuf 内存区域调用。 路径降级策略与编译内核工件继承 MX 按照“P2P RDMA -> ModelStreamer -> GDS -> 默认加载器”的优先级动态探测并执行回退。如果在写入开始后发生故障，系统会重新初始化模型以杜绝部分损坏的权重。此外，针对 Triton、DeepGEMM 和 FlashInfer 等耗时巨大的 JIT 内核编译与 CUDA Graph 捕获，MX 提供了 Artifact Transfer API，通过 CPU 间 RDMA 直接传输编译缓存，使新副本能够直接继承预热成果，避免依赖 Kubernetes 的 RWX 存储卷。 强化学习后训练与生态集成 在 RL 后训练中，策略权重频繁迭代。MX 采用接收端驱动的流程，通过“发布、发现、规划、拉取与转换”四个步骤，让 Rollout 推理工作节点直接从训练节点单向拉取并重构分布式权重分片。目前 MX 已原生集成 vLLM 与 SGLang，并支持 Dynamo、llm-d 等服务框架，团队正联合社区持续推进更深度的 TensorRT-LLM 适配与跨集群增量同步能力。

Every byte moved has a cost. As model checkpoints grow to hundreds of gigabytes or even a terabyte, that cost adds up quickly. To make things even worse, moving these model weights around the cluster is extremely common. For instance, a cold start may pull weights from remote storage into GPU memory; autoscaling and rolling updates must populate each new replica; and RL post-training continuously moves updated weights from trainers to roll out workers. These may look like different workflows, but they impose the same recurring tax: time spent moving weights before useful work can begin. ModelExpress: Accelerating the model weight lifecycle NVIDIA ModelExpress (MX) is built around a simple idea: Before loading a model, first ask where a compatible copy of its weights already lives. Rather than treating every replica as an independent cold start, MX chooses the fastest available source and transfer path. When a serving peer already holds compatible weights in GPU, MX transfers them directly from GPU to GPU over P2P RDMA via NVIDIA Inference Xfer Library ( NIXL ), bypassing redundant access to object storage, local disk, and host memory. When no peer is available, MX bootstraps from the fastest supported path by streaming from an object store without landing on disk or reading local files directly into GPU memory. MX transfers DeepSeek-V4 Pro weights and JIT Kernel cache artifacts from a serving replica into a fresh replica in under 10 seconds, reducing the total startup time to 1 minute 44 seconds from 8 minutes . The rest of the post shows how MX selects the fastest available path to GPU memory, prioritizing P2P RDMA from a serving replica and eliminating redundant downloads and copies along the way. It then extends the same approach to reusing kernel caches and distributing RL weight updates. Figure 1. Overview of ModelExpress Accelerating every stage from remote storage to GPU memory Every new worker must get its weights from one of three places: remote storages (e.g. HF or S3), local storage, or another worker already serving the model. For the first worker, there is no peer yet, so it must bootstrap from storage. MX can stream the checkpoints from object storage or load it from fast local storage, removing avoidable copies along either path. Once that first worker is serving, the preferred source changes. Its weights are already resident, post-processed, and laid out in GPU memory, so every compatible worker after it should load directly from that peer over P2P RDMA. MX makes this transition automatically: bootstrap once from storage, then scale out GPU to GPU, falling back to storage only when no compatible peer is available. Starting the first worker: Bootstrap from storage Remote object storage to GPU: Avoiding local disk When the checkpoint lives in a cloud bucket and you would rather not provision and manage a disk cache tier, MX uses the Model Streamer to pull safetensors through a reusable CPU staging buffer and into GPU. The checkpoint never lands on local disk, eliminating the intermediate download, reload, and storage volume. Model Streamer uses a multithreaded tensor reader to fetch tensor ranges concurrently across checkpoint shards. As tensors arrive, it pipelines remote reads with GPU placement: completed tensors are passed to the inference engine while later tensors are still being fetched. This keeps the storage, network, and GPU copy paths busy while reusing a bounded amount of host memory. In tensor-parallel deployments, the participating ranks divide the remote reads and share the results, typically over NCCL, instead of having every rank download the full checkpoint independently. MX connects this distributed stream directly to the inference engine’s weight loader, preparing the first worker to become the P2P source for every compatible replica that follows. Cluster ingress: Download once, not N times When a cluster maintains a shared disk cache tier (e.g. persistent volumes in K8s), MX ensures that the fleet populates it only once. If 10 replicas concurrently want to fetch the 806 GiB DeepSeek-V4 Pro model, they will need to pull roughly 8 TiB of identical data across the network while competing for the same ingress bandwidth. The MX Model Cache Service collapses those requests into one coordinated download: an atomic claim in Metadata Store selects a downloader, while the remaining replicas track its progress and reuse the cached copy. The cluster pays the external download cost once, then every replica can begin from the same cached checkpoint. Local storage to GPU: Bypassing host-memory staging When GPUDirect Storage (GDS) is supported in the system, MX reads checkpoint files directly from local storage into GPU memory through NIXL’s multithreaded GDS backend. NIXL executes batched tensor reads in parallel directly into GPU memory, bypassing host memory and the staging copy required by a conventional loader. Users don’t need to enable GDS explicitly: MX detects the capability automatically and falls back to another loading strategy when it is unavailable. Local storage to GPU: Pipelining local reads with ModelStreamer MX can also load local checkpoints through ModelStreamer. Multiple OS threads read safetensors concurrently into a configurable CPU buffer while completed tensors move to the GPU and later reads continue in parallel. Unlike GDS, this path still stages through host memory, but it overlaps disk I/O with GPU placement, benefits from the OS page cache, and provides a portable fast path when direct storage-to-GPU access is unavailable. Starting every worker after the first: Fetch from a serving peer This is the key feature of MX. Once another replica is already serving the same model, the weights have completed most of their journey: they are resident in GPU memory, post-processed, and laid out for the inference engine. MX treats that replica as a live weight source. After confirming compatibility, it transfers the tensors directly from the source GPU to the target GPU. Once its weights are loaded, the new replica joins the source pool, giving subsequent replicas another peer to load from. With every successful transfer, that pool grows alongside the deployment, turning scale-out into GPU-to-GPU fan-out instead of repeated cold loads. The MX control plane discovers compatible peers, exchanges transfer metadata, and tracks source readiness, but never handles the weight bytes themselves. On the data plane, MX uses NIXL as a default transfer engine whose pluggable backends allow for peak performance across a variety of networks, such as Infiniband, RoCE, NVLink, EFA, etc. MX has a first-class transport interface that allows libraries such as fabric-lib and standalone Mooncake to integrate with MX. Figure 2. Peer-to-peer GPUDirect RDMA weight transfer via NIXL Before any transfer begins, MX computes an mx_source_id from the model and runtime settings that determine tensor layout, then considers only peers with a matching ID. The control plane discovers those peers through Redis, Kubernetes CRDs, or k8s-service (serverless) metadata backends. Optimizing NIXL memory registration overhead Before NIXL can RDMA a tensor, the GPU memory backing it has to be registered: an ibv_reg_mr call that returns the Remote Key (rkey) used for remote access. A large model has tens of thousands of tensors, and registering them one at a time is slow enough to show up in the budget. By default, MX registers each tensor individually. Two opt-in strategies reduce that registration cost: Pool registration registers each underlying cudaMalloc allocation once instead of each tensor, cutting registration count by 80 to 99 percent on typical models with no change to transfer semantics. VMM arena registration goes further. It installs a CUDAPluggableAllocator that routes every load-time allocation into a single 16 TiB virtual-address arena, then registers the whole used range as one dmabuf-backed memory region at end of load. Registration collapses from one call per tensor to one call, total; each tensor descriptor simply carries an offset into that single region. Using DeepSeek-V4-Pro TP=8 on the vLLM engine, as shown in Figure 3, below, we measured the average NIXL registration time for each approach. Figure 3. NIXL memory registration optimization Runtime path selection and safe fallback At startup, MX probes the available capabilities, automatically skipping any path the environment does not support. The first applicable strategy runs in the current priority order: P2P RDMA -> ModelStreamer -> GDS -> default loader (host-staged POSIX I/O) . If a path is unavailable or fails before modifying the model state, MX falls through automatically. If a failure occurs after weights have begun landing, it reinitializes the model before continuing, so partially written weights are never served. P2P retries alternate peers only for metadata failures before transfer begins, and the native loader remains the final fallback. This capability-driven design keeps the MX core hardware and software agnostic, with platform-specific fast paths enabled only where supported. End-to-end results We ran DeepSeek-V4-Pro on an 8xB200 GPU node with NVIDIA ConnectX-7 NICs and compared the total model loading time across different cold start scenarios. Each replica used vLLM 0.23.0 with TP=8 and --enable-flashinfer-autotune . See Figure 4, below. Figure 4. End-to-end cold start model loading time comparing HF vs ModelStreamer (S3) vs Disk vs P2P RDMA Warm, not just loaded: Inheriting the compiled kernels Getting weights into GPU memory is critical, but a loaded model is not yet ready to serve. During its first forward passes, the engine JIT-compiles and autotunes kernels (e.g. torch.compile, Triton, DeepGEMM, TileLang, and etc.) and captures CUDA graphs for the exact model, dtype, quantization, and GPU. For models such as DeepSeek-V4 Pro, this can take several minutes and can become the dominant startup cost once MX reduces weight-loading latency (see Figure 5, below). Figure 5. Startup time breakdown of DeepSeek-V4 Pro (TP=8 vLLM) That repeated warmup is avoidable. When the model, software stack, and GPU architecture match, one replica can pay the compilation cost and the rest can inherit the resulting caches. MX’s Artifact Transfer API packages these file-backed artifacts, transfers them directly between registered host-memory buffers over NIXL’s CPU-to-CPU RDMA path, then verifies and installs them in the target engine’s cache directory. This eliminates the need for a shared ReadWriteMany (RWX) volume in Kubernetes, while an artifact-specific mx_source_id prevents reuse across incompatible replicas. MX detects standard cache locations automatically when a Redis or Kubernetes metadata backend is configured. We ran using the same setup to measure how much the kernel artifact transfer can reduce the startup time. The artifact-enabled run transferred the Triton/DeepGEMM/TileLang/CuTe DSL/FlashInfer caches. The chart compares the major startup stages and total wall-clock time from process start until the API was ready. Figure 6. Total startup time reduction with ModelExpress When the weights change every Step: RL post-training Everything so far assumes a model’s weights are fixed once loaded. RL post-training breaks that assumption. A trainer updates the policy every step, and the inference actors generating rollouts must pick up those weights before the next round of generation. As with inference startup, weight movement is on the critical path in RL: rollout workers wait while updated weights move from the trainer’s distributed layout (whether FSDP/DTensor shards or Megatron TP, PP, and EP partitions) into the inference engine’s layout. Figure 7. ModelExpress makes RL refit receiver-driven MX drives the refit through the following four stages: Publish: Each trainer rank advertises the tensors or shards it already owns, together with metadata describing their shape, dtype, placement, and parameter mapping to MX. Discover: A rollout worker looks up the requested weight version and its available sources through MX. Plan: The receiver maps the published ownership information onto its own target layout and identifies which sources contain the required tensors or ranges. Pull, convert, and load: The receiver issues one-sided reads directly against those sources. MX includes the core building blocks for receiver-driven refit, and customers are evaluating them in active integrations. We are also testing delta weight diff refits for cross-cluster weight transfer, a technique used by Fireworks/Cursor, Cognition, and more in recent RL runs. Contributing to Dynamo and our roadmap MX has native integrations with vLLM and SGLang and supports serving frameworks including Dynamo and llm-d. The Dynamo open source community is actively working toward deeper TensorRT-LLM integration and broader inference capabilities. Explore the current Dynamo documentation and roadmap , try the available workflows in your own environment, and contribute feedback to help shape the project’s direction. Acknowledgments ModelExpress is a team effort. Thank you to the rest of the MX team, Zhongdongming Dai and Tanushriya Singh for their core work on the project. We’re grateful to Itay Neeman, Anish Maddipoti, Istvan Haller, and Omri Kahalon for their guidance on the project’s technical direction, and Will Eaton at Red Hat for his support on the llm-d integration.