NVIDIA Dynamo Introduces Shadow Engine Recovery to Slash LLM Inference Restart Times

NVIDIA Dynamo Introduces Shadow Engine Recovery to Slash LLM Inference Restart Times NVIDIA has introduced shadow engine recovery as a preview feature in NVIDIA Dynamo, cutting worker recovery times from minutes to seconds during software-level inference failures. The approach leverages a dedicated GPU Memory Service and pre-warmed idle standby engines to avoid duplicate weight storage and eliminate cold restart initialization delays. Cold restarts take minutes because model weights, CUDA contexts, and communicator graphs are dropped upon process failure and must be reloaded and recomputed. The GPU Memory Service operates as a per-GPU sidecar using the CUDA Virtual Memory Management API, allowing physical memory pages to persist across process failures and be shared across engines. A parked shadow engine maintains CUDA contexts, communicator channels, and weight mappings while deferring KV cache allocation, avoiding extra memory overhead for model weights. In a GLM-5.2 benchmark on NVIDIA B200 nodes, shadow recovery reduced worker return time from 283 seconds to 7.3 seconds, mitigating severe TTFT and decode rate degradation. The initial preview primarily supports vLLM on Kubernetes 1.34+ with Dynamic Resource Allocation, while handling for hardware crashes and persistent KV caching remain in progress. The Bottlenecks of Cold LLM Restarts Standard inference engine crashes trigger cold restarts requiring model weights to be reloaded from storage, kernels compiled, and CUDA graphs recaptured. GPU memory is normally bound to an engine's process and CUDA context, so a process termination prompts the driver to release resident memory. Additionally, states such as NCCL communicators and virtual-address-dependent CUDA graphs cannot be transferred between processes, leaving surviving nodes to handle shifted traffic during lengthy re-initializations. Architecture: GPU Memory Service and Standby Engines Shadow engine recovery resolves recovery delays by separating memory lifetimes from process runtimes and precomputing non-transferable state. A GPU Memory Service (GMS) sidecar manages physical GPU memory allocations via the CUDA Virtual Memory Management API. Because allocations survive process exit through GMS references, active and dormant engines map the same physical weight pages into separate virtual addresses without duplicate memory costs. Standby shadow engines precompute CUDA contexts, communicators, and graphs, remaining parked without materializing a KV cache until promoted. Pod Integration and Failover Sequence Each worker pod pairs two engine containers with a GMS sidecar and a POSIX flock mechanism on a shared file. During steady state, the active engine serves traffic while the shadow waits on the lock. If the active engine terminates or is terminated by a Kubernetes liveness probe, the operating system releases the file descriptor lock. The shadow engine immediately acquires the lock, remaps weights via GMS, materializes its KV cache, and registers with the router, while the orchestrator restarts the failed engine to become the new standby. Performance Evaluation on GLM-5.2 In a two-worker GLM-5.2 deployment on NVIDIA B200 nodes, deliberately terminating one worker resulted in a 283-second cold restart baseline, during which surviving workers suffered elevated TTFT and lower decode rates. Shadow engine recovery brought a second worker online in 7.3 seconds—comprising 1.7 seconds for failure detection and 5.6 seconds for promotion. This reduced p50 TTFT during the fault from 23,815 ms to 1,311 ms and maintained a p50 decode rate of 46 tokens per second per user compared to 12 in the baseline. Requirements and Technical Limitations The preview feature addresses software faults, such as process crashes and transient collective failures, but does not cover physical hardware, node, or multi-node failures. The implementation requires Kubernetes 1.34 or newer with Dynamic Resource Allocation (DRA) and the NVIDIA GPU DRA driver enabled, with vLLM as the primary supported backend. Because promoted engines currently initialize with empty KV caches, work is ongoing to enable GMS-backed KV cache handover to mitigate post-cutover TTFT spikes.

NVIDIA Dynamo推出Shadow Engine恢复机制：秒级恢复LLM推理算力 NVIDIA Dynamo上线预览特性Shadow Engine恢复，通过GPU内存服务（GMS）实现多引擎共享权重，并以常驻同卡、预初始化的影子引擎替代传统冷重启，将故障恢复时间从数百秒大幅缩短至数秒。 传统LLM引擎故障冷重启耗时数分钟，主因是权重绑定于进程CUDA上下文且NCCL通信与CUDA图等状态不可跨进程继承。 GMS作为独立Sidecar通过CUDA虚拟内存管理API持有物理显存，实现同卡多进程共享权重且跨进程持久化，影子引擎无需额外显存复制权重。 影子引擎预先完成初始化并休眠，发生故障时通过POSIX flock文件锁迅速接管，在GLM-5.2测试中将恢复时间从283秒降至7.3秒。 当前特性支持vLLM等后端，但要求Kubernetes 1.34+与DRA，且暂不支持硬件级故障容灾，KV缓存跨进程继承功能仍在开发中。 故障恢复瓶颈与架构原理 在生产环境中，进程崩溃、可恢复的CUDA错误等常导致引擎失效。传统恢复必须重新经历权重加载、内核编译和CUDA图捕获，导致幸存节点承受流量冲击。Dynamo通过解耦权重生命周期与预置初始化状态解决此问题：GMS（GPU Memory Service）独立持有物理显存，使权重在引擎崩溃后依然驻留并供新引擎映射；同卡部署的影子引擎预先建立通信器并捕获CUDA图，平时仅占用极小显存并不物化KV缓存，处于待机状态。 切换流程与实测性能 每个Worker Pod内含两个引擎容器与GMS，通过POSIX flock进行选主。活动引擎崩溃后，内核释放文件锁，影子引擎迅速获取锁并重新映射权重、物化KV缓存并向路由注册；原引擎重启后则转为新的影子引擎。在双节点GLM-5.2（B200 GPU、TP=8）测试中，故意杀掉一个Worker后，冷重启需283秒恢复并导致TTFT与解码速率大幅恶化；影子引擎方案仅需7.3秒即可重新提供服务，故障后p50 TTFT降至1,311毫秒，有效避免了SLA违约。 部署要求与未来规划 该方案针对常见软件进程级故障设计，不覆盖硬件故障或节点断电。目前主要支持vLLM，要求集群运行Kubernetes 1.34或更高版本并启用动态资源分配（DRA）及相关GPU驱动。当前版本影子引擎切换后需重新积累KV缓存，团队正在开发直接继承原引擎KV缓存与前缀缓存索引的功能，以进一步平滑切换期间的延迟波动。

When an LLM engine process fails, the standard recovery path involves a cold restart. This requires loading weights into HBM from storage, compiling kernels, and capturing NVIDIA CUDA graphs. For large models, initialization can take several minutes, during which surviving workers must absorb the displaced traffic. Shadow engine recovery, available as a preview feature in NVIDIA Dynamo, moves most of this recovery work off the serving path. It keeps a fully initialized shadow engine idle on the same GPUs as the active engine. The GPU Memory Service (GMS) shares the existing weights between the engines without creating another copy in HBM. If the active process fails, the shadow takes over within seconds. Re-initialization occurs in the background entirely off the serving path. We measured the impact by deliberately terminating one worker in a two-worker GLM-5.2 deployment. Without shadow engine recovery, the remaining worker served all incoming traffic during the 283-second cold restart, increasing TTFT and reducing per-user decode rate throughout the outage. With shadow engine recovery, a second worker resumed serving in 7.3 seconds, nearly 39 times faster, minimizing disruption to service quality. Figure 1. Time until a second worker resumes serving after one of two workers fails. A cold restart reloads weights, sizes the KV cache, autotunes, and recaptures CUDA graphs. A preinitialized shadow engine can begin serving much sooner Why LLM inference recovery is slow: two core problems Production LLM engines commonly experience recoverable software faults, including process crashes, recoverable CUDA errors, and transient collective failures. In these cases, the hardware, drivers, and node remain healthy; only the process holding the corrupted state is lost, and a replacement engine can typically start on the same GPUs. So why can’t that fresh engine skip the initialization cost? Two problems stand in the way: Weights are tied to the engine process. GPU memory is linked to the engine’s CUDA context, which is itself tied to the engine process. When the process exits, the driver releases all resources, including weights already resident in GPU memory. Consequently, a replacement engine process must repeat the full weight-loading procedure. Some initialization states are non-transferable. NCCL and torch.distributed communicators bind to the specific running process, and CUDA graphs are fixed to the virtual addresses present during capture. These states can’t be handed off from a previous engine and must be recreated during every restart. Shadow engine recovery addresses each problem with a targeted optimization: decoupling weight lifetime from the engine process, and completing non-transferable initialization before a failure. How shadow engine recovery works Shadow engine recovery combines persistent GPU memory, a pre-warmed standby engine, and worker-level coordination to recover without a cold restart. GPU Memory Service: Persistent GPU memory for LLM inference The GPU Memory Service (GMS) manages specific memory regions, such as weights, independently of the engine process. By using a process distinct from the engine to own these regions, weights remain resident in memory even as engines are restarted. As a result, a new engine on the same GPU can attach to existing memory. GMS is a per-GPU sidecar that owns physical GPU memory on behalf of inference engines. It is mostly dormant and has no CUDA context of its own; it allocates physical pages, hands out handles to them, and arbitrates which engines may read or write at any time. Engines connect, import handles, and map the underlying pages at virtual addresses in their own CUDA contexts. Mapping happens once, at startup, and GMS is not involved in any subsequent access. This functionality is built on the CUDA Virtual Memory Management API. With this API, physical GPU memory and its associated virtual addresses can have independent lifetimes. Since physical allocations are reference-counted, they survive as long as any process maintains a mapping. Two engines mapping the same weight tensor access the same physical bytes, each using virtual addresses local to its context. A kernel reading a weight dereferences an ordinary pointer into the same HBM the weight would have occupied anyway, so a GMS-backed read costs no more than an engine-allocated one. Figure 2. Each engine maps the weights into its own address space, but there is only one physical copy in HBM. GMS holds the allocation and hands out the handles; it does not sit between an engine and the memory it reads from This architecture provides two benefits. First, weights persist beyond engine failure. While the kernel removes the failed engine’s CUDA context, the GMS reference ensures the physical pages stay resident so a fresh engine can immediately map them. Second, weights can be shared between concurrent engines ; therefore, a secondary engine on the same GPU incurs zero marginal weight cost. Integrating GMS into inference frameworks requires only a narrow change. vLLM, SGLang, and NVIDIA TensorRT-LLM each integrate GMS through a custom torch.cuda.CUDAPluggableAllocator bound to the weight memory pool. From inside the engine, weights remain ordinary torch.Tensor s. Adopting GMS is as simple as flipping a flag at startup. GMS is not limited to weights. The current preview does not support using GMS for the KV cache, but that capability is under active development. The goal is for a promoted shadow to map the outgoing engine’s cache instead of rebuilding it as traffic arrives. Shadow engines: preinitialized standbys with zero marginal weight cost A shadow engine is a fully initialized engine process that remains idle and co-resident on the same GPUs as the active engine. Weight sharing makes this configuration feasible: without it, the second engine would require another full copy of the weights, substantially reducing the memory available for processing requests. A shadow runs through the same startup path as an active engine. On each of its GPUs, it connects to the local GMS and imports the weight mappings, establishes communicators (NCCL and NIXL for KV transfer between workers), captures CUDA graphs, and performs any necessary warm-up. By the end of the startup, it is ready to serve. Then, instead of serving, it parks: it releases the materializable parts of its memory and blocks, waiting for its turn. What a shadow has precomputed before it parks: CUDA context, captured graphs, and communicators. These non-transferable components are ready when the shadow engine is activated because they cannot be inherited from another process. Weight mappings. The GMS handles are already imported, so waking the shadow requires only remapping them to virtual addresses established during initialization. What it has deferred: KV cache materialization. The KV cache is the largest reclaimable allocation held by an engine. The shadow reserves its address range without physical backing while parked and materializes the cache when promoted. A parked shadow therefore retains only its CUDA context, captured graphs, communicators, and weight mappings—no separate copy of the weights and no KV cache. This footprint is small enough for the shadow to remain alongside an active engine on the same devices, enabling recovery within seconds. The worker: a single deployable unit The following figure shows how these components fit within each worker and scale behind a shared router. Figure 3. A fleet of workers behind a single router. The two-engine layout is internal to each worker, so the router, frontend, and orchestrator need no changes to benefit from it These foundations are integrated into one pod. The worker holds two engine containers, a GMS sidecar to mediate GPU memory access, and a shared lock to elect the active engine. At steady state, one engine holds the lock and remains awake, connected to GMS, holding a materialized KV cache, and registered with the frontend router. The other remains fully initialized and connected to GMS but is dormant, holds no KV cache, and waits on the lock. Recovery in depth The following sections trace the recovery sequence and explain the synchronization and memory-management mechanisms that make it reliable. Sequence A worker moves through four phases before returning to steady state. Figure 4. The four phases of a recovery. Active and shadow roles swap between engine A and engine B, and the worker returns to steady state without either engine reloading weights T₀ Steady. Engine A holds the lock and is awake, registered with the router. Engine B is dormant, blocked on the lock. T₁ Failure. Engine A’s process exits, either because it crashed outright or because a liveness probe found it hung and killed it. Either way, the kernel releases its lock as the process is reaped. The worker is briefly unroutable, until the shadow registers. T₂ Cutover. Engine B acquires the lock, wakes, remaps weights through GMS, materializes its KV cache, and re-registers with the router. Engine A’s container is restarted by the orchestrator. T₃ Restarted. Engine A finishes initialization and enters the shadow state. The system returns to a steady state, with the roles swapped. The shadow’s advantage is that it enters T₂ already initialized. The only work on the critical path is acquiring the lock, remapping weights, and materializing the KV cache. Synchronization The worker requires both mutual exclusion, ensuring only one engine is awake at a time, and reliable release to ensure the standby engine takes over if the active one fails. A POSIX flock on a shared file provides these guarantees. When the active process exits due to a shutdown, segfault, or SIGKILL , the kernel reaps its file descriptors, and the shadow engine acquires the lock to begin serving. Each engine’s startup path is therefore a short leader election: await engine.initialize() # weight load, torch.compile, autotune, CUDA graph capture ... # put the engine to sleep while we wait on the lock await engine.sleep() lock = FlockFailoverLock(lock_path) await lock.acquire(engine_id=engine.id) # wait on the lock to wake await engine.wake() A deadlocked engine whose process is still alive falls to the Kubernetes liveness probe, which cascades to a SIGKILL and trips the same kernel-managed release. Memory accounting Fitting two engine processes on one GPU without exhausting HBM takes careful accounting across the lifecycle. Figure 5. GPU memory across a recovery Weights. Allocated once by GMS and mapped read-only by every engine in the worker; never duplicated. KV cache. Held only by the active engine today: materialized when it wakes, released when it dies, freeing the region for the shadow to take over. Buffers and graphs. NCCL buffers, the CUDA context, and captured graphs. Held by each engine even while dormant, and the whole of a parked shadow’s standing cost. Benchmark results: shadow engine recovery vs. cold restart on GLM-5.2 To quantify the benefit, we compared shadow engine recovery with a cold restart after an engine failure in a two-worker fleet. Setup We ran two workers serving GLM-5.2 quantized to NVFP4 on NVIDIA B200 nodes: one worker per node, TP=8, 200K max context, and an FP8 KV cache. A single frontend distributes requests round-robin across the two. The load is synthetic: 32,000 input tokens and 1,000 output tokens per request, arriving at 0.7 requests per second. Both arms run identical engine builds and configurations; the only difference is the shadow engine. The baseline has it off, and the killed worker cold-restarts. In the Shadow Engine Recovery configuration, each worker pod hosts a preinitialized shadow engine that can take over if the active engine fails. We injected the fault only after the workload reached its steady-state operating point: a SIGKILL to one of the two workers, followed by 600 seconds of observation. With two workers in the fleet, losing one leaves the survivor carrying every request until its partner returns. Results Figure 6. Time to first token, 60-second rolling median. The baseline climbs for the whole 283 seconds its second worker is missing, crossing the 5-second line about 90 seconds in. With shadow engine recovery, p50 TTFT is more resilient to disruption Figure 7. Inter-token latency, 60-second rolling median. The baseline steps to 84 milliseconds and holds flat for 283 seconds, well above the 50-millisecond line. The shadow arm rises briefly at the fault and settles back Metric Baseline: cold restart Shadow engine recovery Time until a second worker serves again 283 s 7.3 s TTFT p50, after the fault 23,815 ms 1,311 ms Decode rate p50, after the fault 12 tok/s/user 46 tok/s/user Requests over 5s to first token 201 of 399 1 of 398 Requests below 20 tok/s/user 226 of 399 0 of 398 Table 1. Cold restart versus shadow engine recovery in the window after one of two workers is killed Compared to the cold restart baseline of 283 seconds, shadow engine recovery only takes 7.3 seconds (1.7 seconds to detect the fault and 5.6 seconds to promote the shadow). This results in significantly better TTFT and decode rate after the fault, and allows us to largely avoid SLA violations. Current scope and next steps Having validated that fast recovery for inference workloads on Kubernetes is feasible, we are working to stabilize the implementation and expand support to a wider range of workloads. Shadow engine recovery will roll out incrementally over the coming months. The current preview has several limitations and deployment requirements. Shadow engine recovery addresses common engine process failures but does not cover hardware, node, or multi-node failures, which still rely on standard rescheduling. It requires Dynamic Resource Allocation (DRA) on Kubernetes, so the cluster needs Kubernetes 1.34 or newer with DRA enabled and the NVIDIA GPU DRA driver installed. Dynamo Snapshot can be composed with the recovery feature to minimize contention during the initialization of the shadow while serving. Because promoted shadows start with empty KV caches, the post-cutover TTFT experiences a slight bump. Carrying cache state across a promotion, both the prefix-cache index and the cache memory itself, is the active line of work. vLLM is the primary supported backend. To try Shadow Engine Recovery, start with the Kubernetes quickstart to create a running deployment. Then follow the Shadow Engine Recovery deployment workflow and use the vLLM failover example as a complete manifest. Visit the ai-dynamo/dynamo repository to ask questions, report issues, or contribute.