NVIDIA Dynamo Evaluates Encode-Prefill-Decode Disaggregation for Multimodal Inference

NVIDIA Dynamo Evaluates Encode-Prefill-Decode Disaggregation for Multimodal Inference Encode-prefill-decode (EPD) disaggregation optimizes multimodal AI serving by detaching the vision encoder from prefill and decode stages, improving time to first token and system goodput under media-heavy workloads, smaller or quantized language models, and mixed traffic. EPD disaggregation can deliver up to a 5x improvement in time to first token (TTFT) and 7x faster end-to-end response times under suitable workload profiles. Homogeneous clusters benefit most from colocated encoder setups sharing GPU compute, while heterogeneous clusters excel by offloading encoders to lower-cost GPU tiers. Long output sequences degrade EPD end-to-end gains because decode time increasingly dominates overall latency. Quantizing language model weights to NVFP4 enhances EPD effectiveness by increasing the proportion of runtime spent on vision encoding. Decoupling the encoder resolves head-of-line blocking in mixed traffic, cutting mean text TTFT by 42.2% alongside image TTFT improvements. Architecture and Placement Topologies In aggregated multimodal serving, vision encoding, prefill, and decoding share a single GPU worker and scheduling domain, causing visual token processing to delay subsequent steps. NVIDIA Dynamo implements EPD by separating vision encoder execution from prefill-decode (PD) workers, passing embeddings via the NVIDIA Inference Transfer Library (NIXL). Deployments typically take two disaggregated forms: colocated encoders, which share the same GPU with PD workers under separate request queues to prevent GPU underutilization on homogeneous clusters, and disaggregated encoders, which offload vision processing to lower-tier GPUs (such as RTX 6000D) while reserving high-end GPUs (such as GB200) for prefill and decode. Impact of Media Load and Output Sequence Length Tests evaluated using Qwen3.5 122B NVFP4 demonstrate that as image volume increases from 5 to 50 images, aggregated serving TTFT rises steeply, whereas EPD maintains flat TTFT latencies. In an image-heavy benchmark of 10 images at output sequence length (OSL) 1024, colocated EPD reduced TTFT by 58%, while heterogeneous EPD improved goodput by 70% under an inter-token latency SLO of 100 ms. However, increasing OSL up to 2,048 diminishes end-to-end latency gains because decode operations increasingly dominate total runtime, causing colocated EPD to experience a 2.5% regression due to GPU contention at high OSL with low image counts. Influence of Model Size and Quantization Because vision transformer (ViT) compute requirements remain relatively static while language model workloads scale with size, EPD benefits are higher for smaller models. Tested across Qwen3.5 4B, 9B, and 27B, colocated EPD goodput dropped from 2.62x down to 0.65x relative to aggregated serving, falling below break-even on the 27B model where language model costs dominate. Conversely, quantizing the language model's active weights to NVFP4 while keeping the encoder in BF16 raised EPD goodput gains over aggregated serving from 1.78x to 2.64x, as faster LLM prefill and decode shifted a larger proportion of total request time back onto the encoder. Mixed Modality Traffic and Pipeline Enhancements Under sustained 50:50 mixed text and image traffic at OSL 128, aggregated serving forces text requests to wait behind multimodal preprocessing and ViT forward passes. Disaggregating encoders eliminated this head-of-line blocking, reducing mean text TTFT by 42.2% (from 92.3 ms to 53.3 ms) and image TTFT by 30.8% (from 289.9 ms to 200.6 ms). To address the 43% of TTFT consumed before ViT execution, Dynamo also supports front-end parallel media decoding—which cut mean encoder latency by 26% in testing—alongside CPU DRAM embedding caches for repeated media and multimodal KV-aware routing.

NVIDIA解析多模态大模型EPD解耦推理架构与适用场景 NVIDIA在博客中探讨了多模态AI模型推理的编码-预填充-解码（EPD）解耦优化技术。该技术通过开源推理框架NVIDIA Dynamo将视觉编码器与LLM预填充、解码阶段拆分，最高可将首字生成时间（TTFT）降低5倍，并在特定场景大幅提升吞吐，但在轻图像或长输出场景下可能降低性能。 EPD解耦将多模态推理中的视觉编码阶段与LLM预填充和解码分离，借由NIXL库传输图像嵌入，使各阶段可独立调度、批处理与扩缩容。 硬件部署拓扑分为聚合部署、同构GPU协同部署（Colocated）以及异构分层部署（如RTX 6000D处理编码、GB200处理预填充与解码）。 高图像负载、短至中等输出长度、小尺寸或量化（如NVFP4）以及MoE模型能从EPD中获得最大收益；长输出与轻媒体负载下收益减退甚至倒退。 在文本与多模态混合流量中，EPD消除了多模态预处理对纯文本请求的队头阻塞，使纯文本平均TTFT降低42.2%。 针对图像预处理延迟与重复请求，Dynamo还提供了前端并行媒体解码、嵌入缓存及多模态KV感知路由等优化手段。 EPD解耦工作原理与硬件拓扑 传统聚合服务中，视觉Transformer（ViT）编码与LLM的预填充、解码共用GPU和调度域。当输入多图或视频时，视觉编码耗时可达数百毫秒并拖累整体执行。Dynamo通过EPD将编码器Worker与预填充解码（PD）Worker解耦，并通过NIXL传输视觉嵌入。在拓扑选择上，同构集群更适合在单GPU上同时运行独立队列的编码器与PD Worker（协同部署），避免整卡独占导致编码器算力闲置；异构集群则可采用分离部署，使用低成本GPU（如RTX 6000D）处理轻量编码，将高端GPU（如GB200）留给计算密集的LLM。 输入负载与输出长度对性能的影响 测试显示，随着输入图片数量增加（5至50张），聚合部署的TTFT大幅增加，而EPD架构的TTFT几乎保持平稳。在10张图片的测试中，协同与异构EPD的TTFT分别下降58%与50%，异构部署在相同SLO（ITL小于100毫秒）下的有效吞吐提升70%。然而，当输出序列长度（OSL）从128增加到2048时，解码耗时逐渐主导端到端延迟，EPD带来的端到端收益逐步缩窄；在协同部署下，面对轻图像负载与长输出时，GPU资源争用甚至导致端到端延迟出现2.5%的反向衰退。 模型规模、精度与混合流量收益 EPD的收益高度依赖ViT占总运算时间的比例。在Qwen3.5 4B、9B和27B的模型消融测试中，随着LLM参数增加，ViT参数占比降低，协同EPD的吞吐收益从2.62倍降至0.65倍（27B出现负收益）。而在将活跃LM权重从BF16量化至NVFP4时，LLM耗时缩短使ViT占比相对上升，EPD相对聚合服务的吞吐优势从1.78倍扩大到2.64倍。此外，在50:50的图文混合流量测试中，EPD使纯文本请求无需等待图像编码即可开始预填充，文本平均TTFT从92.3毫秒降低42.2%至53.3毫秒，图像TTFT也下降了30.8%。 配套补充优化手段 基准数据表明，TTFT中约有43%的耗时发生在ViT启动前的媒体下载与解码阶段，EPD本身无法优化该部分。为此，Dynamo提供了前端并行媒体解码（启用后平均编码请求延迟可下降26%）、基于CPU内存的嵌入缓存（避免重复媒体多次编码）以及结合图像哈希的多模态KV感知路由，从全链路提升多模态服务性能。

Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill and decode stages. It is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts (MoE) models. This post shows when and how to use EPD disaggregation with NVIDIA Dynamo to achieve up to 5x faster time to first token (TTFT) and 7x faster end-to-end response time. The post also addresses the scenarios where EPD disaggregation is not recommended. Dynamo is an open source inference framework for serving AI models in distributed environments. It supports EPD disaggregation, separating these phases into independently scalable stages rather than running them together with tightly coupled scheduling and scaling patterns. Dedicated encoder workers can improve batching, memory efficiency, and overall throughput. Why is EPD disaggregation important? A multimodal request adds work before LLM prefill can begin. The serving stack must preprocess the media and run the vision transformer (ViT) to produce embeddings. In aggregated serving, vision encoding, LLM prefill, and decode all share one worker and one scheduling domain. This simple design works well when media processing accounts for only a small portion of the workload. Figure 1. TTFT breakdown across single-image, multi-image, and video requests, separated into media preprocessing, ViT forward and projection, and LLM prefill The balance changes as requests contain more images or video. Vision encoding can take hundreds of milliseconds or longer. Because encoder and LLM work share the same GPU, a media-heavy request can delay its own prefill and contend with concurrent prefill and decode work. In contrast, under mixed traffic, text-only requests may also wait behind multimodal requests even though they do not require vision encoding. Dynamo implements EPD serving by separating the encoder and PD worker roles without fixing their hardware placement. Encoder workers produce vision embeddings, while PD workers consume those embeddings and run the LLM. This separation allows the stages to batch, schedule, and scale independently. Figure 2 compares aggregated and EPD serving, with encode workers passing embeddings to PD workers through NVIDIA Inference Transfer Library (NIXL) . Figure 2. Aggregated and EPD request flows in Dynamo. EPD separates encode execution from the PD workers and transfers embeddings through NIXL Encode disaggregation can reduce TTFT and increase same-SLO goodput only in certain scenarios. The encode work being isolated must be large enough to offset worker coordination and embedding transfer overhead. Encode workers may share the PD GPUs or run on a separate GPU tier, leading to different placement options. Here, we explore the different placement options for the encode worker and the benefits they yield under various scenarios. Three encode placement topologies Figure 3 compares aggregated serving with the two encoder disaggregated topologies, colocated and disaggregated: Aggregated: Each GPU runs a single aggregated worker whose scheduler manages vision encoding, LLM prefill, and decode as part of the same request lifecycle. Colocated encoder : Each GPU runs one or more encoder workers alongside one PD worker. This allows the workers to share GPU compute while maintaining separate request queues and batching. On a homogeneous cluster, Colocated Encoder is generally the better fit. The vision encoder is lightweight relative to the LLM, so reserving an entire same-class GPU for encoder work can leave much of that GPU underutilized. Colocation separates the workers without dedicating a GPU exclusively to the encoder. Disaggregated encoder : Disaggregated Encoder becomes attractive when the cluster includes a lower cost GPU tier better matched to encoder work while the primary GPU tier hosts the PD workers. Dynamo transfers the resulting vision embeddings to the PD tier through NIXL. In our test environment, two NVIDIA RTX 6000D GPUs ran the encoder workers, while four NVIDIA GB200 GPUs ran the PD workers. This placement keeps the lighter encoder workload on the RTX GPUs and reserves the GB200 GPUs for the more compute- and memory-intensive LLM workload. Note that this analysis does not consider the disaggregated setup with homogenous GPUs as it always underperforms the colocated encoder setup. Hardware availability determines where the encoder can run and workload characteristics determine whether separating it pays off. Figure 3. Three encode-placement options: aggregated serving, co-located encoder and PD workers, and disaggregated encoder with a separate lower cost GPU tier for the encoder workers What factors determine the benefits of EPD disaggregation? The benefits of EPD depend on how work is divided among vision encoding, LLM prefill, and decode. EPD is most useful when vision encoding accounts for a significant share of request processing time or limits throughput. Media-heavy requests often create this condition, but media load alone does not determine the result. Output length, model size and precision, and traffic mix also change this balance. Factor Why it matters for EPD When does EPD yield gains Input media load Heavier media load brings more visual tokens, which means more encoder work. EPD allows scaling up encoders, preventing encoding bottleneck. Multiple images, high-resolution images, or video input that yields more visual tokens. Output sequence length (OSL) Longer OSL shifts total latency toward decode. TTFT gains from EPD hold, but end-to-end (E2E) gains shrink. Short OSL: E2E gains hold Long OSL: E2E gains erode Model size / precision ViT compute is largely fixed, while LLM compute decreases with fewer active parameters and lower precision. Small, MoE, and quantized LLMs therefore have higher ViT-to-LLM compute ratios. Smaller, MoE, and lower-precision models benefit more from EPD, while large dense models see less gain. Mixed traffic (text + multimodal) Mixed prefill batches can make text requests wait for ViT. EPD isolates encoder work, allowing text requests to skip waiting for encoder. Heavy multimodal traffic mixed with latency sensitive text requests. Table 1. How various factors affect EPD performance Test environment All benchmarks were run with Qwen3.5 122B A10B NVFP4 except in the precision ablation experiments. We used four GB200 GPUs (and extra RTX 6000D GPUs in disaggregated setup): Aggregated : One TP1 aggregated worker per GB200 Co-located EPD : Two encoder workers plus one PD worker per GB200 Disaggregated EPD : RTX nodes as the encoder tier, GB200 as PD We used NIXL over UCX RC/TCP Ethernet for vision embedding transfer, measured peak 20 Gbps. We used the Dynamo front end with front end parallel media decoding enabled. The goodput SLO is inter-token latency (ITL) under 100 ms. Image-heavy workload example We compared the aggregated versus encoder disaggregated serving with the following request: Ten images per request (capped 256 token per image) at OSL 1024 to simulate moderately heavy visual workload with a long output sequence. Figure 4 shows the results. Figure 4. Mean TTFT and end-to-end latency across topologies for an image-heavy workload (10 images/request) (left). Goodput improvement over the aggregated setup given an SLO of ITL under 100 ms (right) TTFT drops 58% with colocated encoder and 50% with heterogeneous. End-to-end improvements are modest because OSL 1024 means the model generates 1024 tokens and decode time is not reduced by Encoder disaggregation. The larger signal is goodput: the heterogeneous tier serves 70% more traffic at the same latency SLO, because encoder capacity was added without touching the GB200 budget. How do image load and OSL impact performance gains? We analyzed the impact on TTFT and end-to-end latency in a range of experiments with 5-50 images, with each image using 128/256 token budget, keeping the OSL constant. The performance, both TTFT and end-to-end latency, degrades in the aggregated encoder scenario, while the performance is relatively constant for both disaggregated topologies. Figure 5. Mean TTFT versus visual-token load (5 to 50 images, OSL 512). Aggregated latency rises steeply with visual tokens (image count times token per image) image count; both EPD topologies stay nearly flat Next, we varied OSL from 128 to 2,048 while holding the image load fixed at five images. TTFT remains largely unchanged as OSL increases, but decode time increasingly dominates end-to-end latency. Consequently, heterogeneous EPD end-to-end gain over aggregated serving narrows from 20.3% to 5.2%. Co-located encoder shifts from an 11.8% gain to a 2.5% regression because its additional encoder workers share the same GPUs as the PD workers. As OSL increases and decode pressure grows, GPU contention increasingly offsets, and eventually outweighs, the benefit of encoder disaggregation. Figure 6. Mean end-to-end latency versus OSL. The end-to-end gap between EPD and aggregated narrows as OSL grows because decode time increasingly dominates Figures 7 and 8 show percentage improvements in TTFT and end-to-end latency over the aggregated (baseline) across a range of OSL and image load. Encoder disaggregation delivers the most value with high input media load in almost all scenarios. In scenarios with high OSL and low image load, disaggregating the encoder compromises performance. Figure 7. Input image load (number of images times visual tokens). TTFT improvement over aggregated (Qwen 3.5 122B NVFP4). Colocated EPD (top) and heterogeneous EPD (bottom). Both topologies deliver 25-93% TTFT improvement across the full grid. The heaviest cell (50 x 128 images, any OSL) reaches 92-93%. Gains are consistent across the OSL dimension because TTFT is not affected by output length Figure 8. End-to-end request latency improvement over aggregated (same model and hardware). Gains are strong at high image load and short OSL, shrinking toward the upper-left corner (light media, long outputs). Colocated EPD shows slight end-to-end regression at 5 x 128 images / OSL 2048 (-2.5%), where embedding transfer overhead is not recovered by encoder capacity savings How do model size and precision change EPD gains? As model size grows, the LLM accounts for more of the serving workload, while the vision encoder remains roughly the same size. To quantify how this affects EPD, we ran a model-size ablation across Qwen3.5 4B, Qwen3.5 9B, and Qwen3.5 27B. The ViT parameter share decreases from 7.2% at 4B to 4.7% at 9B and 1.7% at 27B. Colocated EPD follows the same trend, delivering 2.62x, 1.50x, and 0.65x the goodput of aggregated serving, respectively. Figure 9. Same-SLO goodput gain versus aggregated serving across model configurations. EPD gain decreases with model size and falls below break-even at 27B. Lower-precision and MoE models with less activated parameters show larger gains Parameter share is only a proxy for the more direct factor: runtime balance among ViT forward, LLM prefill, and decode. EPD provides more benefit when ViT forward accounts for a meaningful portion of request processing time. As prefill and decode become dominant, there is less encoder work to optimize. In our 27B configuration, the LM costs outweighed the benefit of separating the encoder under a colocated setup. We measured how LLM precision affects EPD gain. When both the vision encoder and the LLM used BF16, colocated EPD delivered 1.78x the goodput of aggregated serving. Quantizing only the active LM weights to NVFP4 increased this gain to 2.64x. Figure 10. Effect of LLM quantization on EPD goodput. Bars show colocated EPD goodput relative to aggregated serving. Quantizing the active LM weights from BF16 to NVFP4 increased the ratio from 1.78x to 2.64x. The vision encoder remained in BF16 in both configurations NVFP4 accelerates LLM prefill and decode while ViT forward remains in BF16, shifting a larger share of request time to encoder work. This provides EPD with more room to schedule encoder workload independently from PD workers. How EPD helps mixed modality requests Production workloads often mix text-only and multimodal requests, and the two request types may be scheduled together for prefill. With aggregated serving, one worker handles both vision encoding and LLM prefill. If a batch includes a multimodal request, the worker completes media preprocessing and ViT forward before starting prefill. Text-only requests do not need this work, but can still wait for it, increasing their TTFT. EPD removes this dependency by running vision encoding on separate encoder workers. The front end routes both request types to a prefill worker, but only multimodal requests with missing embeddings invoke the encoder pool. Text-only requests can proceed to prefill while the encoder processes multimodal requests independently, reducing head-of-line blocking between the two request types. To measure this, we compared aggregated serving with co-located encoder under sustained mixed text and image traffic. Figure 11. Mean TTFT for text and image requests under sustained 50:50 mixed traffic at OSL 128, comparing aggregated serving with co-located encoder Encoder disaggregation reduced mean TTFT for text requests from 92.3 to 53.3 ms, a 42.2% reduction. Image-request TTFT fell from 289.9 to 200.6 ms, a 30.8% reduction. Image requests still incur encoder latency, whereas text requests can begin LLM prefill without waiting for vision encoding to complete. This accounts for the larger relative improvement in text-request TTFT. How to select the best encoder topology The best encoder placement depends on where request time is spent. EPD provides the most value when vision encoding represents a significant part of request processing, such as with media-heavy inputs, short to medium outputs, smaller or quantized models, and mixed traffics. Aggregate serving remains a good fit when media processing is light or long decode sequences dominate the latency. Hardware also affects the placement decision. On homogeneous clusters, co-located encoder separates encoder scheduling and batching without reserving an entire GPU for the relatively lightweight vision model. A separate encoder tier is more useful with heterogeneous hardware, where lower-tier GPUs can run the encoder while higher-end GPUs remain dedicated to PD. Note that vLLM and SGLang have roadmaps for further development of their EPD stacks. Get started with EPD disaggregation To reproduce the experiments featured in this post, follow the ai-dynamo/dynamo GitHub guide. As shown in Figure 1, 43% of TTFT was spent before ViT even started. EPD does not help optimize those stages of the pipeline. Try the other levers listed below in Dynamo to address different stages of your stack with it. Parallel media decoding moves download and decode out of the worker and into the Dynamo front end, which ships decoded pixels to the backend. In an encoder-only benchmark at 30×256 on one GB200 with two encode workers, it cut mean encoder request latency 26%, from 281.3 ms to 207.0 ms, and P99 23%, from 752.1 ms to 581.7 ms, at matched throughput. Enable it with --frontend-decoding . Embedding cache stores and offloads computed embeddings in CPU DRAM so repeated media is not re-encoded. When the same multimodal content, such as an image or video, appears in multiple requests, the cached embedding is reused instead of running the vision encoder again. Multimodal KV routing hashes media content alongside text in the Dynamo KV-aware router, so requests sharing media land on workers already holding the relevant KV blocks. Without it, every image looks like the same placeholder token to the router. Acknowledgments We’d like to thank Roger Wang, co-founder of Inferact, for his feedback on this post, and the NVIDIA team, Alexandre Milesi, Ayush Agarwal, Guan Luo, Indrajit Bhosale, J Wyman, Kris Hung, Krishnan Prashanth, Qi Wang, and Zhongdao Ren for their core work on multimodality support in Dynamo.