NVIDIA Outlines Guidelines for Speculative Decoding Co-Design in LLM Inference

NVIDIA Outlines Guidelines for Speculative Decoding Co-Design in LLM Inference NVIDIA presented five technical guidelines for optimizing speculative decoding in LLM inference, analyzing how draft length and drafting mechanisms interact with GEMM arithmetic intensity, attention tile utilization, and serving latency across the Pareto frontier. Speculative decoding accelerates autoregressive generation without altering output sequences by predicting candidate tokens with a draft model and validating them concurrently on a target model. GEMM throughput benefits from longer draft lengths, enabling linear layers to transition from memory-bound to compute-bound states at significantly reduced batch sizes. Attention decode efficiency peaks when draft lengths align with hardware tile sizes, yielding the formula D = (128 / G) - 1 where G is query heads per KV head. At ultra-low latencies, draft generation latency and sequential kernel launches can negate theoretical speedups unless parallel draft heads like DFlash or DSpark are utilized. Model updates impact drafter acceptance, requiring retraining or realignment for tied representations such as MTP, EAGLE-3, DFlash, and DSpark. Mechanics and Speedup Formulation Speculative decoding utilizes a compact draft model to propose D candidate tokens, which the larger target model verifies simultaneously in a single forward pass. Tokens are accepted sequentially until an initial mismatch, achieving an acceptance length (AL) ranging from 1 to 1 + D. Because the target model produces identical ground-truth tokens unless criteria are explicitly relaxed, sequence accuracy is preserved. Net speedup is governed by the ratio of target sequential generation time across AL tokens to the combined latency of parallel target verification and draft generation overhead. Draft Length Impacts on GEMM and Attention Layers In target linear layers, speculation expands the GEMM-M dimension to M * (1 + D), allowing execution to achieve peak compute-bound throughput at lower concurrency without raising KV cache capacity pressure. For example, testing on a 6144x6144 expert GEMM demonstrated that setting D = 7 reduced the batch size required to become compute-bound by a factor of eight compared to non-speculative execution. Conversely, when attention dominates runtime, speculative tokens reuse existing KV cache, shifting arithmetic intensity to 2 * G * (1 + D). On modern GPUs, kernels hit full hardware utilization near GEMM-M = 128, making D = (128 / G) - 1 optimal. When selecting D values above this baseline, keeping G * (1 + D) as an exact multiple of 128 avoids latency spikes from partially utilized software tiles. Tradeoffs Across the Interactivity Curve At the far right of the Pareto frontier where batch size approaches 1, fixed kernel setup and dispatch costs dominate verification runtime. Autoregressive draft models introduce sequential kernel launch overhead proportional to their depth relative to the target. Consequently, increasing D only provides net acceleration if the resulting AL improvement exceeds the marginal draft overhead. Benchmarking on SPEED-Bench using Qwen 3.5 122B A10B showed external draft models attaining higher AL at D > 3, whereas lighter auxiliary architectures leveled off but maintained lower launch overheads. Selecting and Maintaining Draft Architectures Drafting implementations exhibit distinct cost-performance profiles across training overhead, activation memory, and execution steps. Independent external models demand full forward passes and large KV caches, whereas heads like EAGLE-3, MTP, DFlash, and DSpark reuse target hidden states. Parallel drafters like DFlash and DSpark generate D tokens in a single step, mitigating overhead for shallow target models at batch size 1. If the base target model undergoes fine-tuning, attached draft modules such as EAGLE-3 and DFlash require retraining, MTP necessitates continued co-training or realignment, and external models may require distillation to recover degraded acceptance rates.

NVIDIA提出投机解码协同设计指南：加速LLM推理并权衡草稿机制 NVIDIA发布AI模型协同设计指南，探讨如何在保持输出一致性的前提下，利用投机解码（Speculative Decoding）加速LLM的自回归解码阶段。文章提供了在吞吐量与交互性Pareto前沿上选择最佳草稿长度与草稿机制的五项实用原则。 投机解码通过小草稿模型先行生成多个Token再由目标模型单次并行验证，在不影响输出精度的同时提高算术强度并减少解码迭代次数。 选择最佳草稿长度D需平衡算子受限瓶颈：在线性层中可将GEMM推向计算受限区域，在注意力层中最佳D值受制于分组头数G与GPU软件Tile尺寸（128）。 低延迟场景下内核启动固定开销显著，D的增加必须保证接受长度（AL）的收益能抵消草稿产生的串行延迟开销。 对比了独立草稿模型、EAGLE-3、MTP、DFlash、DSpark及后缀/n-gram等机制在架构、训练成本、内存占用和生成开销上的权衡。 目标模型微调会改变输出分布，依赖目标隐藏状态的挂接型草稿机制或独立草稿模型需要同步重新训练或调整。 投机解码原理与加速公式 投机解码在每轮目标迭代中提出D个草稿Token并由大模型一次性验证，目标模型按序接受Token直至出现首个不匹配，最终产生1到1+D个接受长度（AL）的Token。加速比由目标模型串行生成AL个Token的时间，与并行验证D个Token的时间加上生成草稿的时间（T_draft）之比决定。通过提升目标模型的算术强度，验证过程在内存访问量基本不变的情况下将更多计算推向计算受限区域。 线性层与注意力层的草稿长度权衡 对于目标线性层，草稿使得GEMM-M扩大为原来的1+D倍，在D=7时仅需八分之一的批处理量即可进入计算受限区域，有助于缓解稀疏MoE及长上下文下的KV缓存压力（准则1）。对于长文本与推理任务中占主导的注意力层，其算术强度随草稿增加至2×G×(1+D)。由于当前GPU的注意力内核在GEMM-M=128时利用率最佳，建议初始选择D=(128/G)-1（准则2）；若D进一步增大，应确保G×(1+D)为128的倍数以避免Tile欠载（准则3）。 极低延迟工况下的层数开销约束 在Pareto曲线最右端的极小Batch场景中，内核启动与通信的固定开销占主导，验证耗时对D的变化相对不敏感。但对于自回归草稿模型，端到端延迟受模型层数与串行内核启动次数支配。此时加速比近似为AL/(1+ρD)，其中ρ为草稿与目标模型的深度比。因此在超低延迟环境下，增加草稿长度的前提是AL的增益足以弥补草稿深度带来的延迟负担（准则4）。 主流草稿机制特性对比与微调维护 选择草稿机制需平衡AL、生成开销、显存与训练成本（准则5）。独立草稿模型AL高但显存与前向成本大；MTP集成于基础模型但在低延迟小模型中多步串行开销明显；DFlash与DSpark支持一步并行生成D个Token，显著降低延迟开销，更适配小模型或GPU场景；无模型的n-gram适合高重复文本。此外，当目标模型微调后分布改变，MTP需伴随训练，EAGLE-3/DFlash/DSpark等依赖隐藏状态的机制需要重训适配，而外部模型亦可能需重新蒸馏。

This post is the third in a series on AI model co-design. It explores how to accelerate LLM inference while maintaining accuracy using speculative decoding and offers five guidelines for selecting draft length and draft mechanism across the Pareto frontier. For a discussion of how model design choices impact both throughput and interactivity without sacrificing accuracy, see AI Model Co-Design: Hardware-Friendly LLM Design (Part 1). For an explanation of how group size (query heads per KV head), head dimension, and sequence length shape the performance of dense attention, see Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference (Part 2). What is speculative decoding? Speculative decoding is a technique for accelerating the autoregressive decoding phase of LLM inference by predicting multiple tokens per iteration. A small draft model first predicts several likely next tokens. These tokens are then verified in parallel with a single pass through the larger target model. Figure 1. Speculative decoding moves LLM decoding toward a more compute-bound region, shifting the throughput-interactivity Pareto frontier up and to the right. As interactivity increases toward the right of the curve, batch size decreases, and GEMMs account for a larger share of runtime than attention This approach reduces the total number of decoding iterations while increasing the arithmetic intensity of the target model, without requiring higher concurrency. The target model accepts the proposed tokens in sequence until it encounters the first mismatch. The next prediction cycle then resumes from that position. Because only tokens accepted by the target model are retained, speculative decoding produces the same output sequence as standard decoding, unless the acceptance criteria are deliberately relaxed. Figure 2 illustrates this decoding flow. The draft length (\(D\)) is the number of proposed tokens per target iteration. The acceptance length (\(AL\)) is the number of tokens produced (that is, accepted) per target iteration. \(AL\) ranges from \(1\) to (\(1 + D\)) because the target can always produce one new ground-truth token in addition to the accepted draft tokens. The speedup from speculative decoding can be quantified as the ratio of time taken by the target model to sequentially generate \(AL\) tokens to the time taken to verify D tokens in parallel, while also accounting for the latency overhead of drafting \(D\) tokens: \(\mathrm{speedup} = \frac{T_{\mathrm{verif}}(B) \times \mathrm{AL}}{T_{\mathrm{verif}}\left(B \times (1 + D)\right) + T_{\mathrm{draft}}(B, D)}\) where \(B\) is the batch size, \(T_{\mathrm{verif}}(x)\) is the target verification time for \(x\) tokens, and \(T_{\mathrm{draft}}(b, y)\) is the time needed to generate a draft of length \(y\) for batch size \(b\). It is evident that to maximize speedup, it’s necessary to find the optimal \(\left(D, AL, T_{\mathrm{draft}}\right)\) tuple. Figure 2. The draft-then-verify flow of speculative decoding. A smaller draft model proposes \(D\) tokens, and the target model verifies them together in one pass Selecting the optimal draft length For simplicity, if we ignore draft model latency, speculation provides a speedup when: \(\frac{T_{\mathrm{verif}}\left(B \times (1 + D)\right)}{T_{\mathrm{verif}}(B)} < AL\) During verification, compute scales with (\(1 + D)\), but memory access remains unchanged. Hence, the goal is to increase \(D\) until \(T_{\mathrm{verif}}\) remains constant, typically up to the point where verification transitions from memory bound to compute bound. It is evident that this optimal value of \(D\) depends on \(B\) and is expected to vary across the Pareto frontier. Consider Figure 1 with this in mind. Draft length and performance of linear layers With speculation, the GEMM-\(M\) for each target linear layer GEMM grows from \(M\) to \(M \times (1 + D)\), where \(M\) is the GEMM-\(M\) with no speculation. Figure 3 shows how teraflops per second scale with batch for a representative expert GEMM size of 6144 x 6144 at various draft lengths. It is evident that higher draft lengths allow GEMMs to achieve peak performance at lower effective batch sizes. Notably, with \(D=7\), one-eighth of the batch size is needed to become compute bound compared to \(D=0\). As mixture-of-experts (MoE) models become sparser and long-context workloads increase KV capacity pressure, the effective concurrency per expert decreases, making larger draft lengths attractive across the Pareto frontier. Figure 3. Normalized teraflops per second (against \(D=0\) , batch size=1) versus batch size at different draft lengths for a representative expert GEMM with GEMM-N = 6,144 and GEMM-K = 6,144 Guideline 1: Increase speculative decoding draft length to push GEMMs into the compute-bound region without increasing KV cache capacity pressure. Draft length and attention performance For reasoning and agentic workloads, attention tends to dominate the execution time in the throughput-oriented region. Decode attention has an arithmetic intensity of about \(2 \times G\), where \(G\) is the number of query heads that share one KV head, as explained in Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference . Speculation increases this to \(2 \times G \times (1 + D)\), because the speculated tokens reuse the same KV cache. The effective attention GEMM-\(M\) is \(G \times (1 + D)\). On current GPU devices, attention kernels achieve good hardware utilization at GEMM-\(M=128\), thereby making \(D = \frac{128}{G} – 1\) the optimal draft length. Figure 4 shows normalized attention throughput as \(D\) increases for \(G = 8\) and \(G = 32\) across 32K and 128K KV sequence lengths. The \(G = 32\) variant reaches throughput saturation at a lower value of \(D\). Beyond the throughput saturation point, attention is no longer DRAM bandwidth bound, and its runtime scales with \(D\). Since \(AL\) grows sublinearly with \(D\), increasing \(D\) beyond this point is likely to slow down an attention-heavy workload. Figure 4. Normalized attention teraflops per second (against \(D=0\) ) versus \(D\) for \(G=8\) and \(G=32\) . Larger \(G\) reaches high utilization at a smaller draft length Guideline 2: When attention dominates decode time, choose \(D = \frac{128}{G} – 1\). Attention runtime also depends on tile size. Figure 5 shows that runtime increases in steps as \(G \times (1 + D)\) crosses a multiple of 128, the software tile size for the benchmarked attention kernel. If \(G \times (1 + D)\) falls between two tile boundaries, the last tile is only partially utilized but still costs about as much as a full tile. Figure 5. Normalized attention runtime (against \(D=0\) ) versus \(D\) for \(G = 8\) and \(G = 32\) . Runtime increases in steps as \(G \times (1 + D)\) crosses a multiple of 128 Guideline 3: If you choose \(D > \frac{128}{G} – 1\), prefer values where \(G \times (1 + D)\) is a multiple of 128 to avoid tile underutilization. The relative weight of Guideline 1 compared to Guidelines 2 and 3 depends on the percentage of runtime spent in FFNs compared to attention at the preferred operating point. Communication also adds verification cost as D increases, although compute and communication overlap can mitigate the overhead. Draft length at the far right of the Pareto curve Moving towards the far right of the Pareto curve, \(B\) becomes very small. Here, fixed kernel setup and post-processing overheads dominate for both compute and communication kernels. These fixed costs don’t grow significantly with the number of verified tokens, keeping verification overhead largely invariant with the number of draft tokens. While MoEs do see more experts activated with increasing draft length, a combination of model sharding strategies and efficient kernels such as Grouped GEMMs can keep this overhead low. Larger draft lengths can therefore help in the low latency region, as long as acceptance remains high. At very low latency, the number of sequential kernel launches determines workload latency. Because kernel launches scale linearly with layer count, for an autoregressive draft model with a similar layer structure to the target, the speedup can be approximated as: \(\mathrm{speedup} = \frac{L_{\mathrm{target}} \times \mathrm{AL}}{L_{\mathrm{target}} + D \times L_{\mathrm{draft}}}\) where \(L_{\mathrm{target}}\) and \(L_{\mathrm{draft}}\) are the layer counts in the target and draft models,  respectively. Defining a constant draft_depth_ratio, \(\rho = \frac{L_{\mathrm{draft}}}{L_{\mathrm{target}}}\): \(\mathrm{speedup} = \frac{\mathrm{AL}}{1 + \rho D}\) and draft overhead, \(O_d = \rho D\) In other words, increasing \(D\) helps only while the gain in \(AL\) is large enough to offset the draft overhead. Guideline 4: At very low latency, increase \(D\) only while the gain in \(AL\) justifies the added draft cost. Selecting a draft mechanism Selecting \(D\) tells you how many tokens to speculate. Next, decide how to generate these tokens to maximize speedup. A multitude of techniques have been proposed over the years, with different tradeoffs in training, parameter, and runtime costs. External drafting suggests using a small standalone LLM, while MTP, EAGLE-3, DFlash, and DSpark use auxiliary layers combined with information from the target model to predict tokens. Suffix and n-gram methods are model-free and instead reuse patterns already seen in the token stream. Table 1 compares key drafting methods, including how they generate tokens, their serve-time memory, and their draft overhead. External draft model EAGLE-3 MTP DFlash DSpark Suffix / n-gram Draft architecture Small LLM Decoder layer(s) + linear projection Decoder layer(s) + linear projection Decoder layer(s) + linear KV fusion Decoder layer(s) + linear KV fusion + lightweight Markov head String matching, no model Inputs Token IDs Previous-token embedding + target hidden states Target final hidden state Fused target hidden states used as draft KV Fused target hidden states used as draft KV Token stream Generation method \(D\) autoregressive forward passes Runs the module \(D\) times Runs the module \(D\) times Generates \(D\) tokens in one parallel step Generates \(D\) tokens in one parallel step, followed by a lightweight sequential correction One lookup Release method Separate model release Separate post-training head Part of the base model checkpoint Separate post-training head Separate post-training head Independent of target Training cost From scratch: 1T-10T+ tokens Distilled: 100B-400B Adapted: 10M-1B 1-10B additional tokens after target training Usually trained with the target during pretraining 1-10B additional tokens after target training 1-10B additional tokens after target training None Activation memory cost Draft weights + full KV cache Weights + small KV cache Weights + small KV cache Weights + small KV cache built from target Weights + small KV cache built from target None Speculation cost per \(D\) tokens Full draft model forward 1 layer, serial 1 layer, serial ~5 layers, once ~5 layers once + lightweight Markov head, serial O(1) lookup Best for LPU and LPU+GPU Not recommended for GPUs Not recommended – lower \(AL\) than MTP Best for larger models on GPUs Smaller models, batch size 1 Smaller models, batch size 1 Workloads with high repetition Table 1. Comparison of common draft mechanisms across inputs, generation method, training cost, serve-time memory, and draft cost To quantify the tradeoff between \(AL\) and \(O_d\), begin by observing how \(AL\) scales with \(D\). Figure 6 shows how AL changes as D increases on SPEED-Bench , with Qwen 3.5 122B A10B as the target. SPEED-Bench is a speculative decoding benchmark developed by NVIDIA with the goal of representing realistic production workloads. It covers multiple task domains such as coding and summarization and has a variety of splits at different input sequence lengths. We recommend SPEED-Bench for \(AL\) comparisons. On the 32K split, Qwen 3.5 35B A3B reaches an \(AL\) of 6 at \(D = 9\), while the 4B draft reaches an AL above 5. MTP and DFlash ALs level off as \(D\) grows. N-gram has lower acceptance on this workload and is better suited to workloads with repeated token patterns. Figure 6. \(AL\) versus \(D\) for different draft mechanisms on SPEED-Bench, throughput 32K split, with Qwen 3.5 122B A10B as the target Higher \(AL\) does not equal higher speedup. You also need to consider how much it costs to generate the draft. As shown in Figure 6, all external drafts reach higher \(AL\) than other draft methods at \(D > 3\). The Qwen 3.5 122B MTP is much smaller, with 2.5B total parameters and fewer than 150M active parameters. At \(D = 3\), a larger external draft may not be worth the added cost. At \(D = 21\), its higher AL may make that cost worthwhile, assuming an efficient serving strategy. Meanwhile, DFlash \(AL\)s level off relatively quickly. However, both DFlash and DSpark generate \(D\) draft tokens in parallel, lowering \(O_d\). For comparison, at minimum latency and with \(D=11\), a one-layer MTP head takes 11 steps, leading to \(O_d^{\mathrm{MTP}} = \frac{11}{L_{\mathrm{target}}}\), while a five-layer DFlash head generates the draft in one pass, giving \(O_d^{\mathrm{DFlash}} = \frac{5}{L_{\mathrm{target}}}\). For a large target model with many layers, both overheads are negligible, but as \(L_\mathrm{target}\) decreases, the draft overhead becomes significant. For small models at low latency, therefore, DFlash or DSpark can be the optimal choice even with lower \(AL\). To navigate the tradeoff between \(AL\) and draft latency, it is crucial to benchmark both accurately. In the context of speculative decoding, this means benchmarking with realistic prompts and covering a wide set of task domains. We recommend SPEED-Bench for \(AL\) measurements and high performance inference frameworks such as NVIDIA TensorRT LLM to quantify draft overhead. Beyond inference performance, draft training scope and costs deserve consideration.  While MTP must be co-trained with the target model, EAGLE, DFlash, and DSpark can be added to the final model checkpoint. Similarly, a continuum of options exists for external draft model creation. Training from scratch and distillation from the target yield the highest ALs, while adapting an existing model to the desired target through cross-model adaptation techniques can significantly reduce training costs at the expense of \(AL\). Fine-tuning the target model can change its output distribution and hidden representations. Learned drafters trained for a particular target checkpoint can therefore lose acceptance even when the target model improves. After changing the target, remeasure \(AL\) on representative workloads. The required adaptation depends on the draft mechanism. MTP is part of the target model and should continue to be trained during fine-tuning or be realigned in a dedicated follow-up stage. Target-attached drafters such as EAGLE-3, DFlash, and DSpark use the target’s hidden states and will need to be adapted to the updated checkpoint with additional training. External draft models do not consume target hidden states, but they still approximate the target’s output distribution and may require fine-tuning or distillation. Suffix and n-gram methods have no learned drafter and therefore require no retraining, although their effectiveness still depends on how much repetition appears in the deployed workload. Guideline 5: Choose the draft mechanism that gives the best decode speedup for your workload and hardware, considering \(AL\), draft latency, and training and deployment cost. Get started co-designing speculative decoding Use the five guidelines summarized below as a checklist for selecting \(D\) and draft mechanism across the Pareto frontier. Increase speculative decoding draft length to push GEMMs into a compute-bound region without increasing KV cache capacity pressure. When attention dominates decode time, choose \(D = \frac{128}{G} – 1\) as a starting point. For larger draft lengths, prefer values where \(G \times (1 + D)\) is a multiple of 128 to align with the attention kernel tile size. At very low latency, use a fast draft mechanism and increase \(D\) only while the gain in \(AL\) justifies the added draft cost. Choose the draft mechanism by balancing \(AL\) and draft overhead for your workload and hardware. Benchmark both under realistic serving conditions and consider training and deployment cost. For post-trained drafters, ready-to-run training examples are available in NVIDIA/Model-Optimizer for EAGLE-3, DFlash, and DSpark. See how we did this for NVIDIA Nemotron 3.5 Lightning: fine-tune DSpark , then quantize it to FP8 or NVFP4 . Use the examples as starting points, then validate AL and end-to-end speed on your own model, workload, and hardware. Acknowledgments This post is a cross-team NVIDIA effort. We are grateful to Bhargava Gopireddy, Ritika Borkar, Dor Tsur, Andrii Skliar, Benjamin Chislett, Yaniv Galron, Talor Abramovich, Yoav Miron, Rabeeh Karimi Mahabadi, Roger Waleffe, Udi Karpas, Ran Zilberstein, Brian Pharris, and Eduardo Alvarez for their help.