Dense vs. MoE Models: Architectural Mechanics, Throughput, and Deployment Trade-offs

Dense vs. MoE Models: Architectural Mechanics, Throughput, and Deployment Trade-offs Choosing between dense and Mixture-of-Experts (MoE) architectures depends heavily on deployment constraints, as MoE decouples total memory footprint from per-token compute by activating only a subset of expert parameters. Dense models activate every parameter through a shared feed-forward network, whereas MoE models route tokens through a selected subset of expert blocks per layer. MoE decouples VRAM hosting requirements from compute: total parameters dictate memory usage, while active parameters govern per-token FLOPs and inference speed. Benchmark data illustrates that MoE architectures offer higher throughput at equivalent total parameter scales, though high concurrency can compress their latency margin. Deploying MoE architectures introduces distinct trade-offs, including router sensitivity during quantization and potential expert imbalance during fine-tuning. Architectural Differences and Token Routing Dense architectures pass every token through all model parameters and a single shared feed-forward network (FFN) per decoder layer. MoE architectures replace that shared FFN with multiple expert blocks, employing a learned gate router to direct incoming tokens to top-scoring experts alongside common shared experts. This routing decision occurs independently at every layer based primarily on syntax and token-type patterns, while the tokens still pass through standard attention mechanisms. Architectural variants like Nemotron 3.5 Lightning also integrate Mamba-2 layers, which replace attention in most layers with constant-size recurrent states. Decoupling Memory from Compute The primary operational difference between the two architectures lies in how memory and compute interact. Dense models scale hosting memory and inference compute together, whereas MoE models decouple the two. All expert weights must remain loaded in VRAM, consuming similar memory to an equivalent-size dense model and leaving less room for the KV cache. However, idle experts consume zero compute during inference, drastically reducing the active FLOPs and weight bytes read per token. While MoE models excel at low batch sizes, increasing concurrency engages more experts simultaneously, narrowing the latency advantage over well-optimized dense models. Empirical Performance and Cost Trade-offs Benchmark comparisons show that sparsity yields significant serving throughput gains at identical total parameter scales. At approximately 30 billion total parameters, Nemotron 3.5 Lightning activates 3 billion parameters per token to reach output speeds between 235.7 and 494.2 tokens per second, outpacing dense models like Gemma 4 31B and Qwen3.8-27B. Although Lightning delivers several times the output speed of Qwen3.8-27B at lower cost, it scores lower on general capability metrics, demonstrating that dense models convert memory into higher reasoning capability while MoE converts it into execution volume. Deployment and Adaptation Considerations Selecting an architecture requires evaluating memory allocations, fine-tuning plans, and quantization characteristics. Dense models offer uniform gradient flow and simpler full fine-tuning, whereas full fine-tuning on MoE can unbalance the router unless mitigating strategies like LoRA or router freezing are applied. In quantization workflows, both architectures contain sensitive components; MoE routers are prone to having discrete routing choices disrupted by precision loss, requiring routers, embeddings, and output heads to be held at higher precision, similar to how recurrent projections are maintained in hybrid models.

Dense与MoE模型架构解析：核心差异、性能表现与选型权衡 Dense模型在处理每个token时激活全部参数，而MoE模型则通过路由机制仅激活部分前馈网络专家。两者解耦了显存与计算的关联，在吞吐量、复杂度和任务能力上呈现出不同权衡。 Dense模型每个token激活全部参数，MoE模型则将token路由至少数Top-k专家并跳过其余FFN，且现代MoE常配合共享专家。 MoE将显存占用（取决于总参数）与计算成本（取决于激活参数）解耦，在Batch Size为1时受显存带宽限制较少，吞吐优势显著。 高并发下由于专家综合利用率上升，MoE的延迟优势会收窄，同时全部专家驻留显存会压缩KV Cache可用空间。 选型需综合显存预算、并发延迟、微调复杂度和量化敏感模块（如MoE的路由器网络）。 架构机制与参数运作差异 Dense模型在每个解码层通过单一共享FFN激活所有参数。MoE模型则将共享FFN替换为多个专家网络，由路由器网络为每个token动态分配Top-k个专家，其余专家跳过计算；现代架构如Mistral Small 4通常保留一个全量token访问的共享专家。路由决策在各层独立进行，专家并非按学术学科分化，而是主要专精于语法和token模式（如标点、数字）。激活参数统计包括被选中的FFN权重以及必经的全量注意力与嵌入权重。部分变体如Nemotron 3.5 Lightning还融合了Mamba-2与注意力混合结构，以恒定状态缩减长文本KV Cache。 性能指标与计算-显存权衡 在总参数相近时，MoE跳过部分FFN因而速度更快。其核心特征在于解耦显存与计算：全量专家预先载入显存产生固定开销，计算FLOPs仅随激活专家变动。Batch Size为1时，MoE因单token读取权重较少而极具优势；但在高并发下，各token组合会覆盖大部分专家，导致延迟优势收窄。Artificial Analysis基准显示，同为约30B规模，Nemotron 3.5 Lightning（3B激活）输出速度达235.7至494.2 t/s（每百万token成本0.22美元），显著高于全量激活的Gemma 4 31B（36.9至222.4 t/s，$0.40）与Qwen3.8-27B（46.8 t/s，$3.00），但其综合能力跑分相应偏低。 场景选型与工程考量 模型选型取决于部署限制。显存方面，二者30B规模均需约60 GB显存，Dense将其转化为任务能力，MoE则转化为吞吐量；高吞吐低单次推理复杂度的智能体执行层适于MoE，复杂推理更适于Dense。微调层面，MoE全量微调可能导致路由器失衡或专家休眠，通常需采用LoRA/PEFT或冻结路由器。量化方面，MoE的路由器易因扰动导致路由判断翻转，通常需要将路由器、嵌入与输出头保持在更高精度；混合注意力模型如Qwen3.8-27B则需保留线性注意力块在BF16以保障稳定性。

How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the answer: It uses a Mixture-of-Experts ( MoE ) architecture that selects only a subset of its parameters for each token . There are two dominant model architectures: Dense model and MoE. How a model organizes its parameters matters as much as how many it has. The right choice affects throughput, memory cost, and serving complexity more than raw parameter count does. Therefore, choosing between them comes down to your deployment constraints. Think of the difference like two engines with the same total displacement. One fires every cylinder on every cycle; the other activates only the cylinders it needs. This post explains: How dense and MoE architectures work How they affect performance When to choose each one Dense models vs. MoE models In simple terms, dense and MoE models differ in how they use their parameters. A dense model activates all its parameters for every token. An MoE model stores multiple expert networks but routes each token through only a selected subset. Dense models typically favor simpler, more predictable deployment, while MoE models can deliver greater capacity and throughput when memory and serving complexity are manageable. Video 1. Dense vs MoE: How to choose the right AI architecture How parameters differ In a dense model every parameter participates in every forward pass. All 27B parameters of a 27B model fire for every token, through a single shared feed-forward network (FFN) block per decoder layer. An MoE model replaces that single shared FFN with multiple FFN blocks , also known as experts. Through an internal routing process, incoming tokens are processed through a small subset of experts rather than every single parameter. Structurally, inside each decoder layer where a dense model would have one FFN, an MoE layer has multiple (e.g. 8, 64, or 128). A learned gate network , usually called a router network, sits in front of all of them and assigns each incoming token to the top k scoring experts. The selected FFN blocks are the ones that run for that token while the rest are skipped for that particular layer. Most modern MoEs though such as Mistral Small 4 also run one “shared” expert that every token is routed to regardless. How MoE routing works For MoE models, the routing decision made is separate for every layer, meaning a token doesn’t get routed to an expert and stays there for the rest of the computations. At each decoder layer it gets rerouted based on what the token represents at that point in the network. These experts at every layer aren’t experts in the traditional sense where they specialize in a subject, but rather their specializations lie primarily in syntax and token-type patterns (punctuation, numbers, etc), though this varies by architecture or training methods. While the router mechanism decides which FFN blocks are skipped at each decoding layer, the tokens still pass through the full attention mechanism as normal. When a model card says “3B active parameters,” it includes both the attention and embedding weights for every token along with the selected FFN weights. MoE variants There are variants to MoE as well, one of them being the NVIDIA model card for Nemotron 3.5 Lightning specifying a Mamba-2 + MoE + Attention hybrid . The Mamba-2 layers stand in for attention in most layers, carrying a constant-size recurrent state rather than a growing KV cache. That changes the memory profile at long context in a way sparsity alone doesn’t account for. Rather than Lightning making these routing decisions using the full model’s breadth, it compresses them into a smaller space first to make the routing decision cheaper for the model. Figure 1, below, describes the general transformer-MoE case. Note that an MoE model is a type of sparse model. Figure 1. A dense model and sparse model decoder block with different routing mechanisms ( Source: Medium ) Which is faster: Dense or MoE models? MoE models are often faster for token throughput because they activate only a subset of feed-forward parameters for each token. Dense models activate the full network, but can offer simpler serving and more predictable latency. At high concurrency, routing and memory movement can narrow MoE’s advantage. Results also depend on hardware, precision, inference framework, and model design. For example, Nemotron 3.5 Lightning has Mamba-2 layers and speculative decoding. Key model performance differences The difference in performance between the two comes down to two main things. For one, at equal total parameter count the skipped FFN blocks make MoE faster. The more consequential difference between the two is the fact that MoE decouples memory (total params → VRAM to host) from compute (active params → FLOPs per token). In a dense model hosting and inference costs scale together, and MoE breaks that link. With MoE, VRAM is paid up front. When every expert loads into memory the compute is paid per token and scales only with the experts that fire. Idle experts cost nothing to run and while still paying the required memory to store it, making that shift from variable per-token compute to fixed memory the main tradeoff. At batch size 1 decoding is bound by available memory rather than compute, and this is where MoE performs well since it reads fewer weight bytes per token. As the batch size grows, tokens collectively end up using most of the experts in the network and that advantage narrows while the reduction in work per token persists. MoE holds a throughput advantage across batch sizes, but its latency margin compared to a well-optimized dense model compresses at high concurrency. Modern inference frameworks typically handle routing without discarding tokens, though all experts must live in the GPU memory simultaneously. This leaves less room for the KV cache than a dense model of comparable size would typically have. Example comparison Table 1, below, shows a direct comparison. At equal total parameter size, sparsity shows up plainly in serving throughput: Gemma 4 31B and Nemotron 3.5 Lightning are both ~30B total, yet across NVIDIA GPU providers their output-speed ranges don’t overlap. Activating 3B parameters per token is a major reason, though not the only one. Lightning’s Mamba-2 layers and its speculative decoding contribute independently of the hybrid MoE architecture. Model Architecture Total params Active params VRAM (native) VRAM (4-bit) Artificial Analysis Intelligence Index Output speed‡ $/M output‡ Gemma 4 31B Dense; multimodal 31B 31B ~61 GB BF16 (1×H100) ~16 GB 30 36.9 – 222.4 t/s $0.40 Qwen3.8-27B Dense; hybrid linear/full attention; MTP head; multimodal 27B 27B ~56 GB BF16 (1×H100) ~14 GB 52 46.8 t/s $3.00 Nemotron 3.5 Lightning MoE + Mamba-2 attention hybrid 30B 3B ~60 GB BF16 (1×H100) ~20 GB 24 235.7 – 494.2 t/s $0.22 Mistral Small 4 MoE; multimodal 119B 6B (8B incl. embeddings) ~121 GB FP8 (4×H100) ~71 GB 20 147.3 t/s $0.60 Table 1. Median output speed and cost data from Artificial Analysis (10K-token input, retrieved Aug 31, 2026); benchmarks cover NVIDIA GPU providers only The tradeoffs for both can be observed; for example, Lightning delivers four to five times Qwen3.8-27B’s output speed at a fourteenth of the price, and scores less than half as well on general capability. That is right for an agentic execution layer running well-specified steps at volume, and wrong where one hard reasoning pass decides the outcome. When should you use a dense or MoE model? When it comes to deciding which model to use, it depends on which one will best serve your deployment context. There are several factors to consider: Memory Budget: The memory footprint tracks total parameters rather than active ones, so a 30B MoE and a 30B dense model both need roughly 60 GB. The real question is what you get for those gigabytes: dense converts them into capability, MoE into throughput. Concurrency: MoE wins clearly for a single request. Its throughput advantage holds as concurrency scales, but the latency gap narrows. If you’re latency-sensitive at high concurrency, benchmark both before making a decision. Fine Tuning Plans: Dense model fine-tuning is simpler; all parameters activate, gradients flow uniformly. A full fine-tune on an MoE model can unbalance the router, making some experts more popular than others or completely phasing out certain ones. LoRA/PEFT approaches help avoid this along with simply freezing the router. NeMo’s supervised fine-tuning recipe for Lightning handles it correctly for that model specifically Quantization: The compression ratio isn’t where the architectural difference lives. Two things matter more in practice. First, check what precision the checkpoint already ships at: Mistral Small 4 is natively FP8, so 4-bit buys another ~1.7× (121GB → 71GB), not 4×. Second, both architectures have modules that quantize badly, and they aren’t the same ones. In an MoE it’s the router, where small perturbations flip discrete routing decisions. Recipes hold the router, embeddings and output head higher while in hybrid-attention models it’s the recurrent projections that flip routing decisions. For example, quantized Qwen3.8-27B builds keep the linear-attention block in BF16 for exactly that reason. The core tradeoff Dense and MoE are two answers to one tradeoff: capability per parameter against compute per token. Dense keeps everything simple and everything active, which makes it easier to fine-tune and to serve. MoE buys throughput with memory, and pays for it in serving complexity. Nemotron 3.5 Lightning is fully open with weights, data, and recipes so you can adapt it to your workflows and deploy it anywhere. To get started, try it on build.nvidia.com or through OpenRouter . Download the weights from Hugging Face , and ModelScope . For physical AI workloads, AgiBot GO-1 and Tencent Hy-Embodied-VLM-1.0 are popular options in the ecosystem.