NVIDIA NIM 2.0.12 Delivers 2.5x Higher Throughput on Nemotron 3 Ultra for Agentic Workloads

NVIDIA NIM 2.0.12 Delivers 2.5x Higher Throughput on Nemotron 3 Ultra for Agentic Workloads NVIDIA has released optimized NIM serving configurations for Nemotron 3 Ultra, achieving up to 2.5 times higher output throughput compared to an unoptimized baseline on a four-GPU B200 system for agentic AI workloads. In a benchmark of an agentic workload with 76% KV reuse at a target of 50 tokens per second per user, NIM 2.0.12 achieved 1,997 tokens per second across four B200 GPUs, compared to 718 tokens per second on the unoptimized baseline. The 2.5x performance boost is achieved through autotuned kernels, tensor and expert-aware parallelism, prefix and Mamba state-cache reuse, scheduler and memory tuning, and MTP speculative decoding. Developers can validate performance targets by profiling representative traffic via NVIDIA AIPerf sweeps and deploying preconfigured model profiles using Docker. Performance Engineering and Benchmark Results NVIDIA NIM packages validated configurations and container lifecycles to balance throughput and latency targets in production. Testing Nemotron 3 Ultra on a 4xB200 system under an agentic workload—characterized by a 64K context, 400 generation length, 76% KV reuse, and a 50 TPS/user (20 ms inter-token latency) target—demonstrated a jump in output throughput from a 718 tok/s baseline (NIM Off) to 1,997 tok/s (NIM On 2.0.12). Interacting Full-Stack Optimization Layers The performance enhancements rely on multi-layered serving optimizations rather than isolated switches. Key layers include autotuned mixture-of-experts and Mamba kernels mapped to NVIDIA Blackwell GPUs, tensor parallelism across four GPUs with expert-aware execution, and state reuse via prefix caching and tuned Mamba state caches. Furthermore, fine-tuning concurrent sequence limits, batch sizes, and GPU memory allocations works alongside MTP speculative decoding to maintain higher system utilization without violating latency constraints. Workload Profiling and Deployment Because published curves serve as a starting baseline, NVIDIA recommends benchmarking specific traffic profiles using NVIDIA AIPerf with Mooncake-format JSONL traces across concurrency sweeps. Users can deploy the containerized Nemotron 3 Ultra NIM (version 2.0.12) from NGC using Docker, leveraging specific profiles such as `vllm-nvidia-b200-nvfp4-tp4-pp1-throughput-90.0` with speculative decoding enabled.

NVIDIA 全栈 NIM 优化助力 Nemotron 3 Ultra 提升 2.5 倍并发用户吞吐量 NVIDIA 介绍了如何通过全栈 NIM 2.0.12 微服务优化 Nemotron 3 Ultra 模型推理。在 4 块 B200 GPU 的 Agent AI 工作负载基准测试中，优化后的服务栈在保持响应延迟的同时，将输出吞吐量提升至开源基线的 2.5 倍。 在 4xB200 硬件及 50 TPS/用户（20ms ITL）的 Agent 负载测试下，NIM 2.0.12 的输出吞吐量达到 1,997 tok/s，是无 NIM 优化基线（718 tok/s）的 2.5 倍 性能提升来自多层协同优化，涵盖自调优 MoE/Mamba 内核、张量并行与专家感知执行、前缀与状态复用、调度与显存调优，以及 MTP 投机解码 NIM 还提供企业级容器生命周期支持，包括定期推理栈更新、CVE 处理、硬件验证与商业支持 用户可通过 NGC 容器镜像部署 Nemotron 3 Ultra，并利用 NVIDIA AIPerf 重放真实流量以确定满足 SLO 的 Pareto 最佳平衡点 性能表现与基准测试 针对长上下文和频繁流式输出的 Agent AI 场景（基准设定为 64K 输入、400 输出、76% KV 复用），在 4 块 Blackwell B200 GPU 上，未经优化的开源服务栈输出吞吐量为 718 tok/s，而启用包含 MTP 等优化的 NIM 2.0.12 后吞吐量升至 1,997 tok/s，达到了 2.5 倍的并发用户服务能力。 NIM 2.0.12 核心优化技术层 测试提升由多重配置组合实现，而非单点开关叠加：包括适配混合架构的自调优 MoE 与 Mamba 内核；跨 4 张 GPU 的张量并行和专家感知执行；前缀缓存、部分前缀匹配与 Mamba 状态缓存复用；对并发序列、批处理 Token 限制与显存分配的调度调优；以及增配的 MTP 投机解码及其修复机制。 部署与工作负载评估流程 开发者可从 NGC 下载 Nemotron 3 Ultra 2.0.12 镜像，选择针对 B200 优化的 Profile 并开启投机解码运行容器。官方建议使用 Mooncake 格式的 JSONL 跟踪数据结合 NVIDIA AIPerf 工具进行多并发梯度压测，从而根据应用自身的 SLO 和延迟要求选取最佳部署点。

Deploying a large language model is only the first step toward production-ready serving. Production teams also need to serve as many concurrent users as possible on available GPU infrastructure while preserving the interactivity that keeps applications responsive. That tradeoff matters even more for agentic AI workloads, where prompts can be long, context can be reused across steps, and applications often stream extended responses back to users. NVIDIA NIM packages model- and GPU-aware serving choices into a deployable microservice. Instead of starting from a blank runtime configuration, developers get a validated serving configuration and a supported deployment path, while retaining the ability to benchmark the NIM against their own traffic. ## What NIM adds: Performance engineering and production readiness Inference performance is a system property. Precision and kernels, parallelism, scheduling, batching, memory allocation, prefix reuse, model-specific state caches, and decoding strategy all interact. A configuration is useful only if it improves throughput while staying within the application latency target. NIM turns this optimization work into a tested starting point. NVIDIA engineers validate configurations for supported model, GPU, and precision combinations, then package the runtime and model artifacts behind standard APIs. For production deployments, NIM Certified adds regular inference-stack updates, CVE handling, broader hardware validation, and commercial support through NVIDIA AI Enterprise. NIM delivers two benefits in one deployment path: validated performance engineering plus an enterprise-ready container lifecycle and support model. ## Case study: Nemotron 3 Ultra NIM delivers up to 2.5x higher throughput for agentic workloads The benchmark definition used throughout this article is: - Hardware: 4xB200 - Agentic workload: 64K/400/76% KV reuse/50 TPS/user (20 ms ITL) This Pareto chart compares the open-source baseline serving stack (NIM Off) against the fully optimized NIM 2.0.12 serving stack (NIM On). | Configuration | Native 256K max context | What it represents | |---|---|---| | NIM Off baseline | 718 tok/s | No NIM optimizations | | NIM On (2.0.12) optimized serving stack | 1,997 tok/s 2.5x v. Baseline | Optimized NIM stack: cache/state reuse, MTP speculative decoding and associated fixes, autotuned kernels, partial-prefix matching, scheduler, batching, memory, and parallelism tuning | *Table 1.* *System output-token throughput across four B200 GPUs at the 50 TPS/User target* ## How NIM 2.0.12 optimized serving stack works The measured gains come from interacting configuration bundles, not independent switches whose percentages can simply be added. The main optimization layers are: - **Precision and autotuned model-aware kernels.** Autotuned mixture-of-experts and Mamba kernels map the hybrid architecture efficiently to NVIDIA Blackwell GPUs. - **Parallel execution.** Tensor parallelism distributes the model across four GPUs, while expert-aware execution improves utilization for the mixture-of-experts layers. - **Prefix and model-state reuse.** Prefix caching avoids recomputing repeated context, partial-prefix matching recovers reuse when only part of a prefix matches, and Mamba state-cache settings are tuned for the model architecture. - **Scheduler, batching, and memory tuning.** Concurrent-sequence limits, batched-token limits, block size, and GPU-memory allocation keep more work in flight without crossing the latency target. - **MTP speculative decoding.** NIM 2.0.12 optimized serving stack (including MTP) adds MTP and its associated fixes to the same optimized serving stack. The incremental benefit depends on acceptance rate and available memory headroom. ## Benchmark the NIM on your own workload The published curves are a starting point, not a promise that every application will see the same result. The fastest way to determine fit is to replay representative traffic and build a Pareto curve for the latency metric that matters to your users. - **Deploy the exact software versions.** Use NIM 2.0.12 (or newer version), and pin the image tag or digest for every run. - **Prepare representative traffic.** Use a Mooncake-format JSONL trace or capture controlled NIM requests, with appropriate access controls and sanitization for sensitive data. - **Measure performance.** Use NVIDIA AIPerf to replay representative traffic and get perf benchmarks - **Select the Pareto point that meets the SLO.** Compare output throughput among points that satisfy the SLO/latency constraints and determine fit for deployment: ``` for C in 1 4 8 16 32 64; do aiperf profile \ --model nvidia/nemotron-3-ultra-550b-a55b \ --endpoint-type chat --streaming \ --url localhost:8000 \ --input-file ./agentic-trace.jsonl \ --custom-dataset-type mooncake_trace \ --no-fixed-schedule \ --concurrency "$C" done ``` **Example AIPerf concurrency sweep.** Replace the trace, request count, and endpoint details with the workload you want to model. ## Download and run Nemotron 3 Ultra NIM Start from the [Nemotron 3 Ultra NIM page](https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/nemotron-3-ultra-550b-a55b/latest/tags), accept the governing terms, and select the NIM 2.0.12 tag or the exact published digest. After downloading, find and select a profile. The following lists all profiles packaged in the NIM: ``` export NGC_API_KEY=<your-personal-api-key> export LOCAL_NIM_CACHE=$HOME/.cache/nim export NIM_TAG=2.0.12 mkdir -p "$LOCAL_NIM_CACHE" echo "$NGC_API_KEY" | docker login nvcr.io \ --username '$oauthtoken' --password-stdin docker run --gpus all --shm-size=16GB \ -e NGC_API_KEY \ -e NIM_MODEL_PROFILE \ -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \ -p 8000:8000 \ nvcr.io/nim/nvidia/nemotron-3-ultra-550b-a55b:$NIM_TAG list-model-profiles ``` To pick an optimized profile for agentic workloads on a four-GPU B200 system, you would set the `NIM_MODEL_PROFILE` to `vllm-nvidia-b200-nvfp4-tp4-pp1-throughput-90.0` and enable speculative decoding: ``` export NGC_API_KEY=<your-personal-api-key> export LOCAL_NIM_CACHE=$HOME/.cache/nim export NIM_TAG=2.0.12 export NIM_MODEL_PROFILE=vllm-nvidia-b200-nvfp4-tp4-pp1-throughput-90.0 mkdir -p "$LOCAL_NIM_CACHE" echo "$NGC_API_KEY" | docker login nvcr.io \ --username '$oauthtoken' --password-stdin docker run --gpus all --shm-size=16GB \ -e NGC_API_KEY \ -e NIM_MODEL_PROFILE \ -e NIM_SPECDEC_ENABLE=1 \ -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \ -p 8000:8000 \ nvcr.io/nim/nvidia/nemotron-3-ultra-550b-a55b:$NIM_TAG ``` ## A performance-engineered starting point for production NVIDIA NIM packages model- and GPU-aware serving optimizations into a deployable microservice. For Nemotron 3 Ultra on a 4xB200 system, these optimizations let organizations serve up to 2.5x more users at 50 TPS/user compared with a NIM-off baseline, while keeping the deployment path practical for real multi-user serving. NIM combines validated performance engineering with regular inference-stack updates, CVE handling, hardware validation, and commercial support through NVIDIA AI Enterprise. More performance-optimized NIM configurations are planned across a broader range of models, giving developers additional validated starting points for their own latency, throughput, and cost objectives. ## Get started Download the Nemotron 3 Ultra NIM 2.0.12 from NGC, run it on your NVIDIA GPU infrastructure, and replay representative traffic with NVIDIA AIPerf to select the Pareto point that meets your application target. **Resources**