NVIDIA Evaluates Vera Rubin and Blackwell Architectures on Agentic AI Benchmarks

NVIDIA Evaluates Vera Rubin and Blackwell Architectures on Agentic AI Benchmarks Evaluations using the SemiAnalysis AgentX benchmark indicate that NVIDIA's Vera Rubin and Blackwell GB300 architectures deliver significant improvements in token throughput per megawatt for dynamic, multi-turn agentic workloads compared to previous hardware generations. OpenRouter data shows single agentic requests consume 15 times more tokens than standard chat, prompting a shift from static benchmarks to dynamic session replays in AgentX. NVIDIA preview testing pending review shows Vera Rubin NVL72 achieves up to 30x higher throughput per megawatt than GB300 NVL72 on the DeepSeek V4-Pro workload at 160 tokens per second per user. Blackwell GB300 NVL72 demonstrated up to a 15x throughput-per-megawatt advantage and up to 10x lower cost per million tokens over H200 NVL8 on DeepSeek V4 Pro 1.6T, rising to roughly 80x on Kimi K3 2.8T. Performance improvements stem from system-level serving runtimes, mixed-precision MoE kernels, NVLink fabrics, and NVIDIA Dynamo's cache-aware routing. Agentic Workloads and the AgentX Benchmark Agentic AI shifts inference from single-turn chats to multi-step interactions involving reasoning, tool invocation, and growing context accumulation. OpenRouter metrics across 100 trillion tokens show prompt tokens per request have grown roughly fourfold. Because static sequence tests fail to reflect real-world variable requests and cache retention, the open-source SemiAnalysis AgentX benchmark replays prerecorded Claude Code sessions turn-by-turn using the AIPerf client. The framework evaluates system throughput per provisioned megawatt alongside interactivity metrics, end-to-end latency, and time to first token while testing context reuse under variable concurrency. Vera Rubin and Blackwell Benchmark Performance In benchmark measurements conducted by NVIDIA that are pending SemiAnalysis review, Vera Rubin NVL72 achieved up to 30x higher throughput per megawatt than GB300 NVL72 on the AgentX DeepSeek V4-Pro workload at an interactivity target of 160 tokens per second per user. For Blackwell GB300 NVL72, tests on DeepSeek V4 Pro 1.6T demonstrated up to 15x higher throughput per megawatt and up to 10x lower cost per million tokens compared to H200 NVL8. When evaluated on the larger Kimi K3 2.8T model, GB300 NVL72 delivered roughly 80x the throughput per megawatt of H200 NVL8 and extended user interactivity up to approximately 215 tokens per second. Underlying Serving Stack and System Co-Design The reported efficiency gains across GB300 NVL72 rely on full-stack integration across hardware and software. Runtimes such as TensorRT-LLM, SGLang, and vLLM employ Wide Expert Parallelism to distribute mixture-of-experts workloads across 72 GPUs linked by NVLink. DeepGEMM-based kernels and MXFP4/MXFP8 precision formats overlap communication with Tensor Core computations. Additionally, NVIDIA Dynamo decouples prefill and decode execution into independent pools, using session IDs and KV-cache-aware routing to align requests with existing cache states and minimize redundant prefill work.

NVIDIA 发布 Rubin 与 Blackwell 在 Agentic AI 负载下的每瓦性能表现 随着 AI 智能体推理逐步转向包含多步骤推理、工具调用及长上下文累积的复杂工作流，NVIDIA 结合 SemiAnalysis AgentX 基准测试展示了其硬件平台的能效表现。预览数据显示，Vera Rubin NVL72 的每兆瓦吞吐量最高可达 GB300 NVL72 的 30 倍，而 GB300 NVL72 相较前代 H200 亦实现显著跃升。 Agentic 任务相比常规对话单次消耗高达 15 倍的 Token，促使测试标准由固定长度转向模拟真实生产回放的 SemiAnalysis AgentX 基准。 在 DeepSeek V4-Pro 负载及 160 tokens/s/user 交互速度下，NVIDIA Vera Rubin NVL72 实现最高达 GB300 NVL72 30 倍的每兆瓦吞吐量（测试待 SemiAnalysis 审查）。 GB300 NVL72 在 DeepSeek V4 Pro 1.6T 上每兆瓦吞吐量可达 H200 NVL8 的 15 倍，使百万 Token 成本降低多达 10 倍；在 Kimi K3 2.8T 负载下每兆瓦吞吐量优势进一步扩大至约 80 倍。 系统级提升依托于 MoE 服务运行时、DeepGEMM/MXFP 混合精度内核、NVIDIA Dynamo（解耦 Prefill 与 Decode 并支持 KV-cache 感知路由）以及 NVLink 机架级互联架构。 AgentX 基准测试与传统评估的区别 OpenRouter 报告显示智能体请求消耗的 Token 量是普通对话的 15 倍。以往基于固定序列长度（如 8K/1K）的测试已被 InferenceX 列入维护模式。AgentX 采用重放 Claude Code 真实会话的方式，保留了交替推理、工具调用时延和不断累积的上下文，以测量硬件在应对 KV-cache 容量压力、并发交互及每兆瓦可用输出（Tokens per Megawatt）方面的综合表现。 Rubin 与 Blackwell 的能耗与成本表现 NVIDIA 测算的预览数据显示，在每用户 160 tokens/s 的 DeepSeek V4-Pro 负载下，Vera Rubin NVL72 的每兆瓦吞吐量达 GB300 NVL72 的 30 倍。针对现有硬件，GB300 NVL72 在 DeepSeek V4 Pro 1.6T 负载中相较 H200 NVL8 取得最高 15 倍的每兆瓦吞吐量提升，将百万 Token 成本降低多达 10 倍；而在超大规模的 Kimi K3 2.8T 任务中，GB300 NVL72 的能效比更是达到 H200 的约 80 倍，并将交互边界推进至 215 tokens/s/user。 底层系统协同设计支撑 GB300 与 Rubin 平台的性能提升由软硬件全栈协同实现。软件层借助 SGLang、TensorRT-LLM 和 vLLM 的 Wide Expert Parallelism 技术实现 MoE 专家负载均衡；MoE 内核融合与 MXFP4/MXFP8 精度重叠了通信与计算。NVIDIA Dynamo 负责解耦 Prefill 与 Decode 计算池，并利用 KV-cache 感知路由减少重复计算；硬件层则由 NVLink 互联整个机架，协同完成高并发任务下的数据流转。

AI agents have expanded inference from single-turn interactions into multi-step workflows that reason, invoke tools, coordinate subagents, and carry growing context from one turn to the next. The scale of this shift is now visible in raw consumption: across 100 trillion tokens of real-world usage, OpenRouter’s State of AI report found that average prompt tokens per request grew roughly fourfold, and single agentic requests consume 15 times the tokens of ordinary chat. Properly characterizing hardware performance for this workload presents new challenges. A useful benchmark must capture long-context prefill, KV-cache reuse, interactive decode, tool-call gaps, and distributed mixture of experts (MoE) execution under realistic concurrency. It must also show how much of the AI-factory power budget is converted into useful agentic throughput while maintaining an acceptable user experience. This post examines SemiAnalysis AgentX , a benchmark that evaluates AI infrastructure for agentic-coding inference using replayed production-style sessions. It highlights Vera Rubin NVL72 preview results showing up to 30x higher AI-factory throughput per megawatt than GB300 NVL72, while showing that Blackwell GB300 NVL72 extends its order-of-magnitude throughput-per-megawatt advantage over prior generations to dynamic agentic workloads. What is AgentX? AgentX is the agentic-coding benchmark in InferenceX , SemiAnalysis’s open-source benchmark suite. It measures how efficiently accelerators serve the request patterns produced by real coding agents. AgentX testing and measurement methodology Agentic sessions are long, stateful, and variable: they chain model calls, tool use, and growing context rather than following a fixed prompt-and-response pattern (Figure 1, below). AgentX therefore measures whether a platform can serve replayed agent traffic responsively, reuse previously processed context, and maximize agentic throughput per provisioned megawatt. Figure 1. Agentic sessions create variable inference sequences Figure 2, below, shows the legacy InferenceX static 8K/1K sequence length results for DeepSeek-R1-0528. GB300 NVL72 leads H200 by up to 40x more tokens per megawatt. However, the scenario represents controlled, fixed-length serving rather than the traffic generated by an agent in production. Figure 2. GB300 NVL72 delivers higher token throughput per all-in utility megawatt than H200 across a broader interactivity range in the 8K ISL / 1K OSL DeepSeek R1 0528 test In real agent sessions, request lengths vary from turn to turn. Context accumulates throughout a task, previously processed tokens can be reused, and model calls are interrupted by tool execution or delegated work. As agentic AI has become the leading workload, a fixed sequence length scenario has become less representative of true serving performance and has been demoted to “maintenance mode” on the InferenceX suite. AgentX addresses that gap by measuring serving performance across prerecorded Claude Code sessions with interleaved reasoning and tool use. It replays each session turn by turn using the AIPerf client . Because every system receives the same recorded traffic, observed differences reflect the serving stack rather than benchmark-specific tuning. Figure 3, below, illustrates a sample pre-recorded session. Figure 3. Context accumulates across a session The replay preserves each session’s context and input/output sequence lengths, as well as the reasoning time and tool-call latency captured in the original trajectory. These intervals preserve the original session’s timing, reproducing KV cache-capacity pressure that a realistic benchmark must capture. AgentX varies concurrency to map the trade-off between throughput and interactivity. At each operating point, it reports sustained throughput per provisioned megawatt alongside each stack’s ability to reuse repeated context How to interpret AgentX results The AgentX metric that matters most for AI factories is tokens per megawatt. The benchmark reports this metric against four user-experience values: E2E normalized interactivity, standard interactivity, E2E latency, and time to first token (TTFT). Table 1, below, describes how to best leverage these. Metric Definition Usefulness E2E Normalized Interactivity The average output-token rate per user over the full request, calculated by dividing total output tokens by the time from request submission to final-token delivery. Shows how much user-visible output a platform delivers per megawatt for a given end-to-end experience, including time to first token. Higher is better. Standard Interactivity The per-user token rate during generation alone, calculated as output tokens divided by the elapsed time from the first token to the last. Shows how much streamed output a platform delivers per megawatt once generation begins, excluding time to first token. Higher is better. E2E Latency The total elapsed time of a single request, measured from submission to the arrival of the final output token. Tests whether a throughput-per-megawatt result is usable: high efficiency does not help if requests take too long to finish. Less is better. TTFT The elapsed time from submission to the arrival of the first output token. Shows whether power-efficient throughput is paired with a fast first response, which is critical when agents repeatedly begin long-context turns. Lower is better. Table 1. Key AgentX performance metrics NVIDIA Vera Rubin NVL72 Results The following Vera Rubin NVL72 results were measured by NVIDIA using the SemiAnalysis AgentX workload and are pending SemiAnalysis review. As shown in Figure 4, below, at 160 tokens per second per user on the AgentX DeepSeek V4-Pro workload, Vera Rubin NVL72 delivers up to 30x higher AI-factory throughput per megawatt than GB300 NVL72, pointing to a substantial increase in agentic inference capacity while maintaining the same interactive serving target. Figure 4. Vera Rubin delivers 30x higher throughput per megawatt at 160 interactivity To learn more about how NVIDIA Vera Rubin NVL72 was designed to deliver maximum performance per megawatt, see Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI. NVIDIA GB300 NVL72 Results On AgentX, GB300 NVL72 delivers up to 15x higher AI-factory throughput per megawatt than H200 NVL8 for DeepSeek V4 Pro 1.6T on the AgentX workload (see Figure 5, below). In other words, GB300 NVL72 sustains substantially more responsive agentic inference throughput within the same power budget. Figure 5. GB300 NVL72 delivers up to 15x higher AI-factory throughput per megawatt than H200 NVL8 That throughput advantage translates directly into unit economics. GB300 NVL72 delivers up to 10x lower cost per million tokens than H200 NVL8 (Figure 6, below). For operators, this means a fixed power and infrastructure budget can support materially more interactive agentic capacity, or deliver the same capacity with substantially lower operating cost. Figure 6. GB300 NVL72 delivers up to 10x lower token cost than H200 NVL8 The advantage of GB300 NVL72 becomes more pronounced as model scale increases. Figure 7, below, shows throughput per megawatt for Kimi K3 2.8T on the AgentX workload. With roughly 80x the throughput per megawatt of H200 NVL8 at comparable interactivity, GB300 NVL72 also extends the interactivity frontier to roughly 215 tokens per second per user, well beyond the operating range reached by H200 NVL8. Figure 7. GB300 NVL72 delivers 80x higher throughput per megawatt than H200 NVL8 on Kimi K3 2.8T These GB300 NVL72 results reflect system-level work across the serving runtime, model kernels, and scale-up fabric. Together, these layers enable large MoE models to sustain responsive throughput as agent sessions accumulate context, concurrency increases, and decode demand intensifies. MoE serving runtimes: Frameworks including SGLang , TensorRT-LLM , and vLLM can distribute expert execution across the NVL72 domain. Techniques such as Wide Expert Parallelism and DeepEP help balance expert work across more GPUs, increasing the effective batch size available for concurrent agent requests. MoE kernels and communication overlap: DeepGEMM-based kernels, mixed-precision formats such as MXFP4 and MXFP8, and fused MoE execution paths reduce time spent moving data between expert stages. By overlapping expert-parallel communication with Tensor Core computation, the serving stack can improve token throughput for reasoning and coding workloads. NVIDIA Dynamo : Dynamo separates prefill and decode into independently scaled worker pools, allowing each phase to be configured for its own performance requirements. It also supports session-aware serving through session IDs that correlate related model calls, tool spans, and traces across an agent run. Separately, its KV-cache-aware router uses cache overlap and worker load to select a target, reducing unnecessary prefill computation and helping maintain responsive multi-turn serving as agent sessions reuse context. NVIDIA NVLink scale-up fabric: NVLink connects the 72 GPUs in GB300 NVL72 in a high-bandwidth scale-up domain. This enables the compute, memory, expert-parallel communication, and KV-cache movement required to serve large models as a coordinated rack-scale system. What’s next for agentic AI? Vera Rubin NVL72 shows what is possible when a rack-scale system is tuned for the long-context, interactive, and distributed execution patterns of agentic inference. The broader Vera Rubin platform extends that approach across the full workflow. Rubin GPUs process large context and decode efficiently, Vera CPUs handle tool execution and KV-cache offload, and Groq 3 LPX unlocks ultrafast interactivity. Across the AI factory, NVLink 6, ConnectX-9, BlueField-4, and Spectrum-X move tokens, context, and tool results between resources. Dynamo, Attention-FFN Disaggregation, NVFP4, TensorRT-LLM WideEP, and speculative decoding coordinate execution across the most suitable processors. The goal is simple: reduce recomputation and waiting, sustain interactive performance as agent sessions grow, and convert more of a fixed power budget into useful agentic output. To explore the technologies and benchmarks behind this result, start with the resources below. Explore the architecture and capabilities of NVIDIA Rubin Review live benchmark results on the SemiAnalysis InferenceX dashboard Learn how NVIDIA’s extreme co-design addresses the growing complexity, latency, and economics of agentic systems in Building for the Rising Complexity of Agentic Systems with Extreme Co-Design . Acknowledgments This work was made possible through the expertise and engineering contributions of Xin Li, Ankur Singh, Anthony Casagrande, Jonas Li, Po-Han Huang, Xiaoming Chen, and many other talented NVIDIA engineers.