NVIDIA TensorRT Edge-LLM Outpaces llama.cpp by 6.4x on Jetson AGX Thor in MLPerf Edge Agentic Benchmark

NVIDIA TensorRT Edge-LLM Outpaces llama.cpp by 6.4x on Jetson AGX Thor in MLPerf Edge Agentic Benchmark In the MLPerf Inference v6.1 Edge Agentic benchmark, NVIDIA demonstrated that TensorRT Edge-LLM running Qwen3.6-27B on a single Jetson AGX Thor Developer Kit completed the performance workload 6.4 times faster than the llama.cpp reference run. TensorRT Edge-LLM completed the 1,007-turn performance benchmark in 24 minutes and 36 seconds, compared to 2 hours and 37 minutes for the llama.cpp reference submission. The system achieved an output throughput of 52.33 tokens per second, a median time to first token of 247.12 ms, and an overall BFCL accuracy of 87.94%. Optimizations included NVFP4 quantization for weights and activations, recurrent and KV cache reuse that served roughly 96% of prompt tokens via hot cache, and tree-based multi-token prediction (MTP). MLPerf Edge Agentic Benchmark and Results The MLPerf Edge Agentic benchmark tests edge inference using an OpenAI-compatible endpoint in two distinct phases: performance and accuracy. The performance workload replays recorded software-engineering agent trajectories across 20 conversations and 1,007 generated turns with context lengths expanding to about 23.5K tokens, while the accuracy phase evaluates tool and function calling via Berkeley Function Calling Leaderboard v4 prompts. Configured on a single Jetson AGX Thor Developer Kit in MAXN power mode with 128 GB unified memory, TensorRT Edge-LLM delivered 52.33 tokens per second and completed the workload in 24 minutes and 36 seconds, outpacing the 2-hour-and-37-minute llama.cpp Q4_K_M reference. Hardware Quantization and Cache Reuse To alleviate DRAM bandwidth constraints typical of low-batch edge decoding, the Qwen3.6-27B model used NVFP4 4-bit floating-point quantization for weights and activations alongside FP8 for the KV cache. To prevent recalculating shared conversation histories across agent turns, TensorRT Edge-LLM identified reusable prompt prefixes to restore cached attention KV pages and Qwen3.6's recurrent states. Consequently, about 96% of prompt tokens were served from hot cache, leaving only approximately 0.5 million of the total 13.6 million prompt tokens requiring prefill across all turns. Tree-Based Multi-Token Prediction Rather than standard autoregressive decoding or single-path linear drafting, TensorRT Edge-LLM utilized tree-based multi-token prediction. High-probability continuation candidates generated by a draft model were structured into a verification tree—configured for MLPerf with 8 draft steps, top-2 candidates per depth, and a 16-node tree—and verified by the target model in a single forward pass. This approach yielded an estimated additional 40% decoding performance improvement over linear MTP with 3 draft steps, capitalizing on the predictable syntax of tool names, JSON framing, and arguments.

TensorRT Edge-LLM 在 Jetson AGX Thor 上将 MLPerf 边缘 Agent 评测加速 6.4 倍 在 MLPerf Inference v6.1 Edge Agentic 基准测试中，NVIDIA TensorRT Edge-LLM 在 Jetson AGX Thor 开发者套件上运行 Qwen3.6-27B，以 24 分 36 秒完成全部工作负载，较 llama.cpp 参考实现的 2 小时 37 分钟提速 6.4 倍。 在 MAXN 功耗模式的 Jetson AGX Thor 上，TensorRT Edge-LLM 运行 Qwen3.6-27B 达成每秒 52.33 个 Token 的吞吐量，BFCL 准确率达 87.94%。 采用 NVFP4 权重与激活值量化结合 FP8 KV 缓存，大幅降低 DRAM 带宽限制并节省统一内存。 实现跨 Agent 轮次的 KV 缓存与循环状态复用，使约 96% 的 Prompt Token 实现热缓存命中。 引入基于树状的多 Token 预测（MTP），相较于 3 步线性 MTP 额外提升约 40% 的解码性能。 MLPerf Edge Agentic 基准测试表现 MLPerf Edge Agentic 评测包含性能与准确率两阶段。性能阶段重放包含 20 轮对话、1007 次生成的软件工程 Agent 轨迹，上下文长度最高约 2.35 万 Token；准确率阶段采用 BFCL v4 评估函数调用。TensorRT Edge-LLM 在单台配备 128 GB 统一内存的 Jetson AGX Thor 上，首 Token 中位延迟为 247.12 毫秒，每 Token 生成中位耗时 14.68 毫秒，耗时比 llama.cpp 的 Q4_K_M 参考实现缩短 6.4 倍。 NVFP4 量化与 KV 缓存跨轮次复用 针对边缘设备低批次 LLM 解码受限 DRAM 带宽的特性，模型权重、激活值及语言模型头均采用 Jetson AGX Thor 架构支持的 NVFP4 格式，KV 缓存使用 FP8。针对 Agent 多轮交互中不断增长的历史信息，运行时可识别提示前缀并恢复缓存的 KV 页、循环状态和部分 KV 页状态，仅对新增后缀执行 Prefill。在该负载中，总共 1360 万 Prompt Token 仅需重新 Prefill 约 50 万，约 96% 命中热缓存。 树状多 Token 预测加速解码 TensorRT Edge-LLM 采用树状多 Token 预测（MTP）技术，将草稿模型生成的高概率候选项组织为分支树结构，由目标模型单次前向传播统一验证，支持单步推进多个 Token。MLPerf 服务配置采用 8 个草稿步骤、每层 Top-2 候选并构建 16 节点验证树。该结构高度契合工具调用中 JSON 语法及参数结构的规律性，相较于 3 步线性 MTP 解码吞吐提升约 40%。

AI agents are moving from cloud data centers to vehicles, robots, and other edge devices. Unlike a chatbot that answers a single prompt, an agent works through a sequence of steps. It selects tools, evaluates their results, and continues reasoning within an increasingly long conversation. This workflow places new demands on edge inference. The model must generate tokens quickly, process long shared histories efficiently, and produce valid tool calls within a limited power and memory envelope. In the MLPerf Inference v6.1 Edge Agentic benchmark, NVIDIA TensorRT Edge-LLM ran Qwen3.6-27B on a single NVIDIA Jetson AGX Thor Developer Kit. The system achieved 52.33 tokens per second and completed all 1,007 turns of the performance workload in 24 minutes and 36 seconds, 6.4x faster than the llama.cpp reference submission of 2 hours and 37 minutes. The result uses NVFP4 quantization, tree-based multi-token prediction (MTP), and KV cache reuse. How MLPerf measures end-to-end agentic AI performance MLPerf Edge Agentic measures an OpenAI-compatible model endpoint in two phases: performance and accuracy. The performance phase replays recorded software-engineering agent trajectories. The model receives a user request, generates a tool call, observes the tool result, and continues the same conversation. The workload contains 20 conversations and 1,007 generated turns. Input length grows across turns and reaches approximately 23.5K tokens, making long-context processing an important part of the measurement. Intersection of Union (IoU) based inline accuracy is also measured to ensure the performance phase is running the agent correctly. The accuracy phase uses Berkeley Function Calling Leaderboard (BFCL) v4 prompts with single-turn only and reasoning off to balance accuracy and evaluation time on edge devices. It evaluates whether the model selects the correct function, generates valid arguments, and avoids calling a tool when no tool is required. Figure 1. The MLPerf Edge Agentic workload measures a multi-turn loop in which tool results are added to the conversation before the next model request MLPerf Edge Agentic benchmark results on Jetson AGX Thor The TensorRT Edge-LLM submission used Qwen3.6-27B in SingleStream mode on one Jetson AGX Thor Developer Kit with 128 GB of unified memory at the MAXN power mode.. Metric Result Output throughput 52.33 tokens per second Median time to first token 247.12 ms Median time per output token 14.68 ms BFCL overall accuracy 87.94% Table 1. MLPerf v6.1 NVIDIA Edge Agentic submission result summary The MLCommons Edge Agentic example also publishes a llama.cpp reference run on Jetson AGX Thor, which uses Qwen3.6-27B with Q4_K_M quantization and completes in 2 hours and 37 minutes. The TensorRT Edge-LLM submission completes the workload in 24 minutes and 36 seconds, or 6.4x less time. Figure 2. End-to-end completion time for the MLPerf Edge Agentic performance workload on Jetson AGX Thor Using NVFP4 quantization to accelerate Qwen3.6-27B inference Low-batch LLM decoding on edge platforms is known to be heavily bounded by DRAM bandwidth. Reducing the size of both weights and activations reduces the memory footprint of the kernels and therefore boosts decoding performance. The submitted Qwen3.6-27B model uses NVFP4 for weights and activations, including the language-model head, and FP8 for the KV cache. NVFP4 is a 4-bit floating-point format supported by the NVIDIA Blackwell GPU in Jetson AGX Thor. TensorRT Edge-LLM uses optimized kernels to accelerate the quantized model while maintaining the accuracy required by MLPerf. The smaller model representation also leaves more of the 128 GB unified memory available for the long context, speculative decoding state, and application workloads. Developers can start from a published, calibrated Qwen3.6-27B NVFP4 checkpoint . A deployment team can use a published quantized checkpoint or perform post-training quantization once on any development system before deploying the model. Reusing KV cache across agent turns Each new request in an agent trajectory contains most of the preceding conversation plus a new model response or tool result. Without reuse, the model must prefill the shared history again on every turn. The cost increases as the conversation grows. TensorRT Edge-LLM identifies reusable prompt prefixes and restores their cached attention KV pages. Qwen3.6 uses a hybrid model architecture, so the runtime also restores the recurrent state and partial KV-page state required to continue execution correctly. It then prefills only the new suffix of the conversation. KV cache and recurrent-state reuse reduce repeated long-context computation across the complete agent trajectory. This optimization complements tree-based MTP: cache reuse reduces the cost before generation begins, while MTP reduces the number of target-model steps during generation. On this workload, ~96% of the prompt tokens are served with hot cache. The runtime only prefills ~0.5M of the total 13.6M prompt tokens across the turns. Using tree-based multi-token prediction to accelerate LLM inference Standard autoregressive decoding generates one token for each model invocation. Multi-token prediction uses a draft model to predict several future tokens, which the target model then verifies together. Recent models typically have official MTP weights trained and shipped together with the main model. In addition to traditional linear MTP, TensorRT Edge-LLM implements a tree-based MTP implementation. Instead of keeping only one predicted continuation, the runtime organizes high-probability candidates into a tree. The target model verifies the candidates in one forward pass, and the runtime accepts the matching path. If multiple candidates are accepted, the system advances generation by several tokens. You can adjust drafting parameters when launching the server. The MLPerf server configuration uses 8 draft steps, the top-2 candidates at each drafting depth, and a 16-node verification tree. Tree-based verification is useful for function calling because tool names, JSON syntax, and common argument structures are often predictable, while multiple branches can preserve likely alternatives for individual argument values. Compared with a linear MTP with 3 draft steps, tree-based MTP could achieve an additional ~40% decoding performance gain for this workload. How to run the MLPerf Edge Agentic benchmark The implementation used for this submission is available on the TensorRT Edge-LLM release/0.9.1-mlpinf branch . The branch includes the model export settings, TensorRT engine build commands, server configuration, and MLPerf client configuration 1. Clone TensorRT Edge-LLM and initialize its submodules. git clone --branch release/0.9.1-mlpinf \ https://github.com/NVIDIA/TensorRT-Edge-LLM.git cd TensorRT-Edge-LLM git submodule update --init --recursive 2. Download the calibrated NVFP4 checkpoint. huggingface-cli download \ centml/Qwen3.6-27B-NVFP4-W4A4-mlpinf \ --local-dir "$WORK/Qwen3.6-27B-NVFP4-W4A4-mlpinf" 3. Follow mlperf/README.md to build TensorRT Edge-LLM, export the checkpoint with the tree-MTP interface, and build the base and draft TensorRT engines. $VENV/bin/python -m tensorrt_edgellm.scripts.export \ "$WORK/Qwen3.6-27B-NVFP4-W4A4-mlpinf" \ "$WORK/onnx" \ --mtp-tree-base --skip-visual 4. Launch the OpenAI-compatible TensorRT Edge-LLM server. export REPO="$PWD" export VENV=/path/to/venv-edgellm-export export WORK=/path/to/mlperf-artifacts bash mlperf/serve_edgellm.sh 5. Clone the MLCommons endpoint harness, install its BFCL dependencies, update the model and tokenizer paths in mlperf/config.yaml , and run the benchmark. git clone https://github.com/mlcommons/endpoints.git cd endpoints python3.12 -m venv .venv source .venv/bin/activate pip install -e ".[dev,bfcl]" inference-endpoint benchmark from-config \ --config "$REPO/mlperf/config.yaml" The supplied configuration runs the performance and accuracy phases with temperature 0, seed 42, reasoning disabled, and concurrency 1. Use the --accuracy-only option to run only the BFCL accuracy phase. For more information about the dataset and client configuration, see the MLCommons Edge Agentic example . For full MLPerf Inference v6.1 results across all submissions, see the MLCommons announcement . For server-scale Vera Rubin performance, see NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut . Acknowledgments This work represents contributions from the TensorRT Edge-LLM, ModelOpt, Jetson, and MLPerf teams at NVIDIA, especially Zihao Kong, Xiang Guo, Yoco Xiao, Qikai Li and Ashwin Nanjappa. Thanks to the MLCommons community for developing the Edge Agentic benchmark and endpoint harness.