NVIDIA Introduces AIPerf for Scalable LLM Inference Benchmarking

NVIDIA Introduces AIPerf for Scalable LLM Inference Benchmarking NVIDIA has released AIPerf, a ground-up rewrite and successor to GenAI-Perf engineered to eliminate client-side bottlenecks when benchmarking large language model inference under heavy concurrency. AIPerf decouples from Perf Analyzer using a multiprocess architecture coordinated over ZMQ to bypass Python GIL limitations. The tool supports over 15 endpoint types, public datasets like ShareGPT, trace replay formats, and diverse arrival patterns including Poisson and gamma. AIPerf captures percentile breakdowns for key latency and throughput metrics while directly pulling GPU hardware telemetry via DCGM or pynvml. Architecture and Workload Capabilities AIPerf replaces GenAI-Perf with an independent, multiprocessed architecture where dedicated worker processes generate load and separate record-processor services manage results via ZMQ coordination. This design prevents the client from becoming the bottleneck under high concurrency. The tool supports more than 15 endpoint types—such as chat, responses, NIM rankings, and image generation—alongside datasets like ShareGPT and trace replays from platforms including Baseten, Mooncake, and WEKA AgentX. Users can configure arrival shapes using constant, Poisson, or gamma distributions with variable burstiness and concurrency ramping. Benchmarking Workflow and Telemetry Using a reference deployment of Qwen3-0.6B on vLLM, AIPerf executes static and synthetic profiles configured via command-line flags. Enabling streaming is required to properly record Time to First Token (TTFT) and Inter-Token Latency (ITL). The tool reports min, max, average, standard deviation, and percentiles ranging from p25 to p99 across core latency metrics, overall request latency, and output token throughput. When DCGM or pynvml is available, AIPerf simultaneously logs GPU power draw, utilization, and memory consumption to correlate performance anomalies with hardware pressure. Dynamic Traffic and Arrival Modeling Beyond fixed-length static tests, AIPerf allows operators to emulate realistic production conditions by introducing variance into request timing and token counts. By setting a Poisson arrival pattern and standard deviations around input and output sequence lengths, the benchmark introduces bursts, gaps, and varying prefill workloads. Comparative tests show that introducing dynamic Poisson arrivals widens the TTFT and ITL distribution significantly compared to idealized single-user baselines due to overlapping prefill and decode stages competing for GPU resources.

NVIDIA 推出大模型推理压测工具 AIPerf 针对传统单进程压测工具容易陷入 Python GIL 瓶颈和配置繁琐的问题，NVIDIA 推出了 GenAI-Perf 的全新重写替代品 AIPerf。该工具采用多进程解耦架构，支持丰富的流量形态与真实流量重放，用于对大规模 LLM 推理服务进行高保真度性能评测。 AIPerf 是 GenAI-Perf 的全新重构版本，脱离了 Perf Analyzer 底层，改用多进程和 ZMQ 协调架构，避免客户端成为性能瓶颈。 工具支持 15 种以上终端类型，涵盖主流数据集与 Mooncake、Baseten 等真实调用轨迹重放，并允许模拟泊松、伽马等复杂流量分布。 核心监控指标包括 TTFT、ITL、请求端到端延迟及输出 Token 吞吐量，并支持分位数统计及联动 DCGM/pynvml 捕获 GPU 硬件监控数据。 架构改进与负载控制能力 传统基准测试工具（包括早期的 GenAI-Perf）普遍采用单进程架构，在高并发和高请求率下极易受 Python 全局解释器锁（GIL）限制。AIPerf 进行了底层重写，采用 Worker 进程发压、独立 Record 处理器处理结果并通过 ZMQ 进行通信的多进程架构。在负载控制方面，AIPerf 支持恒定、泊松（Poisson）和伽马（Gamma）到达模式，允许微调突发性、逐步增加并发度，并可自定义输入输出长度分布以贴近真实生产环境。 测试实践与核心指标解读 在以 vLLM 部署 Qwen3-0.6B 为例的基准测试中，AIPerf 通过开启流式传输（streaming）来精准测量首字延迟（TTFT）与每字生成间隔延迟（ITL）。测试输出提供核心四大指标：衡量交互响应的 TTFT、反映解码阶段压力的 ITL、结合了预填充与解码开销的请求端到端延迟，以及用于容量规划的输出 Token 吞吐率。指标均提供 p25 至 p99 等百分位数统计，且在具备 DCGM 或 pynvml 时可直接联动记录 GPU 功耗、利用率与显存开销。 动态流量模拟与并发排队表现 相比输入输出长度固定的静态压测，AIPerf 支持通过配置泊松分布参数（如设定均值与标准差）模拟真实流量中的抖动与排队现象。在泊松流量下，不同长度的输入提示词交错到达，导致预填充阶段算力竞争与解码操作重叠，使得实际 TTFT 的分布带宽显著放宽，展现了理想单用户模式下无法观测到的长尾排队效应。

You’re deploying a model on a system. It starts up, prompts are getting responses. Now the hard question: Is this fast? Your instincts might lead you to send curl commands, hand-roll an asyncio script, or vibe code yet another one-off load generator. All of these paths have the same problem: single-process performance limits, Python’s GIL capping concurrency, or numbers measured against a reference you built yourself. Either way, you end up with results you can’t fully trust, attached to tooling you’ll have to rewrite the moment requirements change. What you need is a load client that can saturate a real server without becoming the bottleneck, produce output you can act on, and take five minutes to configure, not five hours. That’s NVIDIA AIPerf. What AIPerf does differently AIPerf is the designated successor to GenAI-Perf and is a ground-up rewrite. The design choices reflect hard lessons from running LLM benchmarks at scale: A clean break from the old architecture. AIPerf doesn’t run on top of Perf Analyzer the way GenAI-Perf did. It’s a clean architectural break and the reason AIPerf can scale the way it does. If you’re porting an existing workflow, the migration guide covers the key deltas. The client shouldn’t be the bottleneck. Most benchmarkers, GenAI-Perf included, use a single-process architecture that becomes GIL-bound under real concurrency or request rate. AIPerf is a multiprocessed system: worker processes generate load, separate record-processor services handle results, and everything is coordinated over ZMQ.. This structure allows for more accurate server benchmarking by preventing AIPerf from becoming a client-side bottleneck. Workload breadth that matches what you actually run. AIPerf supports 15+ endpoint types: chat, responses, NIM rankings, image generation, and more — along with public datasets like ShareGPT and trace replay formats from Mooncake, Baseten, WEKA (AgentX), and others. Whether you’re running a quick synthetic smoke test or replaying captured production traffic, you don’t need a different tool. Load shape you actually control. AIPerf supports constant, Poisson, and gamma arrival patterns with tunable burstiness, gradual ramping for concurrency and request rate, and synthetic distributions including vLLM/SGLang range-ratio for variable ISL/OSL. You control the shape of the load, not just the volume. Your maiden benchmark: Synthetic ISL/OSL on vLLM For this walkthrough we’ll use Qwen3-0.6B served through vLLM. The model choice is deliberate; it’s small enough to run on a single GPU and fast enough to iterate on without waiting. The point isn’t to benchmark Qwen3-0.6B specifically; it’s to establish the measurement loop. Once you have that, swapping in a different model or endpoint is a one-flag change. Start the Server Pull and start vLLM with the reasoning parser enabled: docker pull vllm/vllm-openai:latest docker run --gpus all -p 8000:8000 -e HF_TOKEN vllm/vllm-openai:latest \ --model Qwen/Qwen3-0.6B \ --reasoning-parser qwen3 \ --host 0.0.0.0 --port 8000 Install AIPerf We can use uv to install a central copy: uv tool install aiperf Or for a virtual environment: uv venv venv source venv/bin/activate uv pip install aiperf One platform note: on aarch64, the crick dependency ships as source-only and requires a C toolchain ( build-essential on Debian/Ubuntu, Development Tools on RHEL). If the install stalls on that package, that’s why. Running the benchmark With the server up and AIPerf installed, we can now run our first profile: aiperf profile \ --model Qwen/Qwen3-0.6B \ --endpoint-type chat \ --streaming \ --url localhost:8000 \ --synthetic-input-tokens-mean 128 \ --synthetic-input-tokens-stddev 0 \ --output-tokens-mean 128 \ --output-tokens-stddev 0 \ --extra-inputs min_tokens:128 \ --extra-inputs ignore_eos:true A few flags here are doing more work than they look like: --synthetic-input-tokens-stddev 0 and --output-tokens-stddev 0 pin the workload to exactly 128 input and 128 output tokens per request. This reproduces a commonly used static benchmark that holds request and output lengths constant. --extra-inputs min_tokens:128 and --extra-inputs ignore_eos:true tell the model to actually emit 128 tokens rather than stopping early. Without these, the output token count is a suggestion. The model stops whenever it naturally finishes, which can be well short of your target OSL. Throughput numbers end up lower than they should be, and they’re not reproducible across runs. --streaming is not optional if you want to measure TTFT and ITL. Without streaming, the server batches the full response before sending it, and there are no first- or decode-token events to measure. What you’ll see Figure 1. An example animation of the AIPerf live dashboard user interface. The live dashboard shows the progress of the run, a listing of metrics along with their distributions, as well as a running log of events from the AIPerf backend We’ll walk through how to read these numbers in the next section. For now, notice the shape of the output in Figure 2, below: latency broken down by percentile, throughput in tokens per second, and request-level statistics all in one place. That’s the baseline you’ll be comparing everything else against. Figure 2. An example screenshot of the output metrics at the end of an AIPerf run which includes a summary of effective, active, summary statistics for a variety of different metrics along with percentile breakdowns for quick review, reproduction command line, and output locations Reading the numbers: What AIPerf surfaces Once a run completes, AIPerf prints a metrics table to the console and writes the full results to CSV and JSON. Here’s what you’re looking at. The core four: TTFT (Time to First Token) — How long from request sent to first token received. The primary latency signal for interactive use cases. ITL (Inter-Token Latency) — Time between successive tokens during generation. High ITL means the decode phase is struggling, even if TTFT looks healthy. Request Latency — End-to-end time for the full response. Combines prefill and decode cost into a single number. Output Token Throughput — Tokens generated per second across all concurrent requests. The primary throughput signal for capacity planning. For full definitions of these and every other metric AIPerf reports, see the Metrics Reference . Getting the full picture. Each of the above is reported in percentile breakdowns (p25, p50, p75, p90, p95, p99) alongside their minimums, maximums, averages, and standard deviations. These breakdowns matter because they can highlight long tail distributions; a server with a healthy mean TTFT and an outlier p99 looks fine in aggregate and fails in production. Beyond the core four. With DCGM or pynvml available, AIPerf also pulls GPU power draw, utilization, and memory consumption into the same run output. Correlating a latency spike with a memory pressure event doesn’t require a separate profiling session, the telemetry is already there. Going further: Configuring a traffic pattern Now that our feet are wet with a static benchmark, we can start exploring something more dynamic. The section above provided an extremely fixed traffic pattern, but real inference traffic doesn’t follow a static pattern. To benchmark with a scenario that’s less rigid, we can use some of AIPerf’s synthetic workload knobs to introduce variability to our requests. aiperf profile \ --model Qwen/Qwen3-0.6B \ --endpoint-type chat \ --streaming \ --url localhost:8000 \ --request-rate 10 \ --arrival-pattern poisson \ --synthetic-input-tokens-mean 512 \ --synthetic-input-tokens-stddev 128 \ --output-tokens-mean 128 \ --output-tokens-stddev 32 \ --random-seed 42 \ --request-count 200 A few things changed from the static benchmark above. --arrival-pattern poisson with --request-rate 10 means requests arrive at an average of 10 per second, with inter-arrival times drawn from an exponential distribution. The server now experiences bursts and gaps rather than a single user stream, which is what queuing actually looks like under real traffic. --synthetic-input-tokens-stddev 128 introduces variance around the 512-token mean, producing a mix of short and long prompts. The server has to handle variable prompt lengths during prefill rather than identical ones. --output-tokens-stddev 32 adds variance on the output side. Notice that min_tokens and ignore_eos are gone from this command. In the static benchmark those flags pinned outputs to exactly 128 tokens to keep the baseline clean; we’re deliberately releasing that constraint so the output distribution can vary. --random-seed 42 makes the Poisson timing and synthetic length draws reproducible. Rerunning this command produces the same sequence of requests. --streaming is not optional. Without streaming, the server batches the full response before sending it, and there’s no first- or decode-token events to measure. Looking at the LLM metrics from this run, the distributions are noticeably wider than the static baseline — which is expected when more requests are simultaneously competing for GPU access and prefill lengths vary per request. Figure 3. An example screenshot of the summary statistics from the Poisson arrival pattern run. The distribution of statistics drastically differs from the 512/128 static scenario due to the new traffic pattern Looking at the graphs in Figure 4, below, you can see that the Poisson command line introduced a request rate centered, but not exactly matching, around 10 requests/second. This arrival rate emulates jitter around when requests arrive compared to the constant mode which guarantees a fixed 10 requests/second. Figure 4. The reported delay from first request dispatch, compared to the constant 10 requests/sec mode, showing variation in dispatch timing centered around the specified request rate You can see in Figure 5, below, that there is a variation in the request length centered around the mean of 512 tokens, with input sequence lengths ranging 154 to 818 tokens. Figure 5. A histogram showing the distribution of input (request) lengths centered around the requested 512 average token count Comparing TTFT between the two runs, you can see that the Poisson run shows a much wider spread. More requests are simultaneously competing for GPU access, prefill lengths vary, and prefill and decode operations overlap. The single-concurrency case is an idealized scenario which runs one request at a time presenting the lowest possible TTFT, at the cost of throughput. Figure 6. A histogram comparing the difference in time-to-first-token distribution between a single active user and Poisson arrival pattern AIPerf runs In Figure 6, above, you can see that the single user run experiences less TTFT variability than the much more varied workload in the Poisson experiment. There’s more to explore The tutorials in the AIPerf repo are the fastest way in. The AIPerf repo and docs are the canonical reference for new features and contributions. Acknowledgments AIPerf is a collaborative effort between NVIDIA and external contributors. Thank you to the following: Loki Ravi, Dan Ferguson, and Sheng Moua (AWS) for the continual collaboration, cross-company validation, and efforts to standardize on AIPerf; Aaron Batilo (Coreweave) for the Weights & Biases exporter, acceptance-length spec-decode datasets, and hardening sweep/credit-dispatch reliability under concurrency; Shounak Ray (Baseten) for faithful Baseten trace replay support; Michael Feil (Baseten) for faster trace loading, and session affinity headers. Cristian Lopez (Pinterest) for his close collaboration on the DAG benchmarking methodology. We’re grateful to Ben Hamm for his product guidance while we designed, planned, and implemented AIPerf.