NVIDIA Evaluates Confidential Computing Adaptations and Overhead for Blackwell AI Inference

NVIDIA Evaluates Confidential Computing Adaptations and Overhead for Blackwell AI Inference NVIDIA detailed how runtime adaptations in frameworks like TensorRT LLM enable private production AI inference on Blackwell GPUs while preserving performance. Controlled benchmarking on an eight-B200 system demonstrated that confidential computing maintained over 96% of baseline output token throughput. Tests running DeepSeek-R1-0528-NVFP4 with 32K input/1K output contexts showed CC retained 96.1% to 98.2% of non-CC throughput across concurrency levels 1 to 16. Mean Time Per Output Token (TPOT) overhead remained within 1.2% to 4.3% compared to baseline CC-off execution. Runtime adaptations addressed host-to-device bounce buffer bottlenecks by selecting pageable memory and offloading readbacks to asynchronous workers. TensorRT LLM switched tactic timing to the GPU globaltimer to stabilize kernel autotuning, and adjusted multi-GPU collectives due to the absence of NVLS multicast under CC. Benchmarking Methodology and Results To isolate the performance effects of confidential computing, the engineering team compared CC-on and CC-off configurations holding the model, hardware, software, sequence length, and concurrency constant. The evaluation ran on an NVIDIA DGX B200 system with Intel TDX using the DeepSeek-R1-0528-NVFP4 model configured with 32K input and 1K output tokens. Evaluated across concurrency levels from 1 to 16, CC-on maintained 96.1% to 98.2% of baseline output-token throughput, while the Time Per Output Token overhead remained between 1.2% and 4.3%. Runtime and Framework Mitigations Secure execution alters memory and communication paths, prompting specific mitigations in TensorRT LLM. Because transfers must pass through software-encrypted bounce buffers, the runtime selects pageable memory instead of pinned memory for affected paths and offloads token and sampling readbacks to asynchronous workers to prevent scheduler stalls. In addition, kernel autotuning switched from unstable CUDA-event timing to the GPU globaltimer, and multi-GPU communication strategies were adjusted to avoid unnecessary registration overhead given that NVLink SHARP multicast is unavailable in B200 CC configurations.

NVIDIA 机密计算支持 Blackwell GPU 高性能 AI 推理与性能优化分析 随着大语言模型处理敏感数据需求增加，NVIDIA 通过机密虚拟机、机密 GPU 和加密 NVLink 提供机密计算环境。为解决安全执行引入的通信、调度与内存开销，TensorRT LLM 针对 Blackwell B200 系统进行了特定优化，在维持极高吞吐的同时将延迟开销控制在低水平。 在配备 8 张 B200 GPU 的 DGX 系统上运行 DeepSeek-R1-0528-NVFP4 模型测试，并发 1 至 16 情况下机密计算（CC on）保留了 CC off 基准 96.1% 至 98.2% 的输出吞吐量。 机密计算环境下的每个输出 token 时间（TPOT）平均开销较无机密计算状态仅增加了 1.2% 至 4.3%。 针对机密虚拟机无法直接访问 GPU 内存导致的加密中转缓冲问题，TensorRT LLM 采用分页内存及异步工作线程回读策略以消除主调度阻塞。 针对 CC 环境下 CUDA 事件时间戳不稳定的现象，内核自动调优器改用 GPU %globaltimer 计时，确保选择最优执行策略。 测试方法与量化开销表现 为了充分暴露机密计算的开销，测试团队选取了 32K 输入与 1K 输出的长上下文、长生成及 1 至 16 低并发负载。硬件采用运行于 Intel TDX 平台上的 NVIDIA DGX B200 系统，严格保持模型、框架版本、并行度等环境变量一致。结果显示，CC on 状态下系统保留了 96.1% 至 98.2% 的输出吞吐量，且平均 TPOT 开销限制在 1.2% 至 4.3% 之内。 数据传输与调度机制适配 B200 机密计算的主机到设备数据传输需经过软件加密缓冲，锁定内存不再具备传统异步优势。TensorRT LLM 为此采用感知 CC 的内存选择机制，在受影响路径上改用分页内存；同时将重复的 token 和采样数据设备到主机回读迁移至异步工作线程，防止解密拷贝阻塞解码阶段的主调度器。 调优计时与多 GPU 通信调整 在 CC 运行环境下，传统 CUDA 事件时间戳信号出现不稳定，易误导内核自动调优器选取次优策略，优化方案是在 CC 环境下改用 GPU %globaltimer 进行策略度量。此外，鉴于 B200 CC 无法使用 NVLS 多播导致 NCCL_SYMMETRIC 失去多播增益，推理框架通过自动识别 NVLS 可用性并选用适合当前拓扑与数据尺寸的通信算法，降低跨卡同步开销。

As large language model (LLM) inference increasingly processes sensitive information and proprietary model context across personal, enterprise, and regulated settings, data must be processed inside a trusted environment. NVIDIA Confidential Computing (CC) provides a pathway for running these workloads securely using memory-encrypted confidential virtual machines (CVMs), confidential GPUs, and encrypted NVIDIA NVLink . This enables running production AI inference on trusted hardware. Inference frameworks such as NVIDIA TensorRT LLM deliver best-in-class AI inference by combining framework-level optimizations with NVIDIA accelerated computing. However, when these frameworks run in a CC-enabled environment, secure execution changes assumptions behind memory movement, timing, scheduling, and multi-GPU communication. These changes introduce performance overhead if the runtime does not adapt. Maintaining high performance therefore requires the inference framework and the confidential computing environment to be optimized together. For AI platform engineers evaluating confidential inference on NVIDIA Blackwell GPUs, this post examines the CC-aware adaptations that AI inference frameworks like TensorRT LLM use to account for secure execution while helping preserve inference performance. It presents a controlled methodology that teams can apply to quantify CC overhead on their own workloads. Selecting a workload to expose CC overhead Workload characteristics determine how visible CC overhead can be. High request volume can amortize fixed encryption costs by overlapping stalls with other work, making the direct effects harder to observe. To expose these effects, select a workload with a long input context, extended output generation, and low concurrency. Long context stresses data movement during prefill, extended generation amplifies small per-token CC overhead during decode, and low concurrency limits the opportunity to hide those costs across concurrent requests. The NVIDIA performance engineering team used these characteristics for the workload evaluated here. Parameter Configuration Model nvidia/DeepSeek-R1-0528-NVFP4 Inference framework TensorRT LLM, PyTorch backend I/O sequence length 32K input/1K output Concurrent requests 1, 2, 4, 8, and 16 Parallelism TP=8, EP=1, PP=1 KV cache FP8 Table 1. Workload configuration for the CC on and CC off comparison Measuring CC overhead with a controlled CC-on and CC-off comparison To isolate the performance impact of CC, run the same workload under two conditions: confidential compute disabled (CC off) and confidential compute enabled (CC on) holding the model, hardware, framework version, sequence lengths, parallelism, and concurrency constant so that CC state is the only changing variable. At each concurrency level: Output throughput retained : 100 x (CC on output tokens/s ÷ CC off output tokens/s) Latency overhead Time Per Output Token (TPOT): 100 x (CC on TPOT ÷ CC off TPOT − 1) Performance teams can use these measurements to quantify how much of the CC-off baseline is retained when CC is enabled for the target workload. The NVIDIA performance engineering team applied this comparison using the hardware and software configuration summarized in Table 2. Component Version / Detail Hardware 1 NVIDIA DGX B200 system ( 8 NVIDIA B200 GPUs) Platform Intel TDX Host OS Ubuntu 25.10 Host kernel 6.17.0-20-generic Guest OS Ubuntu 24.04.4 LTS Guest kernel 6.8.0-124-generic Guest vCPUs 256 Guest NUMA 2 nodes NVIDIA driver 595.71.05 VBIOS FW 1.4.x [97.10.64.00.0C] GPU power limit 1,000 W CUDA 13.2 TensorRT LLM nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22 NCCL v2.30 OpenSSL 3.6.0 Orchestration Docker Container + NVIDIA Container Toolkit Table 2. Hardware and software configuration used for both CC-on and CC-off runs Performance results As shown in Figures 1 and 2, across concurrency 1–16, CC on retained 96.1– 98.2% of CC off output-token throughput, while mean TPOT remained within 1.2% to 4.3% of the baseline. Figure 1. CC on output-token throughput relative to the CC off baseline at concurrency 1–16. CC on retained 96.1%  to 98.2% of baseline throughput Figure 2. Mean TPOT with CC enabled relative to the CC off baseline at concurrency 1–16. CC on introduced 1.2%  to 4.3% TPOT overhead (lower TPOT is better) Identifying and reducing CC overhead The NVIDIA Blackwell confidential computing architecture introduces hardware-enforced security paths for protecting data and workloads in use. For a detailed overview of the architecture, see Hardware-Rooted AI Security That Won’t Slow You Down . For TensorRT LLM users and framework developers, the following details show how these secure paths change common runtime assumptions and how TensorRT LLM adapts to reduce the resulting performance overhead. Adapting host-to-device data movement In the B200 CC, host-to-device transfers pass through a software encrypted bounce buffer because the GPU cannot directly access protected CVM memory. This changes the behavior expected by inference frameworks: pinned memory no longer provides its usual asynchronous-transfer advantage, and some copies can block the calling thread. Host-to-device mitigation : TensorRT LLM uses CC-aware memory selection, choosing pageable memory for affected paths instead of unconditionally using pinned memory. Device-to-host mitigation : TensorRT LLM moves repeated token and sampling-data readback to an asynchronous worker, preventing protected copies from blocking the main scheduler during decode. For details, see TensorRT LLM PR #11573 . Stabilizing kernel autotuner timing The kernel autotuner normally uses CUDA events to compare candidate tactics. In the tested CC configuration, CUDA-event timestamps produced an unstable timing signal, which could cause the autotuner to select a slower tactic. Mitigation : TensorRT LLM uses the GPU %globaltimer for tactic measurements under CC while retaining CUDA events outside CC. For details, see TensorRT LLM PR #11657 . Choosing CC-aware multi-GPU communication NVLS (NVLink SHARP) multicast is not available in B200 CC configurations. Without NVLS, NCCL_SYMMETRIC cannot provide its intended multicast benefit but may still incur memory registration and cross-rank synchronization costs before using a non-multicast collective path. Mitigation : Frameworks targeting CC should detect NVLS availability and choose communication algorithms that minimize latency for the given message size, topology, and workload characteristics. Get started with NVIDIA Confidential Computing NVIDIA Confidential Computing extends hardware-enforced protection across confidential VMs, NVIDIA Blackwell GPUs, and encrypted NVLink, protecting proprietary models, enterprise context, and sensitive prompts while they are processed. Confidential computing does not remove the need for performance engineering—it makes framework awareness even more important. With TensorRT LLM CC-aware adaptations to secure data movement, autotuning, and multi-GPU communication in place, confidential DeepSeek-R1 inference retained more than 96% of CC-off output-token throughput while keeping per-token latency overhead below 5% on eight NVIDIA B200 GPUs. As organizations move private inference into production, security configuration and inference optimization should be approached as a single deployment problem. Enable confidential computing, attest the environment, and benchmark CC-on and CC-off using the exact workload you intend to serve. For AI platform engineers and TensorRT LLM users moving private inference into production, security configuration and inference optimization should be treated as a full-stack engineering effort. To start planning your confidential inference deployment, use the NVIDIA Trusted Computing documentation and explore the latest TensorRT LLM features and release notes . To stay up to date with the latest developments, follow NVIDIA Confidential Compute news . Acknowledgments I would like to thank Dan Hansen, Sheel Pethe, Samuel Mendoza-Jonas, Moein Ghaniyoun, Vidhya Krishnan, Avinash Ahuja, Laikh Tewari, Laura Martinez, and Matheen Raza for their engineering contributions, technical guidance, analysis, and thoughtful review throughout this work.