NVIDIA Details Rubin GPU Architecture and Vera Rubin Platform for Agentic AI

NVIDIA Details Rubin GPU Architecture and Vera Rubin Platform for Agentic AI NVIDIA has unveiled details of its Rubin GPU architecture and the Vera Rubin NVL72 platform, engineered to deliver up to 10x more agentic AI throughput per unit of energy than Blackwell. The design targets long-context reasoning, complex tool execution, and multistep inference via hardware improvements across compute, memory, interconnect, and rack-scale power management. Rubin features 336 billion transistors across two reticle-limited dies connected by NV-HBI, 224 SMs, 896 Tensor Cores, and up to 50 petaflops of NVFP4 performance. Integrated HBM4 provides up to 288 GB capacity and up to 22 TB/s peak bandwidth, a 2.8x increase over Blackwell and Blackwell Ultra. Inference optimizations include inline TMA descriptor updates for MoE models, doubling of Tensor Core K-dimension throughput, 3-bit LUT matrix formats, activation sparsity, and higher exponential math throughput. Interconnect and coordination upgrades feature NVLink 6 delivering 3,600 GB/s, counted writes for device-initiated transfers, and tile-level triggering for dependent kernels. Rack-level Vera Rubin NVL72 incorporates Intelligent Power Smoothing and DSX MaxLPS to allow up to 40% more GPUs within the same power envelope. Compute and Core Architecture The Rubin GPU combines two reticle-limited compute dies using the high-speed NVIDIA High-Bandwidth Interface (NV-HBI). In total, the package houses 336 billion transistors, 224 streaming multiprocessors (SMs), and 896 Tensor Cores. Powered by a third-generation Transformer Engine, Rubin outputs up to 50 petaflops of NVFP4 inference performance while supporting a 3-bit lookup-table (LUT) representation for matrix B that retains up to MXFP8 accuracy. Compute resources are managed through Graphics Processor Clusters with a large centralized L2 cache, supported by GigaThread coordination, NV-DEC decoding acceleration, MIG partitioning, and Confidential Computing with TEE-I/O. Accelerating MoE, Attention, and Matrix Operations To address agentic inference bottlenecks, Rubin introduces architectural enhancements across key algorithmic paths. For mixture-of-experts (MoE) workloads, an enhanced Tensor Memory Accelerator (TMA) allows inline descriptor updates, modifying pointers and strides at runtime rather than updating descriptors in memory. Tensor Cores double data processing throughput along the K dimension per clock, halving loop iterations and reducing overhead. In long-context attention, Rubin applies 2:4 structured activation sparsity after dense QK^T generation to reduce data movement into softmax and the second GEMM. In addition, exponential math throughput per clock per SM is doubled for FP32 and quadrupled for BF16/FP16 compared to Blackwell, preventing softmax bottlenecks. Fine-Grained Execution and High-Speed Interconnects Rubin targets kernel-to-kernel latency by moving beyond bulk programmatic launches to tile-level triggering, enabling consumer kernels to begin execution as soon as required tile activations are produced. For scale-up communications, Rubin features NVLink 6 offering 3,600 GB/s of all-to-all GPU bandwidth, NVLink-C2C providing 1,800 GB/s CPU-GPU bandwidth, and x16 PCIe Gen 6 reaching 256 GB/s. Rubin also introduces counted writes for device-initiated NVLink transfers, which reduces synchronization overhead and latency by allowing receiving GPUs to directly track payload completion while kernels remain in flight. HBM4 Memory Subsystem and Rack-Level Infrastructure To satisfy decode-phase demands, Rubin incorporates up to 288 GB of HBM4 across 12-Hi stacks, reaching 22 TB/s of peak bandwidth via a doubled interface width and dedicated controllers. At the system level, the Vera Rubin NVL72 rack incorporates 45°C liquid cooling, cable-free compute and switch trays, and hot-swappable NVLink switches. To handle power swings during dynamic AI workloads, Intelligent Power Smoothing with energy storage reduces average power consumption by approximately 10% and 50 ms peak power by 20%. Operating alongside NVIDIA DSX MaxLPS and DSX OS, the architecture recovers stranded power to enable provisioning up to 40% more GPUs within an identical power budget.

NVIDIA公布Rubin GPU架构细节：专为Agentic AI打造的高能效计算平台 NVIDIA发布了面向智能体AI（Agentic AI）工作负载的Rubin GPU架构。该架构旨在解决持续推理、多步推理规划及大上下文带来的计算与互联瓶颈，相比Blackwell平台可实现单位能耗下最高10倍的Agentic吞吐量提升，并提供高达50 PFLOPS的NVFP4推理算力。 硬件规格：集成3360亿晶体管、224个SM及896个Tensor Core，双Die通过NV-HBI互联，配备288 GB HBM4（22 TB/s带宽）与3600 GB/s的NVLink 6互联。 算力与模型优化：Tensor Core翻倍K维度指令吞吐，支持3位查找表（LUT）权重格式；TMA支持内联描述符更新以加速MoE权重与Token调度。 长上下文与内核效率：采用2:4结构化激活稀疏化压缩中间注意力分数，Softmax指数吞吐大幅提升，并引入细粒度Tile级触发消除内核调用空隙。 机架级与功耗管理：引入NVLink计数写入降低同步延迟；配合智能功率平滑技术及DSX MaxLPS，可在相同功率预算下多部署高达40%的GPU。 Rubin GPU芯片架构与硬件规格 Rubin GPU由两个受光罩限制的计算Die组成，两者通过高带宽接口NV-HBI统一封装。芯片集成3360亿晶体管、224个流式多处理器（SM）和896个Tensor Core，搭载第三代Transformer Engine，可提供最高50 PFLOPS的NVFP4推理性能。存储与互联方面，Rubin整合最高288 GB的12-Hi堆叠HBM4内存，带宽达22 TB/s（较Blackwell提升2.8倍）；同时配备提供3600 GB/s带宽的NVLink 6、1800 GB/s的NVLink-C2C及PCIe Gen 6，并支持带TEE-I/O的机密计算。 MoE与矩阵运算的执行优化 针对专家混合模型（MoE），Rubin增强了张量内存加速器（TMA），支持内联描述符更新，内核可在运行时直接覆盖内存指针和步长，减少元数据管理开销并加速专家权重的定位与调度。在通用矩阵乘法中，Rubin通过使每个时钟周期的K维度处理数据量翻倍，将原本在Blackwell上需要4次K循环的操作减少至2次，从而降低循环开销并提升Tensor Core利用率。此外，Transformer Engine新增对矩阵B的3位LUT格式支持，在保留接近MXFP8精度的同时减少权重存储与传输量。 长上下文注意力机制与内核流水线改进 为应对长上下文对注意力的计算压力，Rubin将结构化2:4激活稀疏化引入注意力流水线，将中间注意力分数压缩为非零值与元数据，降低Softmax及后续与V矩阵乘法的计算与数据传输需求。同时，Rubin大幅提升了Softmax所需的指数运算吞吐量，FP32和BF16/FP16吞吐分别达到Blackwell基准的2倍和4倍。在内核调度方面，Rubin通过Tile级触发实现了更细粒度的内核间协调，消费端内核无需等待生产端完全执行完毕即可在部分数据就绪时提前启动，有效压缩了GPU空闲间隙。 机架级通信、供电与集群可扩展性 在系统与机架层面，Rubin针对设备发起的NVLink通信引入了计数写入（counted writes），接收端GPU可直接高效跟踪传输完成状态，降低融合内核的同步延迟。Vera Rubin NVL72采用第三代MGX机架架构，集成45°C液冷和热插拔NVLink交换机托盘。供电方面，基于荷电状态（SoC）的智能功率平滑技术将平均功耗降低约10%，50毫秒峰值功率降低约20%；结合DSX MaxLPS与DSX OS自动化调度，系统能够在固定兆瓦功耗预算下使运营商部署的GPU数量增加多达40%。

What began as discrete AI model training and human-facing chat interfaces has evolved into always-on AI factories dedicated to producing intelligence at scale. These factories are now tasked with powering agentic workflows that reason, plan, use tools, verify intermediate results, and execute complex multistep tasks across vast contexts. Agentic workloads are not defined by a single prompt and response, but by sustained inference across many reasoning steps. They demand low per-step latency, high decode throughput, efficient long-context attention, large KV cache capacity, and the ability to scale models across tightly coupled GPU domains. The data center must be reimagined as a single unit of compute, a vision realized with the NVIDIA Vera Rubin platform . At the core of the platform is the NVIDIA Rubin GPU, designed to deliver up to 10x more agentic throughput per unit of energy than NVIDIA Blackwell (Figure 1). Enhanced Tensor Cores with expanded precision flexibility, a new HBM4 memory subsystem, and the third-generation Transformer Engine—delivering up to 50 petaflops of NVFP4 performance—work together to accelerate agentic workloads efficiently. Figure 1. Pareto frontiers illustrating the 10x generational uplift in agentic inference performance of the Vera Rubin platform (internal 2T MoE workload) This post examines how the NVIDIA Rubin GPU and its co-designed scale-up system address the end-to-end bottlenecks of agentic inference, from data movement and compute efficiency to long-context execution and rack-scale deployment. How does Rubin GPU architecture support agentic workloads? The Rubin GPU (Figure 2) is constructed from reticle limited compute dies to achieve high density and efficiency. These two dies are unified on a single package through a high-speed inter-die link called the NVIDIA High-Bandwidth Interface (NV-HBI). Figure 2. NVIDIA Rubin GPU chip architecture The architecture begins with the challenge of keeping an enormous amount of compute productive as agentic workloads shift between reasoning, generation, retrieval, and tool use. Its 336 billion transistors, 224 streaming multiprocessors (SMs), and 896 Tensor Cores provide the raw compute density, while the third-generation Transformer Engine adapts precision across numerical formats. That flexibility enables the Rubin GPU to deliver up to 50 petaflops of NVFP4 inference performance while preserving accuracy. But performance depends on more than Tensor Core throughput. The Rubin GPU organizes compute resources into Graphics Processor Clusters (GPCs) with a large centralized L2 cache. The GigaThread Engine coordinates work, MIG Control partitions the GPU for multiple workloads, and NV-DEC accelerates decoding. Together, these capabilities help the Rubin GPU turn compute density into sustained utilization across the diverse and dynamic workloads that define large-scale agentic systems. That utilization also depends on how quickly data can reach compute cores. Rubin integrates up to 288 GB of HBM4 memory, driven by dedicated HBM controllers and 12-Hi stacks, to deliver up to 22 TB/s of peak bandwidth. The enhanced Tensor Memory Accelerator (TMA) manages high-efficiency movement across complex data layouts, while NVIDIA NVLink 6 provides 3,600 GB/s scale-up bandwidth to the NVLink Switch for all-to-all GPU-GPU communication, NVLink-C2C delivers 1,800 GB/s for coherent CPU-GPU communication, and x16 PCIe Gen 6 provides up to 256 GB/s of host connectivity. Finally, large-scale agentic deployments must protect data as it moves through this execution domain. Confidential Computing with TEE-I/O is designed to secure data at rest, in transit, and in use across the AI factory. Together, these compute, memory, connectivity, and security capabilities form the GPU-level foundation for agentic workloads that must sustain efficient execution across increasingly large models and scale-up domains. How does the Rubin GPU accelerate critical inference paths? Peak compute alone is not enough to accelerate agentic inference. Real-world performance also depends on how efficiently the GPU moves data, executes matrix operations, processes long-context attention, and transitions between dependent kernels. This section explores the Rubin GPU features designed to reduce overhead in these critical execution paths. Accelerating rack-scale MoE weight and token movement Mixture-of-experts (MoE) models dynamically route tokens across many expert networks. As the number of experts grows, efficiently locating and moving expert weights becomes increasingly important to inference performance. The Rubin GPU enhances the Tensor Memory Accelerator to reduce data-movement overhead for expert-heavy models. Its improved descriptor handling allows software to work more efficiently with tensors that share common layouts but reside at different locations in memory. Rubin improves this with inline descriptor update support for TMA (Figure 3). Instead of modifying the descriptor in memory, the kernel can keep one unified descriptor for tensors that share the same layout and override fields such as the memory pointer and stride directly in the TMA instruction at runtime. Figure 3. Rubin simplifies MoE descriptor sharing This helps MoE models scale more efficiently as expert counts increase. By reducing metadata-management and data-movement overhead, Rubin enables more GPU time to be devoted to useful inference computation, supporting higher throughput for agentic workloads that rely on large MoE models. Doubling the efficiency of rack-scale matrix operations Rubin doubles the Tensor Core throughput per clock by doubling the amount of data it can process along the \(K\) dimension. This optimization not only helps throughput bound kernels but also memory and latency bound kernels. As model execution is split across many GPUs, each GPU often receives a smaller slice of the output work while the reduction dimension remains large. The benefit of a larger \(K\) dimension is fewer \(K\) loop iterations. In Figure 4, a GEMM that requires four \(K\) iterations on Blackwell can be completed with two iterations on Rubin. Fewer iterations reduce loop overhead, improve Tensor Core utilization, and help both context and decode GEMMs run more efficiently at high tensor-parallel scale. Figure 4. Rubin doubles K-dimension instruction throughput For models that benefit from even more compact weight representations, the Rubin Transformer Engine also supports a 3-bit lookup-table format for matrix B. Rather than store each weight directly, the format stores a 3-bit index into a small table of representative values and Rubin Tensor Cores resolve these indices inline to reduce weight storage and data movement. LUT-based representations can retain up to MXFP8 accuracy, adding another precision option to the Rubin inference toolbox. Addressing key challenges in long-context processing for agentic AI Long-context and agentic AI workloads put increasing pressure on attention. As context windows grow, the model must compare more tokens, normalize larger attention-score matrices, and apply those scores to the value data used to produce the next layer’s output. This makes attention one of the most important performance paths for improving tokens per second per user. Rubin accelerates attention by combining activation sparsity with adaptive compression and improved softmax throughput. One simple, safe, and effective way to use Rubin’s new sparsity features in attention is as follows. The attention pipeline begins with a dense \(QK^T\) computation to generate the intermediate attention scores. Rubin can then load that intermediate data from Tensor Memory into a structured 2:4 sparse compressed form, generating both nonzero values and the metadata needed to use them efficiently while reducing the scores’ write cost and storage requirements. This allows the later attention stages to operate on less data while preserving the dense output format expected by the rest of the model. Figure 5. Rubin sparsity features can be applied to the attention block and MLP block activations This compressed intermediate representation reduces work in two key places: softmax and the second attention GEMM. Softmax can operate on the nonzero attention values, and the following multiplication with the dense \(V\) matrix can use sparse MMA with the nonzeros from Softmax and the metadata from the original compression step. The result is less compute and data movement in the most expensive part of long-context attention improving tokens/watt, without requiring the surrounding model pipeline to change its interface. Rubin also improves the softmax throughput. As Tensor Core throughput increases, softmax can become a bottleneck because it depends on exponential math and reductions across attention rows. Rubin increases exponential throughput, including 2x FP32 and 4x BF16/FP16 throughput versus the Blackwell baseline, helping softmax keep pace with faster matrix operations. NVIDIA GPU platform FP32 exponential throughput (per clk per SM ) BF16/FP16 exponential throughput (per clk per SM) Blackwell 1x 1x Blackwell Ultra 2x 2x Rubin 2x 4x Table 1. Exponential throughput rises from Blackwell to Rubin Together, these features make the Rubin attention acceleration more than a single-kernel improvement. Activation sparsity reduces the amount of intermediate attention work, while faster exponentials reduce the softmax bottlenecks. Improving kernel execution efficiency As inference scales across larger models and more GPUs, raw Tensor Core throughput is only part of the performance story. The GPU also needs to move efficiently from one kernel to the next. In inference, this is especially important because activations often sit on the critical path: one kernel produces activation data, writes it to memory, and the next kernel consumes that data to continue generating the next token. Traditional producer-consumer execution can create bubbles in the GPU timeline. A producer kernel may complete work for some tiles or thread blocks early, but the consumer may not begin useful work until a broader dependency is resolved. The Blackwell programmatic dependent launch improves this by allowing earlier consumer-kernel progress, but dependent work can still wait for required activation data to become available. Figure 6. Producer-consumer overlap: Blackwell bulk triggering (above) and Rubin tile-level triggering (below) Rubin enables more fine-grained coordination between dependent kernels. This allows consumer work to begin earlier as required input data becomes available, rather than waiting for a larger set of producer work to complete. The result is a more tightly packed GPU timeline, reduced idle gaps, and improved overlap between dependent kernels. This is particularly valuable for agentic inference, where activations move sequentially through the model and kernel-to-kernel latency directly affects tokens per second per user. How do Rubin memory and communication sustain high-throughput inference? As models, context windows, and GPU domains grow, data movement becomes as important as computation. Rubin is designed to improve the flow of weights, activations, KV cache data, and communication traffic within the GPU and across scale-up systems. The following sections look at the memory and communication innovations that help sustain high-throughput inference. Accelerated scale-up communications As inference scales from a single GPU to full rack-level systems, communication becomes part of the critical performance path. When communication is fused directly inside a GPU kernel, the kernel does not stop and hand control back to the CPU; it directly writes data or performs reductions over NVLink to another GPU while computation is still in flight. Traditional GPU-to-GPU communication requires coordination and synchronization work in addition to moving payload data. These steps can add latency and consume interconnect bandwidth, particularly when communication occurs frequently within distributed inference workloads. Rubin introduces counted writes for device-initiated NVLink communication. This capability streamlines synchronization for GPU-to-GPU data transfers by allowing the receiving GPU to track transfer completion more efficiently. Figure 7. Rubin accelerates NVLink communications using counted writes NVLink fused communication with counted writes provides a lower-latency mechanism for coordinating GPU-to-GPU data movement, helping keep computation moving instead of waiting on synchronization operations. Co-designing a memory subsystem for maximum power and compute efficiency The decode, or generation, phase of inference is fundamentally memory subsystem bound. It is not about peak bandwidth specs but rather how efficiently each kernel can utilize the entire memory subsystem. Modern reasoning and agentic workloads amplify this constraint by spending more end-to-end runtime in decode: long contexts, large KV caches, and interactive token generation make achieved memory bandwidth a critical performance lever. Figure 8. Memory bandwidth scaling from Blackwell to Rubin Rubin addresses this with HBM4 combined with a highly efficient localized memory subsystem. HBM4 doubles interface width relative to HBM3e. Combined with a new memory controller, deep co-engineering with the memory ecosystem, and tighter compute-memory integration, this subsystem provides up to 22 TB/s of memory bandwidth: a 2.8x increase over Blackwell and Blackwell Ultra. Rubin also provides up to 288 GB of HBM4 per GPU, increasing available on-package capacity relative to Blackwell. Capacity and bandwidth play different but complementary roles: Capacity: Supports model residency, larger context windows, larger KV caches, and higher concurrency without unnecessary KV-cache offload. Bandwidth: Supports the token-by-token generation phase, during which model weights and KV state must be moved rapidly enough to keep compute engines productive. Memory subsystem: TMA and memory-locality strategies help software use the memory subsystem efficiently, supporting high achieved bandwidth for complex data layouts. High-capacity, high-bandwidth HBM4 is critical for hosting multitrillion-parameter models, extending context length without KV cache offload, and supporting high-concurrency, long-horizon inference workloads. How is NVIDIA designing for efficiency, scalability, and reliability? Agentic AI infrastructure must optimize more than individual GPUs; it must make effective use of power, cooling, networking, and rack-level resources across the entire AI factory. NVIDIA Vera Rubin NVL72 extends the GPU architecture into an integrated, resilient rack-scale execution domain. This section examines how NVIDIA approaches power efficiency, operational scalability, and system-level reliability. Increasing the number of GPUs within the same power budget Agentic AI makes energy efficiency an AI-factory problem, not simply a GPU power problem. At rack scale, Vera Rubin NVL72 integrates compute, networking, liquid cooling, power steering, and Intelligent Power Smoothing with energy storage into a single execution domain designed to maximize useful token output within a fixed power envelope. Figure 9. GPU power plot showing impact of power smoothing techniques Power demand can vary sharply during AI workloads, creating transient peaks that strand available capacity and reduce factory-level throughput. Vera Rubin power supplies use state-of-charge (SoC) Intelligent Power Smoothing to absorb these swings, reducing average power consumption by approximately 10% compared with previous-generation power-smoothing techniques and reducing 50 ms peak power by approximately 20%. By delivering a steadier power profile, the system can reduce sustained maximum power requirements, benefit supporting power infrastructure and the grid, and enable more compute within the same AI factory power budget. At the AI-factory level, NVIDIA DSX MaxLPS extends this approach across GPUs, racks, 45°C liquid cooling, and workloads, while DSX OS provides the operational layer for scheduling, lifecycle management, and health automation. Together, they are designed to recover stranded power and increase useful compute within a fixed megawatt envelope. DSX MaxLPS can enable operators to provision up to 40% more GPUs (Figure 10) within the same power budget at energy-efficient operating points, with minimal impact on workload performance. Figure 10. For the same power budget, DSX MaxLPS enables up to 40% more GPUs Designing racks for resiliency and data center scalability Vera Rubin NVL72 extends the Rubin GPU into a rack-scale execution domain for multitrillion-parameter and multi-rack agentic workloads. Its third-generation MGX rack architecture combines cable-free compute and switch trays, 45°C liquid cooling, dynamic rack-scale power steering, and Intelligent Power Smoothing to keep compute, networking, cooling, and power operating as one system. Hot-swappable NVLink switch trays and improved RAS capabilities further support resilient operation at scale. This rack design is the foundation for pod-scale, multi-rack agentic systems, with open and flexible connectivity across NVLink and NVIDIA Spectrum-X Ethernet . Learn more The NVIDIA Rubin GPU is designed for the execution patterns of agentic AI, where long-context reasoning, multistep generation, distributed MoE decode, and low-latency interaction must operate continuously at scale. Its compute, memory, networking, rack-scale, power, cooling, and software layers are carefully co-designed to keep more of the AI factory performing useful work rather than waiting on data, communication, or power constraints. The result is an architecture built to maximize agentic performance per watt—producing more useful tokens and completed AI work within a fixed power envelope. For a deeper look at the rack architecture and the broader NVIDIA Vera Rubin platform, see Inside the NVIDIA Vera Rubin Platform: Six New Chips, One AI Supercomputer . To learn more about NVIDIA Vera Rubin POD, see NVIDIA Vera Rubin POD: Seven Chips, Five Rack-Scale Systems, One AI Supercomputer . Acknowledgments This work was made possible through the expertise and engineering contributions of Jeff Pool, Ran Zilberstein, Ronny Krashinksky, Sailaja Madduri, Shivam Raj, Xiaowei Wang, Manas Mandal, Bita Darvish, Raj Dash, and many other talented NVIDIA engineers.