NVIDIA Details Sixth-Generation NVLink Scale-Up Fabric for AI Factories

NVIDIA Details Sixth-Generation NVLink Scale-Up Fabric for AI Factories NVIDIA outlined its scale-up networking strategy for AI factories, highlighting the sixth-generation NVLink interconnect and NVLink 6 Switch designed to treat clustered GPUs as a unified compute engine. Sixth-generation NVLink in the Vera Rubin NVL72 platform provides 3.6 TB/s bidirectional bandwidth per GPU, 260 TB/s aggregate rack bandwidth, and 130 TFLOPS of FP8 in-network compute. NVLink achieves 3X lower end-to-end latency, 10X higher packet rates, and up to 2.3X higher decode throughput for large MoE models compared to off-the-shelf Ethernet in simulations. NVLink 6 introduces intelligent resiliency features, including hot-swappable switch trays, control plane resilience, dynamic rerouting, and operation with partially populated racks. NVLink-C2C links Vera CPUs and GPUs with 1.8 TB/s of coherent bandwidth, while NVLink Fusion enables third-party custom XPUs to integrate directly into the NVLink ecosystem. Scale-Up Networking Economics and Performance Scale-up fabrics allow accelerators in a single domain to operate as a unified compute unit with predictable low latency and shared memory, which NVIDIA identifies as critical for mixture-of-experts (MoE) workloads and large-scale inference. In the Vera Rubin NVL72 rack configuration, sixth-generation NVLink uses NVLink Switch trays and a 5,000-cable spine to establish an all-to-all topology across 72 GPUs, delivering 3.6 TB/s per GPU and 130 TFLOPS of SHARP in-network compute. NVIDIA reports that the architecture reduces GPU-to-GPU latency by 3X and raises packet rates by 10X compared to standard Ethernet, delivering up to 2.3X higher decode throughput in simulations of models like DeepSeek-R1 and Qwen 235B. Rack Resiliency and Hardware-Software Co-Design To prevent downtime in production AI factories, NVLink 6 incorporates hardware and control-plane resiliency mechanisms. These include hot-swappable switch trays, dynamic traffic rerouting, in-service software updates, fine-grained telemetry, and support for operating within partially populated racks. The interconnect is co-designed with a software stack comprising Dynamo, TensorRT-LLM, NIXL, and NCCL built on CUDA, which NVIDIA notes facilitated a 50X improvement in MoE inference performance per watt from the Hopper to Blackwell generations. Expanding Fabric Connectivity with NVLink-C2C and Fusion The scale-up architecture extends beyond GPU clusters through NVLink-C2C, which delivers 1.8 TB/s of coherent bidirectional bandwidth between Vera CPUs and GPUs—seven times the bandwidth of PCIe Gen6—to support agentic pipelines and KV-cache offloading. Additionally, NVIDIA introduced NVLink Fusion, an interconnect technology and IP offering that allows cloud providers and chip designers to connect semi-custom XPUs directly into the NVLink scale-up platform and rack infrastructure.

英伟达推出第六代NVLink：面向AI工厂的Scale-Up互连架构 英伟达发布了第六代NVLink及其NVLink 6 Switch交换芯片，将其定义为AI工厂的纵向扩展（scale-up）网络核心。该架构旨在解决万亿参数模型与专家混合（MoE）架构下加速器全互连的高带宽与低延迟需求，兼顾机架级生产弹性与供应链成熟度。 在Vera Rubin NVL72平台中，第六代NVLink为每颗GPU提供3.6 TB/s双向带宽，单机架聚合带宽达260 TB/s并提供130 TFLOPS的网内计算算力。 相比现成商用以太网，第六代NVLink端到端延迟降低3倍，数据包速率提升10倍，在DeepSeek-R1和Qwen 235B等大模型模拟中实现最高2.3倍的解码吞吐量。 引入热插拔交换机托盘、非满配机架运行、动态流量重路由等智能弹性特性，保障生产环境高可用性与持续产出（goodput）。 NVLink-C2C为Vera CPU与GPU提供1.8 TB/s一致性互连，同时推出NVLink Fusion IP，允许第三方定制XPU接入NVLink生态。 Scale-Up网络决定AI工厂经济效益 针对超万亿参数模型与MoE架构，算力集群需要大量加速器协同作为单一计算单元运作。MoE推理的专家并行会引发密集的全对全（all-to-all）通信。如果互连网络存在高延迟或低带宽，专家并行的优势将被通信开销抵消。模拟测试显示，在72个加速器的scale-up域中，NVLink相比现成商用以太网使DeepSeek-R1、Qwen 235B及2T参数模型的解码吞吐量提升高达2.3倍，从而降低每Token成本并提升每瓦Token产出。 第六代NVLink与Vera Rubin NVL72性能规格 在Vera Rubin NVL72机架中，NVLink 6通过5000条线缆的NVLink spine与交换托盘构建单一大拓扑，为72颗GPU提供每颗3.6 TB/s的双向带宽，机架聚合带宽达260 TB/s。每个交换托盘集成4颗NVLink 6芯片，具备28.8 TB/s托盘带宽及14.4 TFLOPS的FP8网内计算能力，整机柜提供130 TFLOPS网内计算以加速规约等集合通信。相比商用以太网方案，其GPU端到端通信延迟降低3倍，数据包转发率提高10倍。未来技术路线图计划将scale-up域扩展至1152颗GPU并支持共封装光学（CPO）。 机架级智能弹性保障生产可用率 评估scale-up方案不能仅看纸面峰值带宽，更取决于长期稳定产出（goodput）。第六代NVLink交换机增加了多项面向数据中心运维的弹性设计，包括控制平面容错、支持部分填充（非满配）机架运行、热插拔交换机托盘、带管理控制器故障转移的软件定义路由、动态流量重路由、不中断服务在线软件更新，以及细粒度链路遥测。这些机制保证了单个链路、交换机或管理组件故障时，机架仍能持续提供生产服务。 软硬件协同与生态延伸：C2C与NVLink Fusion NVLink依托CUDA生态，深度整合Dynamo、TensorRT-LLM、NIXL及NCCL等开源框架实现软硬件协同设计。在互连扩展方面，NVLink-C2C在配备88个定制Olympus核心的Vera CPU与GPU之间建立1.8 TB/s一致性带宽，达到PCIe Gen6的7倍，优化了KV缓存卸载与数据编排。此外，新推出的NVLink Fusion向云厂商及AI企业开放互连IP，使其自研定制XPU能直接接入NVLink机架架构，降低异构部署与供应链风险。

The demand for AI continues to accelerate. Workloads are getting larger, models are becoming more complex, and there is mounting pressure to deploy AI compute infrastructure faster than ever. AI factories —data center-scale systems that continuously convert data and energy into intelligence—are being deployed to meet this insatiable demand. This AI factory approach to the data center has fundamentally changed system design and operation. Peak accelerator FLOPS are no longer enough. Today’s AI workloads, including trillion+ parameter models, mixture-of-experts (MoE) architectures, long-context reasoning, and disaggregated serving, require many accelerators working together as a single unit of compute. Achieving this requires high-bandwidth, low-latency GPU-to-GPU communication, fast in-network compute for collectives, and software-aware scheduling. Given the sheer number of components, resiliency needs to be built into the entire data center. Keeping up with the pace of AI requires technology and a supply chain that can move at the speed of industry innovation. This is why scale-up networking has become one of the most important architectural decisions in the AI factory. The scale-up fabric is what enables accelerators to work as a single unit of compute, determining how effectively tokens move across experts, how quickly collective operations complete, and how much useful throughput the factory can deliver. It is a significant factor in how much risk operators take when deploying new platforms. NVIDIA NVLink is the purpose-built scale-up networking fabric for AI factories. It’s designed to accelerate AI inference, training, and other parallel computing workloads that require large, fast GPU-to-GPU  communications. The Sixth Generation NVLink interconnect with NVLink 6 Switch provides the highest  GPU-to-GPU bandwidth at the lowest latency all-to-all topology for scale-up networking, as well as support for SHARP in-network compute for offloading collective operations. It also includes rack-level resiliency features designed for production AI factory uptime. Developed through extreme co-design, where the entire stack from chips to systems to fabric to software are designed and optimized together, NVLink is part of the NVIDIA annual AI infrastructure roadmap cadence. This mature, proven, and widely deployed technology forms the backbone of modern AI infrastructure. How scale-up networking determines AI factory economics Scale-out networks connect servers across the data center. Scale-up networks enable the GPUs inside the domain to behave as a single engine of compute. Both are essential, but they solve different problems. Scale-out fabrics such as NVIDIA Quantum InfiniBand and NVIDIA Spectrum-X Ethernet enable large clusters spanning thousands to hundreds of thousands of GPUs. This is critical for large, data-parallel AI training workloads (as well as large-scale scientific simulations). Scale-up fabrics connect accelerators in a single domain with high bandwidth, predictable low latency and shared high-bandwidth memory (HBM). For modern AI training and inference, the scale-up domain is where most latency-sensitive communication patterns occur. As an example, let’s take a look at inference with MoE models. Efficient inference implementations use expert parallelism to distribute experts across GPUs, and they use large batch sizes to maximize factory throughput. This parallelizes the workload, but it creates intensive all-to-all communication between GPUs. Tokens must be dispatched to the selected experts, processed, gathered, reordered, and passed forward. All of the GPU-to-GPU communication must happen in parallel. If the experts sit behind a low-bandwidth or high-latency fabric, gains from expert parallelism can be erased by communication overhead. The same phenomenon happens during training of MoE models. The takeaway is that all-to-all bandwidth and latency are critical to AI factory performance. AI factories need purpose-built, scale-up networking that is co-designed with the rest of the system to deliver and maintain these capabilities under load, across all workloads. Scale-up networking increases ROI by optimizing the delivered tokens per watt, per dollar, and per factory square foot. It leads to shorter training runs, higher utilization, and lower cost-per-token for AI factories. NVIDIA has demonstrated the impact of high-bandwidth, low-latency scale-up networking in large-scale MoEs. For models such as DeepSeek-R1 and Qwen 235B, as well as a simulated 2T parameter LLM, NVLink delivers up to 2.3X the decode throughput compared to leading off-the-shelf (OTS) Ethernet. Figure 1. Sixth generation NVLink delivers up to 2.3X higher decode throughput compared to off-the-shelf Ethernet (both in a 72 accelerator scale-up domain at 3.6 TB/s bandwidth per accelerator ) . Results based on simulation Evaluating scale-up technologies Spec sheets often compare fabrics using simple bandwidth numbers: link rate, aggregate switch capacity, or headline bandwidth per device. Those numbers are useful, but they aren’t sufficient. Evaluating scale-up capabilities for today’s AI workloads requires taking a factory-level view. Delivered, full-system performance determines how many tokens can be processed and produced per unit time, power, and factory footprint. It depends on all-to-all fabric bandwidth, end-to-end latency (that is, latency from every GPU’s HBM memory, through the fabric, to every other GPU’s HBM memory), the in-network compute for reductions and other collectives. It also requires fully integrated software at every level that can optimize routing, expose collectives, balance link traffic, pipeline data transfers, and has full support for the libraries and frameworks used by today’s AI workloads. Delivered performance only matters to the extent that the factory is operational. The ability to run for extended periods without failure, continuously monitor system health, and support service and maintenance at the component level while the rest of the factory continues operations translates delivered performance to goodput — the capacity of a factory to produce over its full lifetime. Achieving optimal delivered performance and goodput across many workloads through a complex technology is an incredibly difficult challenge, addressed through years of learnings with widely deployed scale-up infrastructure and a robust supply chain. Using an unproven technology stack or a solution not purpose-built for scale-up networking risks sub-optimal factory performance, disruptive downtime events, and undependable supply. Together, these metrics drive the only result that matters: sustained, dependable ROI over the lifetime of the AI factory. The Key Metrics for Scale-Up Networking in AI Factories Delivered Performance Factory Resiliency Platform Maturity and Proven Supply Chain Delivered factory performance on today’s AI workloads starts with bandwidth and latency, but also includes end-to-end scale-up network performance, in-network compute for reductions and other collectives, and a mature, full-stack software implementation integrated through every part of the solution. Translating delivered performance to goodput requires system-wide resiliency, including long up-time, continuous system health-monitoring and telemetry, and support for service and maintenance at the component level while the rest of the factory continues to run. Scaling AI factories is an incredibly complex challenge with many dependencies. Operators want to minimize risk by leveraging a mature technology stack with a proven extended track record of large-scale deployments and realized ROI. Table 1. These three key metrics are critical for evaluating a scale-up networking solution. NVLink: Performance, resiliency, maturity Now in its sixth generation, NVLink  provides the delivered performance and resiliency needed to maximize factory goodput, and over ten years of scale-up technology investment and proven deployments including world-leading hyperscalers, Cloud Service Providers (CSPs), and supercomputing centers. World-leading performance With Vera Rubin NVL72 , sixth generation NVLink provides 3.6 TB/s per GPU of bidirectional GPU-to-GPU bandwidth and 260 TB/s of rack-level GPU bandwidth in a 72-GPU domain. The end-to-end latency for GPU-to-GPU transfers is 3X lower than alternative solutions based on off-the-shelf Ethernet, and the packet rate is 10X higher. 3X Lower latency 10X Higher packet rate 130 TFLOPS In-network compute 6th generation NVLink delivers lower latency and higher packet rates for GPU-to-GPU transfers compared to alternative solutions based on off-the-shelf Ethernet, as well as in-network compute capabilities for collective operations NVLink Switch trays and NVLink spine of 5,000 cables form a single all-to-all topology so any GPU can communicate with any other GPU with uniform latency and bandwidth. Each tray includes four NVLink 6 switch chips, 28.8 TB/s of total tray bandwidth, and 14.4 TFLOPS of FP8 in-network compute. In a single Vera Rubin NVL72 rack, NVLink 6 provides 260 TB/s of aggregate bandwidth and 130 TFLOPS of in-network compute for accelerating reductions and other collective operations (e.g. all-reduce, reduce, broadcast, etc). The NVLink technology roadmap includes support for scale-up domain sizes up to 1152 GPUs and connectivity through co-packaged optics. Figure 2. The Vera Rubin NVL72 NVLink Spine, rack, and the NVLink 6 Switch trays provide 260 TB/s of all-to-all bandwidth (3.6 TB/s per GPU), and 130 TFLOPS of in-network compute Software is a critical piece of the scale-up networking stack and includes NVIDIA Dynamo, NVIDIA TensorRT-LLM, NVIDIA Collective Communications Library (NCCL) and more (discussed in detail in the section on platform maturity), all built on top of NVIDIA CUDA , the world-leading parallel computing platform, first released nearly 20 years ago The hardware and software are developed with extreme co-design . This results in performance speedups that can’t be achieved through siloed stack elements. For today’s AI workloads, this translates directly into infrastructure economics. As an example, in the transition from NVIDIA Hopper to NVIDIA Blackwell, which included doubling the NVLink bandwidth, expanding the size of the NVLink scale-up domain to 72 from 8 GPUs, and incorporating Dynamo for disaggregated inference, NVIDIA achieved a 50X improvement in MoE inference performance per watt. The NVIDIA Vera Rubin platform further extends this by doubling both NVLink bandwidth and in-network compute. Figure 3. By connecting 72 GPUs in a high-bandwidth, low-latency scale-up domain with in-network compute NVLink enables GB300 NVL72 to achieve a 50X improvement in tokens/watt versus H200 As model architectures evolve, the hardware fabric and software communication stack evolve together. This extreme co-design approach to development requires tight collaboration across the entire stack. It’s not feasible to replicate with a siloed approach to development, especially under hyperscale deployment pressure, but it’s the difference between just adding accelerators versus scaling to useful delivered performance. Built for speed and intelligent resiliency AI factories must run continuously. As racks grow denser and more valuable, serviceability and fault management become part of the performance equation. A fabric that delivers high bandwidth but requires disruptive maintenance can reduce effective capacity and revenue. NVLink 6 introduces management and intelligent resiliency features designed for AI factory operations. The features make rack maintenance easier, provide better insight into component performance and load balancing, and keep the factory running even if individual nodes require service or updates. The sixth generation NVLink Switch supports intelligent resiliency features to maximize uptime and goodput: Control plane resilience Support for operation with partially populated racks Hot-swappable switch trays Software-defined routing with management controller fallback Dynamic traffic rerouting In-service software updates Fine-grained link telemetry for monitoring and fault attribution These capabilities aren’t secondary. In a production AI factory, a failed link, switch, tray, or management controller shouldn’t force an entire rack out of service. Operators need fault isolation, telemetry, and service procedures that match the economic value of the infrastructure. NVLink 6 brings scale-up networking into the same operational discipline expected from the rest of the AI factory stack. Mature technology on a fast cadence The strongest technology strategies combine maturity with pace. NVLink has both. NVLink is now in its sixth generation of purpose-built scale-up networking fabric. It has been production-deployed at scale for nearly a decade, with millions of NVIDIA chips deployed across NVLink-capable systems and an ecosystem of servers, racks, cables, switches, software, and operations tooling. The NVIDIA annual platform cadence delivers new hardware capabilities at the pace of AI innovation, enabling the industry to rapidly deploy new models and workflows to meet the world’s AI needs. In addition to hardware, NVIDIA and the community release software as a key component of the scale-up networking stack. These include: Dynamo : Open source framework for scaling  generative AI and reasoning models in multi-node GPU environments, with  disaggregated serving , dynamic GPU allocation, KV-cache-aware routing, and NIXL-based data movement. TensorRT-LLM : Open source NVIDIA library for high performance LLM inference on NVIDIA GPUs, using optimized kernels, compute/communication overlap, and lower-precision formats like NVFP4 and FP8 to boost throughput including for  MoE models. NIXL : Open source NVIDIA library for fast data transfers across GPU memory, CPU memory, NVMe, and remote storage helping  move KV-cache and inference state efficiently in distributed systems. NCCL : Open NVIDIA library for high-speed GPU communication, with more than 10 years of open source innovations. Includes topology-aware collectives, AI framework integration, and SHARP support for in-network reductions and other collectives. Fabric and Memory Management : Includes address space management APIs, extensible memory semantics, and built-in memory coherency for efficient HBM access. Software innovations continue long after hardware release, enabling X factor speedups throughout the life of the product. NVLink-C2C extends the fabric to CPUs AI factories need more than GPU-to-GPU bandwidth. They also need high-bandwidth, coherent CPU-GPU connectivity for orchestration, data movement, memory management, storage services, and agentic workloads that mix compute phases. NVIDIA NVLink-C2C provides that path. With Vera CPUs in the Vera Rubin NVL72 platform, NVLink-C2C delivers 1.8 TB/s of coherent bandwidth between CPUs and GPUs, 7x the bandwidth of PCIe Gen6. This enables high-speed data sharing and a unified coherent memory architecture across CPU and GPU memory. For infrastructure teams, that means fewer bottlenecks between control, memory, and compute. For software teams, it simplifies programming models and supports workloads such as KV-cache offload, multi-model execution, data processing, and agentic orchestration. Vera itself is designed for this role, with 88 NVIDIA custom Olympus cores, high memory bandwidth, and power-efficient operation for AI factory workloads. In the NVIDIA platform, CPU, GPU, NVLink, HBM, system memory, networking, and software are co-designed to optimize performance. NVLink Fusion: Semi-custom XPU infrastructure without starting over Hyperscalers and AI natives build specialized silicon for targeted workloads and to offer options to their customers, but they face several challenges in deploying them. Integrating state-of-the-art scale-up networking, designing and deploying a complete rack-scale architecture, committing to data center design amid silicon supply risk, and supporting heterogeneous infrastructure each present unique difficulties. NVIDIA NVLink Fusion provides that path. NVLink Fusion is the high-bandwidth, low-latency interconnect technology and IP that connects custom silicon to the NVIDIA world-leading AI infrastructure platform. With NVLink Fusion, they can leverage the proven NVLink scale-up stack and ecosystem to reduce development and deployment complexity, increase performance, and accelerate time-to-market for semi-custom AI factories. And by standardizing on a single unified architecture, NVLink Fusion simplifies operations across the data center, enables flexible reprovisioning of data center capacity, and allows custom AI XPUs to integrate seamlessly with GPUs for disaggregated compute. This breaks a fundamental constraint by eliminating the need to choose between custom XPUs and a world-class AI platform. And because NVLink Fusion keeps pace with the NVIDIA technology roadmap, adopters can keep pace with the company’s annual cadence. The path for production AI factories The next AI factory won’t be won by the fastest standalone accelerator. It will be won by the infrastructure that can deliver the most intelligence, at the lowest cost per token, on the fastest cadence, with the least deployment risk. NVLink delivers leading performance, with 3X lower latency and 10X higher packet rates as well as 130 TFLOPs of in-network compute, factory resiliency features, and a mature, proven platform. AI factories are becoming the defining infrastructure of the AI computing era. NVLink delivers the performance, operability, maturity, and continuous innovation that powers them. Learn more about the NVIDIA Vera Rubin Platform , NVLink , and NVLink Fusion .