Alibaba Releases 2.4T-Parameter Qwen3.8-Max with NVIDIA GB300 NVL72 Serving Support

Alibaba Releases 2.4T-Parameter Qwen3.8-Max with NVIDIA GB300 NVL72 Serving Support Alibaba has released open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), a 2.4-trillion-parameter fine-grained MoE model featuring configurable reasoning and hybrid attention. NVIDIA collaborated to provide Day 0 inference recipes, reporting over 4,000 tokens per second per GPU on the GB300 NVL72 rack-scale platform. Qwen3.8-2.4T-A95B activates 95B parameters per token across a fine-grained MoE architecture, supporting up to a 1M token context window and 128K output length. The model alternates between full-attention and linear-attention layers to bound compute and KV cache memory growth during long-context execution. On NVIDIA's GB300 NVL72 running FP8 precision out of the box, throughput exceeds 4K tokens per second per GPU and 350 tokens per second per user. Inference is supported across SGLang, vLLM, NVIDIA Dynamo, and model-free NVIDIA NIM, with post-training supported via NVIDIA NeMo AutoModel. Architecture and Reasoning Controls Designed for multi-step agentic tasks, coding, and large-scale document analysis, Qwen3.8-2.4T-A95B combines full attention with linear attention. The linear layers substitute the expanding KV cache with a bounded recurrent state to restrain memory overhead across contexts up to one million tokens. Its fine-grained mixture of experts routes each token to a specialized sub-cluster of smaller experts, keeping compute costs aligned with the 95B activated parameters. Developers can also set reasoning depth per request across low, high, or xhigh presets to balance output depth against throughput. Hardware Performance on GB300 NVL72 Deploying the 2.4T model relies on the NVIDIA GB300 NVL72 platform, which links 72 Blackwell Ultra GPUs inside a single NVLink domain with 130 TB/s of all-to-all communication bandwidth. Operating in FP8 precision without extra tuning, the system outputs more than 4,000 tokens per second per GPU and over 350 tokens per second per user. NVIDIA expects subsequent enhancements, including NVFP4 precision, to yield further performance gains. Serving and Post-Training Ecosystem Developers can deploy the open weights via runtime recipes in SGLang, vLLM, and NVIDIA Dynamo, or use a model-free NVIDIA NIM container on NVIDIA NGC. For customization, NVIDIA NeMo AutoModel supports Day-0 Hugging Face checkpoints without format conversion, offering full supervised fine-tuning and LoRA options. Hosted APIs are also available through providers including DeepInfra, DigitalOcean, Fireworks AI, Modal, and OpenRouter.

阿里巴巴开源2.4T参数模型Qwen3.8-2.4T-A95B，支持在英伟达GB300 NVL72上部署 阿里巴巴发布了开源大模型Qwen3.8-2.4T-A95B（Qwen3.8-Max），拥有2.4万亿总参数且每Token激活950亿参数，英伟达同步提供了基于GB300 NVL72系统的推理与部署支持。 模型总参数量达2.4T，每Token激活95B参数，支持最高100万Token上下文及128K输出长度。 采用全注意力与线性注意力的混合架构，配合细粒度MoE以控制计算与KV缓存增长。 在英伟达GB300 NVL72系统（FP8精度）上实现单GPU超4K Token/秒、单用户超350 Token/秒的吞吐量。 支持low、high、xhigh三档推理深度调节，并适配SGLang、vLLM、NVIDIA NIM及NeMo AutoModel等软件工具。 针对长上下文与智能体任务的架构设计 Qwen3.8-2.4T-A95B专为复杂的Agent任务设计，如编程、大规模文档分析与多步工作流。为了解决长上下文带来的计算与KV缓存压力，模型交替使用全注意力层与线性注意力层，在线性层中用有界循环状态替代不断增长的KV缓存。同时，细粒度MoE将容量分配给更多小型专家，通过学习型路由器按需激活，使实际服务成本仅取决于激活参数量而非全量2.4T参数。此外，内置的推理控制功能支持开发者按请求配置推理深度（low/high/xhigh）。 英伟达GB300 NVL72性能表现 GB300 NVL72机架级架构集成了72颗Blackwell Ultra GPU，通过72-GPU NVLink域提供130 TB/s的全互连通信，消除了传统网络在专家路由传输时的瓶颈。在FP8精度且无需额外调优的条件下，该系统可实现单GPU超4K Token/秒和单用户超350 Token/秒的吞吐量，未来通过NVFP4精度等优化预计可带来进一步性能提升。 软件支持、微调与部署渠道 英伟达针对该模型提供了SGLang、vLLM和NVIDIA Dynamo等开源推理方案，并支持通过model-free NVIDIA NIM容器直接部署。开发者还可使用NVIDIA NeMo AutoModel进行无需转换权重的原生全量SFT或LoRA微调。目前，模型权重已上线Hugging Face与ModelScope，并接入了DeepInfra、DigitalOcean、Fireworks AI、Modal和OpenRouter等托管API服务。

Alibaba released the open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), its largest open-weight model, bringing near-frontier capabilities to the open ecosystem. It has 2.4T total parameters with 95B activated per token. It’s a fine-grained mixture of experts (MoE) architecture with a hybrid of full and linear attention, a context window of up to one million tokens, and an output length of up to 128K, designed for demanding reasoning and agentic workloads. Deploying a 2.4T parameter open-weight model requires data-center-scale accelerated compute. Inference at this scale depends on extreme co-design across chips, system architecture, and software. NVIDIA is working with the open-source ecosystem to bring the model to multinode deployments through optimized kernels, inference runtimes, and distributed serving recipes. Without additional model tuning, the model achieves a throughput of over 4K tokens per second per GPU and over 350 tokens per second per user on NVIDIA GB300 NVL72 in FP8 precision on Day 0. Further optimizations, including NVFP4 precision, are expected to deliver enhanced performance gains over time. ## Architectural innovations for long-context inference  Qwen3.8-2.4T-A95B is built for the hardest agentic workloads like coding, large-scale document analysis, and long-running multi-step workflows. Unlike chat-first models that send a single prompt and receive a single reply, agentic applications accumulate system instructions, tool outputs, retrieved documents, code, logs, and multi-step reasoning traces across a workflow. As context grows, attention, compute, and KV cache memory become the binding constraints. The full-attention and linear-attention hybrid architecture addresses this, and the model alternates between the two. In the full-attention layers, every token attends to every other token, and in the linear-attention layers, the growing KV cache is replaced with a bounded recurrent state. Qwen3.8-2.4T-A95B keeps both compute and memory bounded as context scales to up to one million tokens. Fine-grained MoE makes the 2.4T parameter count practical to serve. Instead of a small number of large experts, capacity is distributed across a larger population of smaller experts, improving specialization and routing efficiency per unit of activated compute. A learned router activates only the experts needed per token, so serving costs track active parameters, not the full 2.4T parameters, delivering frontier-scale capacity at a fraction of the cost of a comparable dense model. Built-in reasoning controls (low/high/xhigh) enable developers to configure inference depth per request, trading compute for reasoning quality depending on the task: dial up for complex multi-step reasoning or dial down for high-throughput document processing. ## Qwen3.8-2.4T-A95B optimized performance on GB300 NVL72  The GB300 NVL72 features a rack-scale architecture that integrates 72 NVIDIA Blackwell Ultra GPUs into a single platform. Its large, 72-GPU NVIDIA NVLink domain enables efficient all-to-all communication at 130 TB/s, eliminating bottlenecks that appear when expert traffic must cross traditional off-the-shelf networks. Out of the box, Qwen3.8 2.4T-A95B running on NVIDIA Blackwell GB300 NVL72 delivers over 4K tokens per second per GPU and over 350 tokens per second per user, enabling AI factories to run large-parameter models in production at high throughput and low latency. ## Post-train Qwen3.8-2.4T-A95B and choose a serving path NVIDIA supports multiple inference stacks to meet a variety of developer needs. [SGLang](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8), [vLLM](https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B), and [NVIDIA Dynamo](https://docs.nvidia.com/dynamo/dev/recipes/qwen-3-8-2-4t-a95b) provide open-source inference recipes for developers who require greater control over performance on the NVIDIA-accelerated platform. It’s also available to [deploy](https://docs.nvidia.com/nim/large-language-models/latest/get-started/advanced/get-started-qwen3.8.html#get-started-with-qwen3-8-2-4t-a95b) via a [model-free NVIDIA NIM](https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/sglang-model-free-nim/-), a single inference container that serves any supported model. Download the model weights and deploy on Day-0 to serve fine-tuned checkpoints, and scale to production. Developers can post-train the model for domain-specific use cases using [NVIDIA NeMo AutoModel](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/model-coverage/llm/qwen/qwen3-8-2-4t-a95b.mdx), a PyTorch-native fine-tuning library with Day-0 Hugging Face checkpoint support. Train directly on existing checkpoints without model conversion, with support for full SFT or memory-efficient LoRA fine-tuning. ## Get started with Qwen3.8-2.4T-A95B   Try out the model from these hosted APIs: [DeepInfra](https://deepinfra.com/Qwen/Qwen3.8-2.4T-A95B), [DigitalOcean,](https://www.digitalocean.com/solutions/gpu-inference) [Fireworks AI,](https://fireworks.ai/models/fireworks/qwen3p8-max) [Modal](https://modal.com/products/inference), and [OpenRouter](https://openrouter.ai/qwen/qwen3.8-2.4t-a95b) Download Qwen3.8-2.4T-A95B model weights from [Hugging Face](https://huggingface.co/nvidia/Qwen3.8-2.4T-A95B-NVFP4) or [ModelScope](https://modelscope.cn/models/Qwen/Qwen3.8-2.4T-A95B) and [deploy with](https://docs.nvidia.com/nim/large-language-models/latest/get-started/advanced/get-started-qwen3.8.html#get-started-with-qwen3-8-2-4t-a95b) a [model-free NVIDIA NIM](https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/sglang-model-free-nim/-) from NVIDIA NGC.