Meta Custom MTIA 300 Accelerator Hits 98% Compute-Network Efficiency

@Scale

Integrating dedicated message engines and dynamic network fungibility directly into MTIA 300 silicon bypasses host CPU bottlenecks and achieves 98% execution efficiency under heavy AI workloads.

AI-assisted summary of Live from SCCC: Co-Designing Communication for AI Accelerators | Rajeev Nair and Wes Bland — original source. Check the original for full context.

Source published: 2026-09-13

Detailed notes

## Background Published by @Scale on YouTube on August 31, 2026, this page features Rajeev Nair and Wes Bland presenting "Co-Designing Communication for AI Accelerators" at the Santa Clara Convention Center. According to the page description, the talk presents Meta's communications architecture for MTIA 300, showing how co-designing hardware, software, and models drives up compute and network efficiency through offloading message engines and flexible network fungibility.

## TL;DR The presentation details how Meta addresses the compute and data movement challenges of generative AI and agentic workloads by decoupling communication from compute on its custom MTIA 300 silicon. By embedding dedicated hardware Message Engines alongside Processing Elements and High Bandwidth Memory, MTIA 300 offloads communication logic from host CPUs, achieving 98% compute-communication overlap efficiency and sub-6 microsecond small-message latency. Furthermore, the architecture utilizes standard Ethernet to achieve network fungibility, allowing NICs to adapt dynamically between scale-up and scale-out topologies across Meta's evolving hardware roadmap.

## Atmosphere & style The presentation carries a professional, technical, and forward-looking tone typical of an engineering conference keynote. Rajeev Nair and Wes Bland deliver a structured, authoritative walkthrough of Meta's infrastructure, physically holding up a chip package to ground the high-level silicon architecture in reality. The dynamic alternates between big-picture strategic framing on workload evolution and precise technical benchmark metrics, giving viewers an inside look at cutting-edge industrial hardware-software co-design.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=9OcKS-lJETM&t=4s) Hardware-software co-design—integrating silicon, communication stacks, and compute kernels—is essential to keep pace with evolving AI workloads from DLRM to agentic systems. - [02:01](https://www.youtube.com/watch?v=9OcKS-lJETM&t=121s) Traditional serialized compute-and-communication execution models leave up to 50% of data center hardware idle during AI operations. - [03:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=212s) Hosting network management directly on the CPU creates severe latency bottlenecks when executing modern fused compute and communication kernels. - [05:00](https://www.youtube.com/watch?v=9OcKS-lJETM&t=300s) MTIA 300 integrates dedicated hardware Message Engines onto the chip to fully offload network management and message dependencies from the host CPU and compute units. - [09:19](https://www.youtube.com/watch?v=9OcKS-lJETM&t=559s) MTIA 300 achieves 98% compute-communication overlap efficiency, significantly outperforming leading commercial accelerators that achieve 70% to 80%. - [09:19](https://www.youtube.com/watch?v=9OcKS-lJETM&t=559s) For latency-critical generative AI workloads, direct PE-to-NIC packet injection achieves small-message latencies as low as 6 microseconds. - [12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) Network fungibility allows MTIA 300's 12 Ethernet-based RDMA NICs to dynamically shift bandwidth between scale-up and scale-out configurations. - [14:28](https://www.youtube.com/watch?v=9OcKS-lJETM&t=868s) Meta's multi-generational silicon roadmap aims for a release pace of roughly two generations per year (with MTIA 450 and MTIA 500 next), featuring native PyTorch integration across all generations.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=9OcKS-lJETM&t=4s) The Imperative of Hardware-Software Co-Design

Rajeev Nair introduces the presentation, setting the central theme for the talk: building a complete communication stack for AI accelerators from silicon all the way up to models. He emphasizes that the single overarching takeaway for the session is "co-design." In his view, efficient AI infrastructure cannot be built by treating software as an afterthought or by bolting networking on at the end of hardware development. Instead, networking silicon, the communication stack, and compute kernels must all be designed together in a single, unified loop.

To contextualize why this paradigm shift is necessary, Rajeev traces the rapid evolution of Meta's workloads over recent years. Prior to the rise of generative AI, ranking and recommendation models were the main workload driving infrastructure design. By 2022, generative AI training began scaling at an unprecedented rate, and by 2025, agentic workloads emerged as a primary driver of compute demand. Each of these architectural shifts brought unique systemic challenges, including varying message sizes, extreme latency requirements, and distinct collective communication patterns. Because compute demand grows universally across the industry, Rajeev argues that co-designing hardware directly around data movement and model needs is the only viable strategy to maintain efficiency.

- **Pre-2022 Primary Workload:** Ranking and recommendation models. - **2022 Infrastructure Shift:** Rapid scaling of generative AI training. - **2025 Emerging Workloads:** Agentic workloads with distinct latency and message-size demands. - **Core Philosophy:** Co-designing hardware, software, networking silicon, and kernels in a tight, iterative loop.

[02:01](https://www.youtube.com/watch?v=9OcKS-lJETM&t=121s) Breaking the Serial Compute-Communication Bottleneck

Rajeev walks through the internal execution dynamics of an AI serving run to illustrate why traditional accelerator designs falter under heavy generative AI workloads. Modern AI models introduce massive volumes of weights, activations, and tokens that far exceed the memory capacity of a single machine. Consequently, systems rely on two interdependent functions: compute operations performed by accelerators, and communication operations that distribute and aggregate data across nodes.

In conventional system architectures, execution alternates strictly between compute and communication in a serialized timeline (compute $\rightarrow$ communication $\rightarrow$ compute $\rightarrow$ communication). While this model is straightforward to design and debug, it leaves half of the data center infrastructure idle at any given moment. To overcome this inefficiency, systems must achieve "overlap"—restructuring models so data for subsequent layers moves across the network while the current layer computes. Fully saturating both compute chips and network interfaces yields higher throughput per unit time.

- **System Bottleneck:** Model parameters (weights, activations, tokens) exceed single-node memory capacity. - **Traditional Execution Model:** Strictly serialized compute and communication, leading to ~50% hardware idleness. - **Target Execution Model:** Fine-grained overlap where communication runs concurrently with compute tasks.

[03:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=212s) Conventional Host-Accelerator Architecture Limitations

Examining typical accelerator implementations, Rajeev outlines the structural bottlenecks that occur when host CPUs manage networking tasks. In standard hardware layouts, a CPU generates compute kernels for the accelerator while simultaneously generating network instructions for network interface cards (NICs). The NIC pulls data directly from accelerator memory to avoid routing data across the PCIe bus, which would introduce severe bandwidth constraints.

However, placing the host CPU directly in the critical path introduces latency and limits overall throughput. As boundaries between compute and networking blur—and as fused kernels merge computation and data transfer more aggressively—silicon architectures must be future-proofed to handle changing execution patterns without host-level contention. Rajeev, who works on Meta's host networking team supporting MTIA (Meta Training and Inference Accelerator), hands over the stage to Wes Bland to explain how MTIA 300 resolves these architectural friction points.

- **Conventional Control Flow:** Host CPU generates both compute kernels and NIC instructions. - **PCIe Avoidance:** NICs access accelerator memory directly to bypass bus bottlenecks. - **Host Bottleneck:** Keeping the CPU in the critical path limits the execution speed of fused compute-communication kernels.

[05:00](https://www.youtube.com/watch?v=9OcKS-lJETM&t=300s) Introducing MTIA 300 and Dedicated Message Engines

Wes Bland introduces MTIA 300, Meta’s custom-designed AI training and inference accelerator designed to solve the communication bottlenecks inherent in Deep Learning Recommendation Models (DLRM) and generative AI. Showing a physically compact chip package to the audience, Wes explains how MTIA 300 embeds dedicated communication hardware directly onto the silicon to offload management tasks entirely from both the host CPU and the primary compute cores.

At the core of this design are dedicated hardware "Message Engines" integrated alongside the main grid of Processing Elements (PEs). The host CPU batches communication instructions and offloads them directly to these Message Engines, which then drive network traffic to the NICs, monitor transfer completions, and enforce data dependencies automatically. Once offloaded, neither the host CPU nor the accelerator's PEs need to spend compute cycles managing communication tasks.

- **Chip Architecture:** Integrates a central grid of Processing Elements (PEs), dedicated Message Engines, High Bandwidth Memory (HBM), and Near-Memory Compute (NMC) units. - **Message Engine Functionality:** Batches instructions, drives NIC traffic, monitors completion, and manages message dependencies without CPU involvement. - **Integrated NIC Capacity:** Dual banks of six Broadcom RDMA NICs providing up to 1.2 TB/s of aggregate bandwidth. - **Offload Mechanism:** Near-Memory Compute (NMC) units perform in-memory arithmetic (such as accumulations) directly next to HBM, preventing data from traveling back to the main PE grid. - **Publications & Documentation:** Published technical details in Meta Engineering blog posts (March and recent updates) and peer-reviewed papers at ISCA and Supercomputing (SC).

[07:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=452s) Architectural Pillars: Offloading, Fungibility, and Co-Design

Wes outlines the three core technical pillars that define MTIA 300's execution model. The first is message offloading, which enables hardware overlap between networking and computation to maximize execution efficiency. The second is network fungibility, providing dynamic hardware flexibility to adapt as workloads shift over time while simplifying the software stack. The third pillar is model-to-silicon co-design, which Rajeev returns to detail later in the talk.

In traditional setups, host CPUs inject network traffic, adding latency and consuming accelerator compute resources to perform arithmetic during message reception. On MTIA 300, communication patterns are defined on the host, copied into High Bandwidth Memory (HBM), and processed asynchronously by the Message Engines via a CUDA-like streaming library interface. This queue-and-execute streaming model decouples host queuing from accelerator execution, allowing compute and network transfer to occur simultaneously.

- **Pillar 1:** Compute-communication offloading via hardware Message Engines. - **Pillar 2:** Network fungibility for flexible resource allocation between scale-up and scale-out topologies. - **Pillar 3:** End-to-end software/hardware co-design tailored for PyTorch workloads. - **Execution Model:** Host enqueues work streams into HBM; Message Engines execute tasks asynchronously.

[09:19](https://www.youtube.com/watch?v=9OcKS-lJETM&t=559s) Benchmark Results and Latency Optimizations for GenAI

To quantify the performance gains of MTIA 300, Wes shares benchmark results comparing its overlap efficiency against a leading commercial accelerator. Using a workload that simultaneously executes matrix multiplication and collective communication, MTIA 300 achieved **98% efficiency** (where 100% represents zero performance degradation when running compute and communication concurrently). In contrast, commercial alternatives achieved only **70% to 80% efficiency** due to resource contention between compute and networking tasks on shared hardware.

While offloading provides massive throughput for bandwidth-bound tasks—achieving **94% of roofline memory bandwidth** (~1.1 TB/s across a 16-node cluster)—the shift from DLRM to generative AI inference introduced heavy latency constraints for smaller message sizes. To adapt, MTIA 300 supports a direct-injection mode. For extremely latency-sensitive workloads, Processing Elements (PEs) inject RDMA packets directly into the NIC work queues while Message Engines handle completion monitoring in the background using hardware counters. This direct path reduces small-message communication latency down to approximately **6 microseconds**.

- **Overlap Efficiency:** MTIA 300 achieves **98% efficiency** under concurrent compute and collective communication, compared to **70%–80%** on leading commercial chips. - **Bandwidth Utilization:** Achieves **94% of roofline bandwidth** (~1.1 TB/s aggregated over 16 nodes) for large-message transfers. - **Small-Message Latency:** Direct PE-to-NIC injection drops small-message latency to ~**6 microseconds** ($\mu\text{s}$).

[12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) Flexible Topologies via Network Fungibility

Addressing data center architecture, Wes discusses how MTIA 300 handles scale-up networks (high-bandwidth communication between accelerators within the same rack) and scale-out networks (communication across different racks). Training workloads typically demand massive scale-out capacity across clusters, whereas generative AI inference demands maximum intra-rack scale-up bandwidth.

Rather than using disparate hardware for scale-up and scale-out connectivity, MTIA 300 utilizes standard Ethernet across all on-chip Broadcom RDMA NICs, enabling complete "network fungibility." By default, the system provisions 10 NICs for scale-up interconnects and 2 NICs for scale-out interconnects. However, for inference-heavy deployments, the system can be reconfigured dynamically to allocate all 12 NICs to scale-up topologies or split them into dual scale-up networks. This flexibility relies on standard Ethernet, allowing Meta to partner with broad ecosystem vendors to keep AI serving infrastructure inexpensive and open.

- **Standard Deployment Config:** 10 scale-up NICs + 2 scale-out NICs per accelerator node. - **Inference Reconfiguration:** Dynamically reallocates all 12 NICs for scale-up operations or secondary intra-rack domains. - **Interconnect Protocol:** Unified Ethernet-based RDMA across all scale-up and scale-out interfaces.

[14:28](https://www.youtube.com/watch?v=9OcKS-lJETM&t=868s) Silicon Roadmap and PyTorch Native Integration

Rajeev Nair returns to present Meta's multi-generational MTIA silicon roadmap. Initial iterations (MTIA 100 and MTIA 200) targeted single-node inference specifically optimized for ranking and recommendation models. MTIA 300 represents a major architectural milestone as Meta's first unified chip capable of handling both high-throughput training and low-latency inference on the same platform.

By incorporating software and hardware learnings from each production generation, Meta has accelerated its silicon release cadences. Next-generation accelerators—**MTIA 450** and **MTIA 500**—are already well under development and scheduled for landing next year, targeting a release pace of roughly **two silicon generations per year**.

Closing the talk, Rajeev emphasizes that MTIA is built from the ground up as a native PyTorch system. This deep integration allows machine learning developers to deploy custom kernels directly without spending manual effort porting software to proprietary hardware layers.

- **MTIA 100 / 200:** Targeted single-node inference for ranking and recommendations. - **MTIA 300:** Combined training and inference chip supporting dynamic offload and network fungibility. - **Future Roadmap:** **MTIA 450** and **MTIA 500** scheduled for rollout next year, sustaining a target cadence of two generations per year. - **Developer Stack:** Fully PyTorch-native platform designed for direct model-to-silicon compilation.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=9OcKS-lJETM&t=4s) co-design — Rajeev Nair

> [09:19](https://www.youtube.com/watch?v=9OcKS-lJETM&t=559s) 98% efficiency — Wes Bland

> [09:19](https://www.youtube.com/watch?v=9OcKS-lJETM&t=559s) 70% to 80% efficiency — Wes Bland

> [09:19](https://www.youtube.com/watch?v=9OcKS-lJETM&t=559s) 94% of roofline memory bandwidth — Wes Bland

> [09:19](https://www.youtube.com/watch?v=9OcKS-lJETM&t=559s) 6 microseconds — Wes Bland

> [07:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=452s) network fungibility — Wes Bland

> [14:28](https://www.youtube.com/watch?v=9OcKS-lJETM&t=868s) two silicon generations per year — Rajeev Nair

## Mentioned Rajeev Nair, Wes Bland, Meta, MTIA, MTIA 100, MTIA 200, MTIA 300, MTIA 450, MTIA 500, Broadcom, PyTorch, CUDA, ISCA, Supercomputing (SC), Santa Clara Convention Center

[Source](https://www.youtube.com/watch?v=9OcKS-lJETM)

中文简报

Meta定制MTIA 300加速器计算与网络效率达98% 通过将专用消息引擎和动态网络互换性直接集成到 MTIA 300 芯片中，绕过主机 CPU 瓶颈，在繁重的 AI 工作负载下实现 98% 的执行效率。该演讲详细阐述了 Meta 如何通过在其自研 MTIA 300 芯片上实现通信与计算解耦，以应对生成式 AI 及智能体工作负载中的计算与数据移动挑战。通过在处理单元和高带宽内存旁嵌入专用硬件消息引擎，MTIA 300 将通信逻辑从主机 CPU 中卸载，实现了 98% 的计算与通信重叠效率，并将小消息延迟控制在 6 微秒以内。此外，该架构利用标准以太网实现网络互换性，使网卡能够在 Meta 不断演进的硬件路线图中，在纵向扩展（scale-up）与横向扩展（scale-out）拓扑之间动态适配。 Meta 指出传统计算与通信串行执行会导致数据中心约 50% 的硬件处于闲置状态，必须通过底层芯片、通信栈与计算核的软硬件协同设计（co-design）打破这一性能瓶颈。 (2:01) Meta 推出兼顾训练与推理的自研芯片 MTIA 300，通过片上集成的专用硬件消息引擎（Message Engines），将网络控制与消息依赖处理完全从主机 CPU 和计算单元中卸载。 (5:00) 基准测试显示 MTIA 300 实现了 98% 的计算通信重叠效率（显著优于主流商业加速器的 70%–80%），并通过处理单元直通 NIC 的数据包注入将小消息延迟降至 6 微秒。 (9:19) 依托标准以太网 RDMA，MTIA 300 具备“网络同质可塑性（network fungibility）”，其 12 个 NIC 能根据负载在机架内 Scale-Up 与集群 Scale-Out 拓扑间动态切换和分配带宽。 (12:13) Meta 加速推进自研芯片迭代，计划以每年约两代的节奏推出 MTIA 450 和 MTIA 500，并在全代际硬件中保持端到端原生 PyTorch 深度集成。 (14:28)

Reading summary

Meta engineers Rajeev Nair and Wes Bland argue that overcoming the communication bottlenecks of modern AI workloads requires tight hardware-software co-design, shifting network orchestration away from host CPUs to dedicated silicon engines. Eliminating Accelerator Idleness Through Hardware Overlap (2:01) As AI workloads evolved from recommendation models to generative AI training and agentic systems, model parameters rapidly exceeded single-node memory capacities. In conventional systems, execution alternates serially between compute and network communication, leaving up to half of the data center infrastructure idle at any given point. Nair emphasizes that saturating hardware requires continuous overlap, executing compute tasks for a current layer while simultaneously moving network data for subsequent layers. Traditional accelerator deployments struggle to maintain this concurrency because the host CPU remains squarely in the critical path, generating instructions for both compute kernels and network interface cards (NICs). Nair points out that relying on host-level intervention introduces latency and limits execution speed, especially as software architectures increasingly demand fused compute-communication kernels. Decoupling Communication with Dedicated On-Chip Engines (5:00) To eliminate host-level contention, Wes Bland details the architecture of MTIA 300, Meta's accelerator built for both deep learning recommendation models and generative AI. MTIA 300 integrates dedicated hardware Message Engines alongside its central grid of Processing Elements (PEs), High Bandwidth Memory (HBM), and Near-Memory Compute (NMC) units. The host CPU batches communication instructions into HBM, allowing the Message Engines to drive NIC traffic, track message completions, and handle dependencies autonomously. This streaming offload model fully frees both the host CPU and the primary PE compute grid from managing network operations. Furthermore, the embedded NMC units execute arithmetic operations like accumulations directly adjacent to HBM, preventing intermediate network data from traveling back and forth to the primary compute cores. Benchmark Performance and Low-Latency Direct Injection (9:19) Bland presents benchmark data illustrating the efficiency gains of MTIA 300's hardware offload mechanism. Under concurrent matrix multiplication and collective communication workloads, MTIA 300 achieved 98% overlap efficiency—defined as execution without performance degradation—compared to 70% to 80% on leading commercial accelerators constrained by shared-hardware contention. Across a 16-node cluster, the system reached 94% of roofline memory bandwidth, delivering roughly 1.1 TB/s of throughput across its integrated Broadcom RDMA NICs. To address the extreme latency requirements of generative AI inference on smaller messages, MTIA 300 also provides a direct-injection pathway. In this mode, PEs inject RDMA packets straight into NIC work queues while Message Engines manage completion tracking via hardware counters in the background, lowering small-message communication latency to approximately 6 microseconds. Dynamic Topology Reconfiguration via Network Fungibility (12:13) Rather than employing separate, specialized hardware for intra-rack scale-up and inter-rack scale-out networks, MTIA 300 utilizes standard Ethernet across all 12 on-chip RDMA NICs. Bland describes this design as "network fungibility," which allows dynamic interconnect adaptation to match fluctuating workload demands. While training typically requires extensive scale-out bandwidth, generative AI inference prioritizes intra-rack throughput. MTIA 300 supports a default configuration of 10 scale-up NICs and 2 scale-out NICs, but deployments can dynamically reallocate all 12 NICs to scale-up operations or split them into dual intra-rack domains. Nair and Bland stress that standardizing on Ethernet allows Meta to avoid proprietary networking ecosystems and maintain cost-effective AI serving infrastructure. Meta is accelerating its hardware cadence to roughly two silicon generations per year, with MTIA 450 and MTIA 500 slated for release next year. While native PyTorch integration provides a direct compilation path for custom kernels, sustaining this deployment velocity will depend on how effectively the unified communication architecture scales across increasingly diverse agentic workloads.

精炼阅读摘要

Meta工程师Rajeev Nair与Wes Bland阐述了针对AI加速器构建软硬件协同设计全栈通信体系的必要性，并展示了通过专有通信引擎、网络可塑性与原生PyTorch集成以消除算力与网络瓶颈的MTIA 300架构方案。突破算力与通信的串行执行瓶颈 (0:04) Rajeev Nair回顾了Meta基础设施承载负载的演进历程：2022年之前以推荐排序模型为主，2022年起生成式AI训练迅速规模化，至2025年智能体工作负载已成为算力需求的新驱动力。各类工作负载对通信消息大小、延迟敏感度及集合通信模式提出了截然不同的严苛要求。然而，现代前沿AI模型的参数、激活值与Token体量远超单卡显存容量，必须依赖加速器计算与跨节点网络通信协同推进。在传统系统中，软硬件常遵循“计算与通信严格串行”的交替模型，导致数据中心在任意时刻均有近一半的基础设施处于空闲等待状态，无法实现吞吐最大化。此外，传统加速器架构通常由主机CPU统一生成计算内核并直接向网卡（NIC）派发指令。尽管网卡可通过旁路直连加速器内存规避PCIe总线带宽瓶颈，但将主机CPU置于关键控制路径上仍会引发控制面延迟，并严重限制计算与通信融合算子的运行效率。Nair指出，仅靠硬件完工后补充软件或外挂网络的割裂模式已无法应对算力增长，业界必须从网络芯片、通信栈到计算内核推行软硬件一体的协同设计，通过深度计算与通信重叠来提升硬件利用率。专用消息引擎实现高效率软硬件解耦 (5:00) 为消除主机控制瓶颈并解决深度学习推荐模型（DLRM）与生成式AI的通信摩擦，Wes Bland介绍了Meta自主研发的AI训练与推理加速器MTIA 300。该芯片在主处理单元（PE）网格旁直接集成了专用的硬件“消息引擎”（Message Engines），并配置了高带宽内存（HBM）与近内存计算（NMC）单元。在该架构下，主机CPU仅需将通信指令批量提交至消息引擎，后续的网卡流量驱动、传输完成度监控及依赖关系解析均由消息引擎在硬件层自主闭环处理，无需占用CPU周期或加速器计算核心。同时，近内存计算单元可直接在HBM旁完成数据累加等算术运算，避免了数据回传至主PE网格所产生的内部互连拥堵。在执行机制上，主机将通信任务写入HBM形成工作队列，消息引擎通过类似CUDA的流式接口进行异步处理，使计算与通信实现细粒度硬件解耦。根据Bland引用的基准测试数据，在同时运行矩阵乘法与集合通信的混合负载下，MTIA 300实现了98%的重叠执行效率（100%代表两者并发时无任何性能衰减），而共享硬件资源的商用主流加速器在此类场景下的效率通常仅为70%至80%。在大消息传输测试中，16节点集群的聚合带宽利用率达到了理论上限（roofline）的94%，聚合吞吐达约1.1 TB/s。小消息直注低延迟与网络拓扑弹性 (9:19) 随着工作负载向生成式AI推理转移，系统不仅需要高吞吐，更对小消息传输提出了苛刻的低延迟要求。为解决此类场景下的排队开销，MTIA 300支持“直接注入”模式：在对延迟极度敏感的应用中，处理单元（PE）可绕过常规队列，直接将RDMA数据包注入网卡工作队列，而消息引擎则利用硬件计数器在后台静默跟踪传输状态。据Bland介绍，这种直连路径成功将小消息通信延迟压缩至约6微秒（μs）。在机架互连层面，MTIA 300通过“网络可塑性”（Network Fungibility）设计统一了机架内纵向扩展（scale-up）与跨机架横向扩展（scale-out）网络。芯片集成了两组共12个Broadcom RDMA网卡，提供最高1.2 TB/s的总带宽，且全部基于标准以太网构建。系统默认采用10个scale-up网卡搭配2个scale-out网卡的配置以适配通用训练；而在高度依赖机架内通信的生成式AI推理场景中，系统支持动态重构，可将全部12个网卡调配至scale-up互连或划分为双域网络。这一基于通用以太网的弹性方案避免了定制互连硬件的高昂成本，有助于Meta维持开放、低成本的供应链体系。快速迭代路线图与PyTorch原生集成 (14:28) Rajeev Nair随后梳理了MTIA芯片家族的技术演进路径。早期版本MTIA 100与MTIA 200主要针对单节点的推荐与排序模型推理进行优化，而MTIA 300则标志着Meta首次实现单芯片同时支持高吞吐模型训练与低延迟推理。通过将实际生产中的软硬件协同经验快速反哺至研发流程，Meta大幅缩短了芯片研发周期。Nair透露，下一代芯片MTIA 450与MTIA 500目前已处于深度开发阶段并预计于明年落地，研发团队正力求维持每年推出约两代芯片的高速迭代节奏。在开发者生态层面，Nair强调MTIA从物理设计伊始便被构想为原生支持PyTorch的软硬件平台。通过深度绑定的编译器栈与模型抽象层，机器学习工程师可以直接部署自定义模型与算子，无需耗费大量人力针对专有硬件API进行底层重写与迁移适配。这种从上层开发框架直达底层芯片的一体化设计，旨在消除新硬件在大规模生产环境部署中的软件迁移门槛。 Nair与Bland总结指出，MTIA 300通过底层专用通信引擎与通用以太网协同，为应对AI模型的持续膨胀提供了高效、弹性的体系支撑。然而，演讲中提及的每年推进两代芯片（如MTIA 450与MTIA 500）的高强度研发节奏，以及智能体工作负载对通信拓扑与通信规模带来的动态扰动，仍将在Meta未来的超大规模数据中心生产部署与集群稳定性维护中面临长期的工程考验。

Detailed notes — zh

## Background 本视频由 YouTube 频道 @Scale 于 2026 年 9 月 3 日发布。视频收录了来自 Meta 的 Rajeev Nair 和 Wes Bland 在圣克拉拉会议中心（SCCC）发表的技术演讲，重点介绍了 Meta 如何通过软硬件与算法的协同设计来应对生成式 AI 带来的庞大数据与算力挑战，并全面拆解了自研 AI 加速器 MTIA 300 的专用通信引擎、近内存计算以及网络可替代性（Network Fungibility）等前沿架构。 ## TL;DR 演讲系统性地阐述了 Meta 应对现代大模型与智能体负载的核心解法——全栈协同设计（Co-Design），旨在彻底攻克传统加速器中计算与网络串行等待导致的资源闲置痛点。团队详细披露了 MTIA 300 加速芯片的通信微架构：通过片上集成的专用硬件消息引擎（Message Engine）与近内存计算（NMC）单元，将通信控制与基础规约运算完全从主机 CPU 和主计算核心剥离，达成了惊人的 98% 计算与通信并发重叠效率。为适配生成式推理的极低时延需求，MTIA 引入了主核直注小报文模式，将点到点延迟压缩至约 6 微秒。此外，系统通过统一以太网卡实现了 Scale-up 与 Scale-out 的灵活重构拓扑，并展现了 MTIA 家族后续一年两代的密集演进路线图。 ## Atmosphere & style 整场演讲洋溢着前沿体系结构研发团队特有的务实、严谨与硬核工程自信。两位主讲人配合默契且节奏极快，从宏观的分布式瓶颈剖析自然切换到 Wes 手持实体芯片向全场诙谐展示的极客场景，给听众带来极为强烈的现场临场感与工业级震撼。演讲不仅通过与顶级商业芯片横向对比的扎实微基准数据冷静立论，更在收尾处自然升华至以 PyTorch 生态赋能全球数十亿终端用户的工程师自豪感，兼具深度技术底蕴与前瞻视野。 ## Bulletin - [04:57](https://www.youtube.com/watch?v=9OcKS-lJETM&t=297s) Meta 推出自研训练与推理加速器 MTIA 300，集成硬件消息引擎（Message Engine）与近内存计算（NMC），实现了计算与通信在物理层的解耦与极致并发。 - [07:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=452s) 在矩阵乘法与集合通信并发的微基准测试中，MTIA 300 取得了高达 98% 的重叠效率，显著领先于传统主流商业芯片（70% 至 80%）。 - [10:38](https://www.youtube.com/watch?v=9OcKS-lJETM&t=638s) 针对生成式 AI 推理任务的严苛微秒级时延要求，MTIA 引入主核（PE）直注 RDMA 工作队列机制，将小消息点对点延迟压缩至约 6 微秒。 - [12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) 系统基于统一以太网 NIC 实现“网络可替代性（Network Fungibility）”，可在训练所需的 Scale-out 与推理所需的 Scale-up 模式间动态重构网络配置。 - [14:28](https://www.youtube.com/watch?v=9OcKS-lJETM&t=868s) Meta 加速芯片研发迭代，通过模块化沉淀实现每年推进约两代芯片，后续 MTIA 450 与 500 研发已步入正轨。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=9OcKS-lJETM&t=4s) 全栈协同设计（Co-Design）打破了传统软硬件及网络的孤岛划分，是应对 AI 模型范式急速演进的唯一高效解法。 - [00:04](https://www.youtube.com/watch?v=9OcKS-lJETM&t=4s) AI 工作负载正经历从推荐系统（DLRM）到大规模生成式 AI 训练，再到自主智能体（Agentic）推理的跨越式迁移。 - [02:01](https://www.youtube.com/watch?v=9OcKS-lJETM&t=121s) 克服分布式系统资源空转的关键在于实现计算与通信的极致重叠（Overlap），使两套硬件流水线始终处于饱和状态。 - [03:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=212s) 传统加速器架构中 CPU 主机卡在网络调度的关键控制路径上，算子融合趋势进一步加剧了调度抖动与性能开销。 - [04:57](https://www.youtube.com/watch?v=9OcKS-lJETM&t=297s) MTIA 300 在晶圆物理层面集成了专用消息引擎与近内存计算（NMC），允许直接在显存边沿就地消化集合通信规约累加。 - [04:57](https://www.youtube.com/watch?v=9OcKS-lJETM&t=297s) 芯片配备 12 个 Broadcom RDMA NIC，双向聚合网络带宽高达 1.2 TB/s，且支持按需使能以平衡峰值能效。 - [07:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=452s) 通过自定义通信流库将高阶通信编排下发给消息引擎，MTIA 300 彻底将 Host CPU 和主计算核心从搬移任务中解放出来。 - [10:38](https://www.youtube.com/watch?v=9OcKS-lJETM&t=638s) 在 16 节点纵向与横向聚合场景下，MTIA 300 实测大消息带宽达到理论物理 Roofline 峰值的 94%（吞吐达 1.1 TB/s）。 - [10:38](https://www.youtube.com/watch?v=9OcKS-lJETM&t=638s) 为适应生成式推理微秒级低延迟响应，MTIA 允许计算主核（PE）直注 RDMA 队列并由消息引擎异步批量确认，换取极低延迟。 - [12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) 数据中心网络分为机架内高带宽的 Scale-up 与跨机架的 Scale-out，两者在训练与推理场景下对带宽形态的需求截然不同。 - [12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) Meta 坚守“以太网既必要也充分”的理念，借助统一样式的以太网 NIC 实现网络可替代性，避免受制于专有网络互联协议。 - [12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) 系统支持灵活的硬件网络重组：默认采用 10 张 Scale-up 与 2 张 Scale-out 网卡，推理时可快速切换为全量 Scale-up 模式。 - [14:28](https://www.youtube.com/watch?v=9OcKS-lJETM&t=868s) MTIA 产品演进从面向推荐排序的专用推理芯片（MTIA 100/200），蜕变为兼具大模型训练与大规模推理的统一加速平台（MTIA 300）。 - [15:53](https://www.youtube.com/watch?v=9OcKS-lJETM&t=953s) MTIA 坚持 PyTorch 原生微架构设计，消除了开发者为异构硬件单独移植底层算子的负担，保障算法研发专注前沿模型创新。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=9OcKS-lJETM&t=4s) AI 基础设施的核心法则：软硬件与通信的全栈协同设计 Meta 团队的 Rajeev Nair 登台开场，向与会者开门见山地指出本次演讲的核心：剖析 Meta 如何构建一条贯穿底层芯片硬件、网络通信栈直至上层模型算法的完整通信技术路径。Rajeev 强调，如果整场演讲只能记住一个词，那就是“协同设计”（Co-Design）。在传统开发模式中，团队往往先完成软件设计再匹配硬件，或者在系统全部搭好之后才把网络通信像打补丁一样生硬地拼凑上去；但在 Meta 看来，网络、硅片、通信软件栈和计算内核必须在同一个敏捷闭环中统筹共创，这是构建未来高能效 AI 推理服务平台的必由之路。 AI 模型与业务场景的裂变式演进，逼迫底层基础设施必须打破传统层级界限。Rajeev 回顾了近几年的架构迁移历程：在生成式 AI 爆发之前，驱动 Meta 整体数据中心基础设施演进的核心支柱是推荐与排序模型（DLRM）；到了 2022 年，生成式 AI 的大模型分布式训练规模呈几何级数膨胀；而步入 2025 年，自主智能体（Agentic）工作负载又异军突起，成为全新的算力吞吐增长极。每一次范式转换都会向底层通信系统抛出全新挑战，包括数据包大小的剧变、极端严苛的时延优化目标以及更为复杂的集合通信拓扑（Collectives）。正如 Rajeev 所断言：“算力需求的大幅增长是每个企业都在面对的普遍常态，并非决定性差异；真正能跟上这一演进节奏的唯一解法，就是全栈协同设计——将模型在算法层的严苛诉求，一路深度穿透到底层数据传输的物理实现。” - 核心驱动演进路径：传统推荐与排序系统（DLRM） $\rightarrow$ 2022 年生成式 AI 大规模分布式训练 $\rightarrow$ 2025 年智能体（Agentic）异构推理与服务。 - 基础设施核心矛盾：通信模式由常规批量传输转向多变的集合通信范式（Collective patterns）、动态的消息体积以及微秒级延迟敏感性。 --- ### [02:01](https://www.youtube.com/watch?v=9OcKS-lJETM&t=121s) 摆脱串行停滞：数据移动与计算重叠（Overlap）的吞吐关键 Rajeev 进一步切入实际 AI 运行（AI run）中的核心物理限制。高效的模型服务在本质上所面临的根本挑战，在于模型庞大的参数权重（Weights）、中间激活值（Activations）以及输入输出的 Token 序列在物理空间上根本无法完全装入单台机器的局部内存中。为了突破单机单卡的物理壁垒，分布式系统必须同时把两件事做到极致：一方面由专用加速器执行高强度的矩阵运算，另一方面由高速通信系统在集群节点间进行高效的数据分发与规约聚合。然而，传统系统的时序执行逻辑往往存在致命的效率缺陷。在常规设计中，系统遵循“计算 $\rightarrow$ 通信 $\rightarrow$ 计算 $\rightarrow$ 通信”的严苛串行时序。这种模式虽然设计难度低、代码逻辑简单明了，却会导致数据中心在任意时刻都有将近一半昂贵的硬件资源处于完全闲置的空转状态。为了解决这一痛点，Meta 的唯一目标就是实现极致的“重叠”（Overlap）：通过在算子与编译层对模型结构进行深度重构，确保加速器在执行当前块数据运算的同时，网络引擎已经同步完成下一块数据的传输与拉取。当计算阵列与网络通道同时处于饱和吞吐状态时，集群单位时间内的 Serving 吞吐量才能达到理论峰值，正如 Rajeev 所总结：“这种高效的重叠能力，就是系统架构博弈的全部精髓所在。” ```mermaid flowchart TD accTitle: 串行与重叠模式执行对比 accDescr: 对比传统串行执行模式导致的资源闲置与重叠流水线模式实现的最大化吞吐。 subgraph Serial["传统串行模式（低效闲置）"] direction LR S1["计算核工作 (计算)"] --> S2["网络搬移 (计算核闲置)"] S2 --> S3["计算核工作 (计算)"] --> S4["网络搬移 (计算核闲置)"] end subgraph Overlap["重叠协同模式（算力饱和）"] direction LR O1["计算块 N：矩阵乘法流水"] O2["通信块 N+1：数据预拉取与规约"] O1 -. "完全并发".- O2 end ``` --- ### [03:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=212s) 传统加速器架构瓶颈与 Host 在关键路径中的痛点传统通用加速器在硬件架构设计上普遍存在不可忽视的系统瓶颈。在常见拓扑中，系统主要由 CPU 主机（Host）和专用加速器构成：主机 CPU 负责向加速器分发计算内核（Compute Kernels），同时向网络接口卡（NIC）下发网络传输指令；网卡随后直接从加速器的显存或高带宽内存（HBM）中拉取数据，避免让数据经过 PCIe 等通道导致进一步拥塞。尽管这种直连策略有其合理性，但主机 CPU 却依然牢固地卡在整个通信调度的关键控制路径（Critical Path）之上。不仅如此，随着融合内核（Fused Kernels）在现代算子中被激进采用，计算与网络之间的边界正变得越来越模糊。当计算与通信指令高度交错时，CPU 频繁介入指令派发与同步轮询会引入巨大的控制开销与调度抖动。为了应对这一演进趋势，Meta 必须在自研芯片设计中提供完全面向未来的原生支持，将控制与搬移机制从核心路径中彻底剥离。Rajeev 借此介绍自己的职责：他隶属于 Meta 的主机网络（Host Networking）团队，同时负责支撑 MTIA（Meta 训练与推理加速器）的通信研发；随后，他将舞台交给了来自该项目通信库团队的资深专家 Wes Bland。 - 传统架构痛点：主机 CPU 需要频繁参与每条网络指令的生成、排队和完成确认，成为微秒级网络时延与吞吐的重大卡点。 - 算子融合挑战：计算和通信边界的消融（Fused Kernels）要求底层硅片必须具备自主追踪依赖和协同搬移的能力，而非依赖 Host 中断。 --- ### [04:57](https://www.youtube.com/watch?v=9OcKS-lJETM&t=297s) MTIA 300 芯片亮相：消息引擎（Message Engine）与近内存计算（NMC）架构革新拥有二十余年高性能计算（HPC）与 AI 通信系统研发背景的 Wes Bland 走上讲台。Wes 手持芯片实体向全场展示了 Meta 全新自主研发的 MTIA 300 加速器，并诙谐地表示“虽然它尺寸很小可能在台上看不清，但欢迎大家散场后在走廊找我近距离观摩”。Wes 解释道，针对 Meta 内部核心的深度学习推荐模型（DLRM）对极端算力与巨量数据交换的双重饥渴，团队决定将专用的通信硬件功能直接固化并刻蚀进芯片晶圆之中，MTIA 300 应运而生。关于该芯片的设计细节与协同软件栈，Meta 已在近期工程博客（包括大会前一天的技术长文和当年 3 月的技术发布）以及当年体系结构顶级学术会议 ISCA 和超算大会 SC 上进行了全方位公开披露。在 MTIA 300 核心物理微架构方面，其平面分布展现出高度针对性的革新：芯片中央区域布局了大规模密集处理单元阵列（Processing Elements, PEs）；左右两侧则对称分布着高带宽内存（HBM）通道，并在其近端深度集成了自主研发的硬件**消息引擎（Message Engines, MEs）**。在消息引擎内部，Meta 创新性地引入了**近内存计算（Near-Memory Compute, NMC）**单元。这一设计允许消息引擎在接收网络数据时，直接利用近内存算力就地完成集合通信所需的算术规约（如求和累加），而完全无需把数据倒腾回主计算核心（PEs）去执行算术运算，从而实现了网络数据接收与内存端就地累加的硬件级流水线并行。 ```mermaid flowchart TD accTitle: MTIA 300 芯片内部物理微架构与通信路径 accDescr: 展示 MTIA 300 芯片内部 Processing Elements、Message Engines、NMC、HBM 以及 RDMA NIC 的连接布局。 subgraph TopNIC["顶部网络接口"] TNIC["6 × Broadcom RDMA NICs (600 GB/s)"] end subgraph ChipCore["MTIA 300 芯片核心布局"] direction LR subgraph LeftMem["左侧内存与通信引擎"] ME_L["消息引擎 (Message Engine)"] NMC_L["近内存计算 (NMC)"] HBM_L["高带宽内存 (HBM)"] ME_L --- NMC_L --- HBM_L end subgraph CenterPE["中央计算阵列"] PE["大规模处理单元网格 (PE Grid)<br/>专职执行重度矩阵乘法"] end subgraph RightMem["右侧内存与通信引擎"] ME_R["消息引擎 (Message Engine)"] NMC_R["近内存计算 (NMC)"] HBM_R["高带宽内存 (HBM)"] ME_R --- NMC_R --- HBM_R end LeftMem <--> CenterPE <--> RightMem end subgraph BottomNIC["底部网络接口"] BNIC["6 × Broadcom RDMA NICs (600 GB/s)"] end TNIC <--> ChipCore ChipCore <--> BNIC ``` - 核心消息引擎功能：自主批量处理来自 Host 的高级指令、驱动网卡流量、监控完成状态、硬件级追踪报文间的数据依赖；卸载后，Host 与主计算核心均完全脱离通信执行路径。 - 近内存计算（NMC）红利：直接在显存边沿就地消化集合通信中的基础代数规约，彻底避免 PE 矩阵核心因处理通信辅助计算而被打断。 - 网络接口配置：芯片顶部与底部各部署 6 个 Broadcom RDMA NIC，合计可提供高达 1.2 TB/s 的双向聚合网络带宽，且支持按需使能部分通道以换取最佳能效。 --- ### [07:32](https://www.youtube.com/watch?v=9OcKS-lJETM&t=452s) 消息卸载与计算-通信并发效率：MTIA 300 与主流竞品的实测对比 Wes 重点介绍了利用 MTIA 300 硬件架构的三大核心策略：首先是消息卸载（Message Offloading），利用计算与网络的高度重叠获取极致效率；其次是网络可替代性/通用性（Network Fungibility），赋能拓扑灵活重塑并精简软件层；最后则是模型与硬件由下至上的端到端贯通。进入消息卸载的实现剖析时，Wes 指出传统架构中 Host CPU 必须频繁打断并介入消息注入和事件完成信号的轮询，这不仅直接放大了网络排队时延，还强占了加速器主核计算算力。而在 MTIA 300 体系下，Meta 构建了类似 CUDA 流式处理机制的自定义通信流库（Streaming Library）。Host 仅需一次性把完整的高阶通信拓扑编排写入 HBM 工作流（Work Stream），随后消息引擎接管一切硬件操作。为了量化消息卸载带来的真实性能飞跃，Meta 设计了一项严谨的微基准测试（Benchmark）：让加速器在执行密集矩阵乘法（GEMM）的同时满负荷运行集合通信，若基准测试效率达到 100%，则表明计算与通信并发执行的耗时与两者完全独立单跑时毫无衰减，即达成了无损的理论完美重叠。在面对业界顶尖的主流商业加速芯片（Leading Commercial Product）时，MTIA 300 取得了惊人的 98% 并发重叠效率，而竞品芯片在相同场景下的实测效率仅徘徊在 70% 至 80% 之间。Wes 解密道，这正是因为 MTIA 300 拥有物理级解耦的异构硬件资源，双方运行时互不争抢、没有资源饥饿感，最终促成双向满速运转。 | 对比维度 | 传统商业 GPU / 通用加速器架构 | Meta MTIA 300 架构 | |:--- |:--- |:--- | | **通信控制关键路径** | Host CPU 持续介入网络发包与事件轮询 | 硬件消息引擎（Message Engine）全自主闭环卸载 | | **规约运算执行位置** | 主计算处理核心（PE/Tensor Core）争抢调度 | 硬件近内存计算（NMC）单元就地处理 | | **矩阵乘与集合通信重叠率** | 约 70% - 80%（硬件核心资源抢占产生性能折损） | **高达 98%**（几乎达到无损并发的理论极限） | | **大消息带宽利用率** | 随拓扑拥塞波动较大 | 16 节点聚合 scale-up/out 达到 **94% Roofline** 理论值（1.1 TB/s） | --- ### [10:38](https://www.youtube.com/watch?v=9OcKS-lJETM&t=638s) 从推荐训练到生成式推理：适配极低延迟的小消息直接注入模式在充分压榨高带宽之后，Wes 强调了另一个同等重要但截然不同的物理边界：时延瓶颈（Latency Bound）与小消息传输。在大规模集群场景下，将纵向扩展（Scale-Up）与横向扩展（Scale-Out）网络合二为一，MTIA 300 在 16 节点的实测中达到了物理 Roofline 理论峰值的 94%（吞吐达 1.1 TB/s）。对于超大报文而言带宽是绝对指标，但生成式推理时代的到来彻底颠覆了设计假设。在 MTIA 300 早期立项到最终规模化落地的周期内，核心负载直接从以海量带宽需求为主的 DLRM 训练，跨越到了以低时延为第一优先级的生成式 AI 推理任务上。这种业务形态的剧烈转向，意味着数据中心的通信形态由大作业集群吞吐全面切换为极度敏感的微秒级低延迟响应。Meta 研发团队迅速调整了通信库与硬件交互的底层逻辑：在低时延短报文场景下，原本先在 Host 定义并经由消息引擎层层下发的批处理模式反而会引入不必要的控制延迟。为了榨干每一纳秒的性能，MTIA 允许计算阵列（PE）跳过中间控制环节，直接将 RDMA 数据包精准投递到网卡的工作队列（Work Queue）中；而繁琐的消息完成追踪与细粒度状态确认则继续异步交给消息引擎通过计数器批量监听。Wes 总结道，尽管直接注入模式会轻微占用少量主核计算时间，但对于对延迟极端苛刻的推理服务来说，这笔工程置换极其划算——该方案最终成功将小消息的点到点网络延迟强势压缩至惊人的 6 微秒左右。 - 负载范式转移动因：DLRM 训练聚焦于海量参数交换的“大包大带宽”，生成式 AI 推理更偏向碎片化 Token 交互的“小包极低延迟”。 - PE 硬件直注通道：主算力核直接向 RDMA 网卡工作队列投放数据报文，消息引擎在后台通过硬件计数器批量监控交付状态，两相结合将小消息时延压制到约 6 微秒（$\mu s$）。 ### [12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) 数据中心网络架构与 MTIA 300 的网络可替代性（Network Fungibility） Wes Bland 带领观众重新审视典型数据中心网络拓扑。在现代 AI 架构中，系统主要面对两种网络形态：一种是位于同一机架（Rack）内部、加速器之间共享交换机的“Scale-up（纵向扩展）网络”，其核心优势在于提供极其充沛的峰值带宽；另一种则是连接不同机架之间交换机集群的“Scale-out（横向扩展）网络”，负责将训练或推理作业扩展到所需的海量节点规模。由于机架内的物理与电气特性，Scale-up 带宽天然远高于跨机架的 Scale-out 带宽，工程师必须借助软件层面的流水线编排（Pipelining）来掩盖跨机架通信瓶颈。在设计时，业界通常纠结于两套网络到底该采用同一种互联技术，还是采用异构的定制化技术，两者在特定负载与数据中心环境各有权衡取舍。 Meta 在业务中发现了一个清晰的计算需求分化：模型训练任务对大规模横向扩展（Scale-out）依赖极深，而生成式 AI 推理任务为了保证超低延迟响应，则极度渴求纵向扩展（Scale-up）提供的极致内存与通信带宽。为了兼顾这两种截然不同的模式，MTIA 300 确立了兼顾两者的互联战略——它让 Scale-up 与 Scale-out 共享完全相同形态的网络接口卡（NIC），实现了所谓的“网络可替代性（Network Fungibility）”。Wes 强调，Meta 的技术判断是“以太网（Ethernet）对于这一目标既是必要的，也是充分的（Ethernet is necessary and sufficient）”。团队通过与行业生态伙伴深度协作，共同推动 AI 服务基础设施的标准化并大幅压低建设成本。在物理落地配置上，MTIA 300 展现了惊人的重构弹性。标准生产环境下，单系统默认部署 10 张 Scale-up 网卡和 2 张 Scale-out 网卡，以维持支撑超大规模并行训练的经典平衡。而一旦业务转向高吞吐、低延迟的生成式 AI 推理主导模式，该系统可以快速重新配置：既可以将全部 12 张网卡统一部署为纯 Scale-up 网络，也可以将原有的 Scale-out 物理链路无缝复用为第二条 Scale-up 互联拓扑。这种灵活的硬件拓扑重组能力，为 Meta 的大规模 GenAI 服务提供了随时适配业务峰值波动的充沛容量。 ```mermaid flowchart TD accTitle: MTIA 300 网络可替代性架构 accDescr: 展示 MTIA 300 如何通过重构 12 个统一以太网网卡，在训练与推理两种模式间自由切换。 subgraph Hardware["MTIA 300 统一以太网网卡资源池（共 12 张 NIC）"] NIC12["12x 统一样式以太网 NIC"] end subgraph ConfigDefault["默认拓扑配置（侧重模型训练）"] SU_Def["10x Scale-up 网卡<br>(机架内超高带宽互联)"] SO_Def["2x Scale-out 网卡<br>(跨机架大规模扩展互联)"] end subgraph ConfigInference["重构拓扑配置（侧重生成式 AI 推理）"] SU_Inf["方案 A: 全部 12x 转换为 Scale-up 网卡<br>方案 B: 将 Scale-out 链路转为第二条 Scale-up 网络"] end Hardware -->|"默认静态划分"| ConfigDefault Hardware -.->|"动态网络重构"| ConfigInference ``` - **默认网络接口划分**：系统标配 10 张 Scale-up 网卡配合 2 张 Scale-out 网卡，精准锚定大体量混合集群的平衡点。 - **推理模式网络重构**：可直接调配全部 12 张网卡统一服务于 Scale-up，或将跨机架 Scale-out 链路重组为双重 Scale-up 通道。 - **互联技术选型理念**：坚持基于以太网构建通用通信底座，避免绑定私有封闭互联协议。 --- ### [14:28](https://www.youtube.com/watch?v=9OcKS-lJETM&t=868s) MTIA 芯片演进路线图：从专用推理到软硬件协同的飞跃 Rajeev Nair 重回讲台，向与会者公开了 MTIA 产品家族的迭代演进历程与最新研发节奏。回顾早期产品，初代的 MTIA 100 与 MTIA 200 主要聚焦于单节点推理场景，其核心设计目标是承载 Meta 内部极为关键的推荐与排序（Ranking & Recommendation）业务。而 MTIA 300 则标志着一次真正的“代际飞跃”——它成功演变为一款既能从容应对前沿模型训练、又能高效承载复杂模型推理的单芯片全功能处理器。在半导体研发领域，传统芯片项目的开发周期漫长且充满不确定性。Rajeev 直言，Meta 正在有意识地大幅压缩这些原本漫长的交付窗口。通过将前代产品的实战经验快速沉淀为通用模块与可复用资产，团队建立了极为高效的迭代体系，使得设计架构兼具极强的可扩展性与平滑演进能力。目前代号为 MTIA 450 与 MTIA 500 的后续芯片研发均已步入正轨，预计将在来年正式落地。这意味着 Meta 已经具备了“每年大约推进两代芯片”的超快工程研发节奏。 - **MTIA 100 / 200**：专注于单节点推理加速，驱动 Meta 核心推荐系统与内容排序流水线。 - **MTIA 300**：技术里程碑式节点，首次在统一单芯片硅片上兼备前沿模型训练与大规模生成式推理能力。 - **MTIA 450 / 500**：研发进展顺利，预定次年完成交付部署，拉动 Meta 实现一年两代的超密集硬件迭代。 - **技术白皮书细节**：演讲提及有关 MTIA 硅片内部微架构更详细的工程拆解，已于当年 3 月在其官方工程技术博客（Engineering Blog）上公开发布。 --- ### [15:53](https://www.youtube.com/watch?v=9OcKS-lJETM&t=953s) 软硬件协同设计的未来愿景：以 PyTorch 为根基连接数十亿用户 Rajeev 在总结演讲时指出，虽然整场分享穿插了网络可替代性、底层通信延迟消除机制以及业务端剧烈变化的吞吐需求等海量细节，但如果观众只能带走一个最核心的结论，那就是“高效 AI 服务的未来完全取决于协同设计（The future of efficient AI serving is co-design）”。硬件、底层系统软件与 AI 模型算法之间的极度紧密耦合，正是当今整个行业技术创新迸发得最激烈的核心交汇点。这项庞大的工程并非闭门造车，Meta 始终将自身定位于开源生态与产业协作的推动者。MTIA 在微架构设计之初就确立为“PyTorch 原生系统（PyTorch-native system）”。这种设计对上层模型算法研发人员带来了立竿见影的收益——他们无需再消耗数月时间去为异构硬件逐个手写、移植底层计算算子内核（Kernel），而是可以直接调用熟悉的工具链，将精力百分之百投入到“构建未来前沿模型”这一最具价值的核心业务上。演讲最后，Rajeev 动情地表示，Meta 的技术团队始终为能亲手构筑这套全球规模的基础设施而深感自豪。正是这套横跨硬件、通信、系统软件的全栈协同设计成果，让最前沿的人工智能能力得以顺畅穿透到底层网络，通过 Meta 的全系列核心应用程序与服务，全天候赋能全球数十亿终端用户。现场在充满期待的掌声与走廊交流邀请中圆满落幕。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=9OcKS-lJETM&t=4s) 如果整场演讲只能记住一个词，那就是“协同设计”（Co-Design）。 — Rajeev Nair > [00:04](https://www.youtube.com/watch?v=9OcKS-lJETM&t=4s) 算力需求的大幅增长是每个企业都在面对的普遍常态，并非决定性差异；真正能跟上这一演进节奏的唯一解法，就是全栈协同设计——将模型在算法层的严苛诉求，一路深度穿透到底层数据传输的物理实现。 — Rajeev Nair > [02:01](https://www.youtube.com/watch?v=9OcKS-lJETM&t=121s) 这种高效的重叠能力，就是系统架构博弈的全部精髓所在。 — Rajeev Nair > [04:57](https://www.youtube.com/watch?v=9OcKS-lJETM&t=297s) 虽然它尺寸很小可能在台上看不清，但欢迎大家散场后在走廊找我近距离观摩 — Wes Bland > [12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) Ethernet is necessary and sufficient — Wes Bland > [12:13](https://www.youtube.com/watch?v=9OcKS-lJETM&t=733s) 以太网（Ethernet）对于这一目标既是必要的，也是充分的 — Wes Bland > [15:53](https://www.youtube.com/watch?v=9OcKS-lJETM&t=953s) 高效 AI 服务的未来完全取决于协同设计（The future of efficient AI serving is co-design） — Rajeev Nair > [15:53](https://www.youtube.com/watch?v=9OcKS-lJETM&t=953s) The future of efficient AI serving is co-design — Rajeev Nair ## Mentioned Rajeev Nair, Wes Bland, Meta, MTIA 300, MTIA 100, MTIA 200, MTIA 450, MTIA 500, DLRM, Message Engine, Near-Memory Compute (NMC), Processing Elements (PEs), HBM, Broadcom RDMA NIC, CUDA, GEMM, PyTorch, ISCA, SC (Supercomputing Conference), Santa Clara Convention Center ## Video description Rajeev Nair and Wes Bland from Meta present "Co-Designing Communication for AI Accelerators" at the Santa Clara Convention Center. This presentation details how Meta is addressing the massive compute and data demands of modern generative AI workloads by co-designing hardware, software, and models. The talk highlights the communications architecture of MTIA 300, Meta's custom-designed AI training and inference accelerator featuring dedicated message engines to efficiently offload communication workloads. By decoupling communication from the primary compute grid, the system minimizes resource contention and substantially drives up both compute and network efficiency. Additionally, the speakers explore how "network fungibility" allows the same network interface cards (NICs) to flexibly adapt to both scale-up and scale-out configurations depending on production needs. Ultimately, the presentation demonstrates how a tight co-design loop across infrastructure layers provides a scalable, future-ready roadmap for Meta’s rapidly evolving AI demand. Learn more about @Scale here: atscaleconference.com [Source](https://www.youtube.com/watch?v=9OcKS-lJETM)

Source quotes

[0:04] Rajeev Nair: 如果整场演讲只能记住一个词，那就是“协同设计”（Co-Design）。 [0:04] Rajeev Nair: 算力需求的大幅增长是每个企业都在面对的普遍常态，并非决定性差异；真正能跟上这一演进节奏的唯一解法，就是全栈协同设计——将模型在算法层的严苛诉求，一路深度穿透到底层数据传输的物理实现。 [2:01] Rajeev Nair: 这种高效的重叠能力，就是系统架构博弈的全部精髓所在。 [4:57] Wes Bland: 虽然它尺寸很小可能在台上看不清，但欢迎大家散场后在走廊找我近距离观摩 [12:13] Wes Bland: Ethernet is necessary and sufficient [12:13] Wes Bland: 以太网（Ethernet）对于这一目标既是必要的，也是充分的 [15:53] Rajeev Nair: 高效 AI 服务的未来完全取决于协同设计（The future of efficient AI serving is co-design） [15:53] Rajeev Nair: The future of efficient AI serving is co-design

Source quotes

[0:04] Rajeev Nair: co-design [9:19] Wes Bland: 98% efficiency [9:19] Wes Bland: 70% to 80% efficiency [9:19] Wes Bland: 94% of roofline memory bandwidth [9:19] Wes Bland: 6 microseconds [7:32] Wes Bland: network fungibility [14:28] Rajeev Nair: two silicon generations per year

Original transcript

[0:04] Good afternoon everyone. Thanks for [0:06] being here. [0:08] Today we are going to unpack the journey [0:11] of building full communication stack [0:13] right from silicon all the way to models [0:17] for a accelators. Les will be back in a [0:21] bit to talk more about this a little [0:22] later. [0:25] The one word to leave with is core [0:28] design. [0:29] Not software followed by hardware. Not [0:32] networking bolted on at the end. [0:35] Networking silicon communication stack [0:39] kernels all designed together in one [0:43] loop. [0:45] The future of efficient AI serving is [0:49] core design. [0:51] Let us ground us with how this matters [0:55] with how fast the workloads have [0:57] evolved. [1:03] Before Gen AI, ranking and [1:06] recommendations were our main focus that [1:10] drove infra. [1:12] In 2022, [1:14] Genaii training scales fast. By 2025, [1:19] agentic workloads evolve as a new source [1:22] of demand. [1:25] Every one of these shifts introduce new [1:29] challenges [1:30] like message sizes, latency [1:33] optimizations and a demand on [1:37] collective patterns. [1:40] The constant isn't compute demand. [1:43] Everyone has that. It is that the only [1:47] way to keep pace is code design. [1:52] Pulling models needs all the way down to [1:56] how we move data. [1:59] Let's make that concrete. [2:01] Let's see what actually happens in an A [2:04] run. [2:09] At its core, [2:11] an efficient serving is challenged with [2:13] the data that will not fit on one [2:15] machine, [2:17] weights, activations, tokens. [2:21] So, we build systems that do two things [2:24] well. An accelerator [2:27] um that does the compute and [2:30] communications that distribute and [2:34] aggregate data. [2:37] Let's look at on a time scale what [2:40] happens right in a conventional design [2:43] compute communications compute [2:46] communications strictly serialized [2:50] easy to design easy to understand [2:53] but half your data center is idle at any [2:57] given moment. [2:59] What we need is overlap. [3:02] restructure the model in a way that we [3:05] have data to move while we compute on [3:08] some other data. [3:11] Compute chips and networks are fully [3:14] saturated [3:16] that is more serving per unit of time. [3:20] That overlap is a whole game. [3:24] Let's look at the hardware that meant to [3:26] deliver it. [3:32] Here is a simplified view of most [3:34] systems, a CPU and an accelerator. [3:38] The host generates [3:40] compute work for the accelerator. We [3:43] call it compute kernels. [3:46] Then the host generate work for network [3:49] interfaces, [3:51] network instructions. [3:54] uh then the nick pulls data directly [3:57] from accelerators memory. A deliberate [4:00] choice nick sits very closer to uh [4:02] memory so that it doesn't pull data [4:05] across something like PCIe adding [4:07] additional bottlenecks [4:10] but the host is still in the critical [4:13] path [4:15] as we discussed boundaries between [4:17] compute and networking is blurring. Fuse [4:21] kernels merge compute and communications [4:24] aggressively. [4:26] So our job is to keep the silicon design [4:29] futurep proof to support these varying [4:33] sets of requirements. [4:35] I am in Meta's host networking team. I [4:38] support the communications team for [4:41] MTIA. [4:42] With that, let me hand to Wes to talk [4:45] more about how we solve these important [4:48] challenges in MTA 300 meta straining and [4:52] influence accelerator. [4:57] So, hi, I'm Wes. I've been working in [5:01] HPC and AI communications for a couple [5:03] of decades and I work on the team that [5:05] develops the communications library for [5:07] MTIA. [5:08] So with the design that Rejie just [5:10] described, we saw a lot of opportunities [5:13] where we could optimize not just our [5:15] models but our silicon for those models [5:18] and that led to MTI300. [5:20] So deep learning recommendation models [5:22] or DRM require lots of compute and lots [5:26] of communication and so we baked that [5:28] communication directly onto the chip and [5:30] this is what it what came out. So this [5:32] is MTI300. I know it's very small and [5:34] you can't see it here, but feel free to [5:36] find me in the hall after. Be happy to [5:38] show you. So, we want to look at some of [5:40] the differences between this chip and [5:42] some of the other chips of a similar [5:43] generation. [5:46] So, this is a cartoon that's similar to [5:47] what Rejie just showed, but has some [5:49] important additions that come along with [5:51] MTI300. [5:52] So, in the center, you can see that we [5:54] added message engines. These batch [5:56] instructions from the host then drive [5:58] that traffic to the nick monitors for [6:01] completion and tracks dependencies [6:03] between messages. Once the messages go [6:06] to the message engine, the host isn't [6:08] involved and the accelerator doesn't [6:10] need to drive any traffic or compute for [6:13] the communication side. There's a lot of [6:15] details about the the chip design and [6:18] the software stack that goes along with [6:20] it in some recent uh meta-engineering [6:22] blog posts. one from yesterday, another [6:24] one from March and we also have papers [6:26] this year at ISA and coming up this year [6:29] at supercomputing that you can go deep [6:30] dive into all of this. [6:33] So this shows you a little more detailed [6:35] description of the chip itself. So in [6:38] the center you can see a big grid of [6:39] processing elements or pees and then on [6:42] the sides you can see uh the message [6:44] engines as well as the high bandwidth [6:46] memory. on those message engines we have [6:49] some components called nearmemory [6:51] compute or NMC's. This allows us to do [6:55] uh to offload compute work that we need [6:57] in order to do the communication [6:59] patterns. Uh and that happens without [7:02] needing to go back to the pees to to [7:04] drive that um that arithmetic. So we're [7:07] able to pipeline receiving data with [7:09] accumulating it directly in memory [7:11] because those NMC's as the name suggests [7:14] are right next to the memory. Then on [7:17] the top and the bottom we have two banks [7:18] of six Broadcom RDMA nicks and those [7:21] aggregate to 1.2 terabytes of bandwidth [7:24] although we can choose to enable fewer [7:26] if we don't need all that bandwidth and [7:28] that gives us better efficiency. [7:32] So how do we actually leverage this [7:34] hardware? We want to focus on three [7:36] things in this talk. The first is [7:38] offloading. How do we accelerate our [7:40] networking by overlapping with compute [7:43] and how does that give us better [7:45] efficiency? The second is network [7:47] fungeibility. [7:49] This allows us to be flexible as our [7:51] workloads change over time. It also [7:53] simplifies the software that we that we [7:55] develop in order to support this. And [7:57] then last, Rejie will come back and talk [7:59] more about how we integrate our models [8:01] with the hardware and bring that codees [8:04] all the way through the stack. [8:07] So first, let's go into message [8:08] offloading. [8:10] So traditional accelerators have a host [8:12] CPU that injects traffic into the [8:14] network which adds latency to in to both [8:17] add those messages to the network and [8:19] signal the readiness and arrival of of [8:21] messages that have come in. So these [8:24] also use uh the compute resources on the [8:27] accelerator itself because they need to [8:29] do the arithmetic uh as the as the [8:32] messages come in. On MTI300, [8:36] we have those message engines which we [8:38] can use, as I said, to define the entire [8:40] communication pattern first on the host. [8:43] It's copied into uh host or into the [8:45] HPM, the high bandwidth memory. And at [8:48] that point, the host CPU isn't involved [8:49] at all. We also can use NMC to avoid [8:52] using the the PE cores. And all of this [8:55] is enabled through a streaming library [8:57] similar to CUDA which allows us to encue [9:00] work uh on the host CPU while something [9:02] else is taking place on the accelerator. [9:05] So we copy everything to the HBM. It [9:08] goes into a work stream and this [9:10] overlaps queuing the work and executing [9:12] the work and and compute and comput and [9:16] communication. [9:19] All of this gives us better efficiency. [9:21] So we created a benchmark that overlaps [9:23] matrix multiplication with collective [9:26] communication. So in this benchmark 100% [9:28] would indicate that you get the same [9:30] performance when you run compute and [9:32] communication at the same time as you do [9:34] when they are uh as you do when they're [9:36] separate. [9:38] So that means we're achieving the best [9:39] overlap as Rejie was talking about. [9:42] So we compared MTI300 to a leading [9:45] commercial product. What we saw was that [9:47] while MTI300 could achieve about 98% [9:50] efficiency, when we compared that to [9:52] other chips, we would see something more [9:53] like 70 to 80%. Depending on the [9:56] workload and the uh communication [9:58] pattern and this is because we're we [10:00] have distinct resources for the two [10:02] different types of of workloads that [10:04] we're trying to run at the same time. [10:05] There's no resource starvation and this [10:08] speeds up both at the same time. [10:12] So all of this improves bandwidth which [10:14] is important for large messages. And the [10:17] figure you can see here on the top shows [10:19] that we're about we're able to achieve [10:20] about 94% of roof line if we combine our [10:23] scale up and scale out networks. Uh that [10:26] would be one point one terabyte per [10:28] second in this example and this is with [10:31] 16 nodes. So higher is better and we [10:34] care about the largest messages on the [10:36] right side of the graph. [10:38] However, sometimes we're latency bound. [10:40] We care about small messages. So as [10:43] we've already said in between the [10:45] beginning MTI300's design and its [10:48] deployment, we moved from DLRMS being [10:51] the primary workload to generative AI [10:54] completely changing the type of workload [10:56] we had and also moving from training to [10:58] inference as uh as an important use [11:00] case. And this means we go from [11:03] largecale jobs primarily bandwidthbound [11:06] to smaller much smaller jobs that are [11:09] very latency sensitive. This meant that [11:11] we had to adapt not not only the [11:13] workloads but how the communication [11:15] library would interface with them. [11:17] Suddenly it didn't always make sense to [11:19] predefine everything on the host offload [11:22] it to the me. Sometimes we want to [11:24] inject messages directly from the [11:26] compute core into the the network uh [11:28] itself. So in this case the PE puts to [11:31] an RDMA packet directly into the nick [11:33] work Q and then the MES can monitor uh [11:36] completion which is something that the [11:38] uh the processing elements don't [11:40] necessarily need to know on a fine grain [11:42] level. The PE can batch all that [11:44] together with a counter and then uh the [11:46] PE can find out about that later. So all [11:49] of this does consume time on the compute [11:52] core but in this case for very latency [11:54] sensitive workloads that's a good [11:56] trade-off. [11:57] And what we were able to see is that we [11:59] can lower latency to uh around six [12:01] microsconds for small messages here on [12:04] the bottom. And in this case, of course, [12:06] lower is better. And we care most about [12:08] the message sizes on the left side of [12:09] the graph. [12:13] So to talk about other ways we can [12:14] adapt, it's helpful to quickly look at [12:16] the design of a typical data center [12:17] network. And I know we've already seen [12:19] this picture multiple times through the [12:21] day, but just as a quick reminder, uh we [12:23] basically have two types of networks [12:25] that we're that we're concerned about in [12:27] this case. Uh accelerators within the [12:29] same rack share one switch. We call that [12:31] usually a scale up network. This is [12:33] where we're going to have maximum uh [12:35] bandwidth. Between racks, we have [12:37] another switch or set of switches that [12:39] would be a scale out network. And this [12:42] allows us to scale the job to be as [12:43] large as we need it to be. So of course [12:46] the bandwidth within the rack is going [12:47] to be much higher than bandwidth uh [12:49] between racks and we you need to use uh [12:52] pipelining and software to optimize for [12:54] that. Now in this case you could have [12:56] the same interconnect for both networks [12:58] or you could have different [12:59] interconnects. There are trade-offs [13:01] depending on your workload depending on [13:03] your your data center. Either one could [13:05] be uh could be a good choice. What we [13:08] found was that because training is [13:11] needed for or because training requires [13:12] large scale up and inference requires [13:15] maximum bandwidth in the scale or [13:17] training requires scale out and [13:19] inference requires scale up. We wanted [13:21] to optimize the system for both at the [13:23] same time [13:25] and MTI300 gives us that best of both [13:28] worlds. We use the same nicks for both [13:30] scale up and scale out and that means we [13:32] can reconfigure the system to adapt to [13:34] our needs. And so what we're saying is [13:36] Ethernet is necessary and sufficient for [13:38] this and we can work with ecosystem [13:40] partners to optimize what they're good [13:43] at and then together we can make AI [13:45] serving as standard and inexpensive as [13:47] possible. So by default we deploy 10 [13:49] scale up nicks and two scale outs on our [13:52] system and this allows us to strike that [13:54] balance for larger deployments. But then [13:56] when we want to have a a more inference [13:58] focus, we can reconfigure it to either [14:00] use all 12 nicks for scale up or we can [14:02] use the scale out network as a second [14:04] scale up network. And this gives us the [14:07] capacity we want for inference scales uh [14:09] for serving genai. [14:12] So next Rajie is going to come back and [14:13] talk a little bit more about our [14:15] hardware roadmap and how our codeesign [14:17] work allows us to tie back feedback from [14:19] the models all the way down to the [14:20] hardware. [14:28] Thanks Wes. Um the same timeline now [14:31] with our products on it. Initial MTA [14:35] versions 100 and 200 were mostly focused [14:38] on single node influence influence [14:42] primarily for ranking and [14:43] recommendations. MTA 300 is the leap, a [14:47] single chip that can do both training [14:50] and inference. [14:52] Each generation banks on past [14:54] generations learning so we can push [14:57] newer generations faster into [15:00] production, some general, some [15:02] specialized. [15:04] Silicon programs are difficult and [15:07] lengthy as you all aware are aware. We [15:11] are compressing these timelines. We have [15:13] improved our processes to launch [15:17] successive generations to production [15:20] faster. Moreover, we made the design [15:24] scalable and easy to evolve. [15:28] Work on 450 and 500 are well underway. [15:32] We anticipate to land them next year. [15:36] So, we are able to hold the curtains [15:38] that is roughly two generations every [15:41] year. [15:43] Deep dive on all the various uh MTA [15:46] silicons are actually available on our [15:49] engineering blog post from past March. [15:53] Let's bring it all together. [15:58] We have covered a lot network funs [16:02] latency optimizations the customer [16:05] changing demands and stuff like that. [16:08] Right? [16:10] If you take one thing away, the future [16:13] of efficient AI serving is core design. [16:18] It's a tight integration between [16:19] hardware, software, and the models [16:21] themselves. [16:24] That intersection where hardware meets [16:26] the software is exactly where most [16:29] exciting innovation is happening right [16:31] now. [16:33] And we are not doing this alone. We [16:36] build alongside industry partners. That [16:38] means all of you. [16:41] Meta is deeply committed to this [16:43] collaborative spirit. A fully integrated [16:46] system takes all our expertise combined. [16:52] We have built MTIA as a PyTorch native [16:55] system. That means for you model [16:58] developers, less time porting the kernel [17:01] and doing what you do best. Build the [17:03] future. [17:06] Ultimately, we are proud to build the [17:09] infrastructure that brings AI to [17:11] billions of people around the world [17:13] through our core apps and services. [17:18] It's an exciting journey and we are [17:20] delighted to share it with you all. [17:24] Thank you for listening. [17:27] If this work is interesting to you, [17:28] please catch us up in the holidays.