Meta Deploys MTIA 300 Silicon To Decouple AI Communication

Co-Designing Communication for AI Accelerators | Rajeev Nair and Wes Bland from Meta — @Scale

@Scale

By offloading communication tasks to dedicated hardware Message Engines, Meta's MTIA 300 chip achieves 98 percent overlap efficiency and eliminates the core contention plaguing standard GPUs.

AI-assisted summary of Co-Designing Communication for AI Accelerators | Rajeev Nair and Wes Bland from Meta — original source. Check the original for full context.

Source published:

Detailed notes

## Background This presentation was published on YouTube by @Scale on August 27, 2026. According to the page description, it details how Meta is addressing the massive compute and data demands of modern generative AI workloads by co-designing hardware, software, and models across infrastructure layers.

## TL;DR Meta engineers Rajeev Nair and Wes Bland outline Meta's co-design philosophy for AI accelerators, showcasing how integrating silicon, networking, software, and models optimizes distributed AI infrastructure. The presentation introduces the MTIA 300 accelerator, detailing its dedicated hardware Message Engines and Near-Memory Compute (NMC) units that offload communication and reduction workloads to maintain near-perfect compute efficiency. Furthermore, it demonstrates how network fungibility allows identical Broadcom Ethernet NICs to dynamically adapt between scale-up and scale-out topologies to serve evolving recommendation and generative AI demands.

## Atmosphere & style The presentation carries an authoritative, highly technical, and forward-looking tone characteristic of an engineering conference showcase. The transition between Rajeev Nair's macro-level strategic vision and Wes Bland's granular architectural deep-dive creates a cohesive narrative arc bridging workload demands with concrete hardware solutions. The delivery feels precise, methodical, and data-driven, conveying a sense of rapid, relentless innovation behind Meta's custom silicon pipeline.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=4s) Meta relies on holistic co-design across silicon, network architecture, communication libraries, kernels, and machine learning models to keep pace with evolving AI demands. - [00:04](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=4s) Rapid shifts in Meta's workload landscape—from pre-2022 Ranking & Recommendation systems to 2022 Gen AI training and 2025 agentic AI—continually reshape network traffic and hardware requirements. - [01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) Sequential compute-then-communicate execution leaves hardware idling roughly half the time, making execution pipeline overlapping essential to keep compute cores and network links saturated. - [03:05](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=185s) Traditional setups that require the host CPU to orchestrate every network command introduce significant latency overheads on the critical path. - [04:26](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=266s) MTIA 300 (introduced March 2024) is Meta's custom accelerator designed specifically for both DLRM and Generative AI training and inference workloads. - [04:26](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=266s) MTIA 300's architecture offloads messaging to autonomous Message Engines and Near-Memory Compute (NMC) units, keeping main Processing Element (PE) compute cores free for matrix multiplication. - [07:22](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=442s) Stream-based queuing decouples host CPU work enqueueing from accelerator hardware execution, enabling 100% pipeline overlapping without host dependency. - [08:55](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=535s) In benchmarking, MTIA 300 sustained ~98% efficiency during simultaneous compute and communication, outperforming NVIDIA H100 GPUs (70%–80%) due to hardware communication offloading. - [09:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=592s) MTIA 300 delivers ~94% of roofline large-message bandwidth, while allowing PE direct packet injection to reduce single-message latency below 6 microseconds for small-message inference. - [11:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=712s) Meta uses standard Broadcom RDMA Ethernet NICs to deliver network fungibility, allowing NICs to be dynamically reconfigured between scale-up intra-rack and scale-out inter-rack domains. - [14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) Meta aims for an aggressive release cadence of roughly two custom silicon generations entering production per year, spanning MTIA 300 through upcoming MTIA 450 and 500. - [14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) MTIA is built as a PyTorch-native platform to minimize developer friction and allow seamless lowering from high-level models to silicon execution.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=4s) The Shift from R&R to Gen AI and the Case for Co-Design

The talk opens with Rajeev Nair introducing the central thesis of Meta’s infrastructure development: co-design. Rather than designing hardware first, layering software on top, and bolting networking on at the end, Meta designs the entire stack—silicon, network architecture, communication libraries, kernels, and machine learning models—in a single continuous loop. This holistic integration is presented as the only viable path to efficiently serve modern AI workloads given how drastically traffic patterns and model requirements have shifted over recent years.

Rajeev walks through the rapid evolution of Meta’s workload demand, tracing how the infrastructure shifted from traditional Ranking and Recommendation (R&R) systems prior to 2022, to the massive scale-up of Generative AI training in 2022, and eventually to agentic workloads by 2025. Every single shift alters message sizes, collective communication patterns, latency tolerances, and overall traffic shapes. Addressing these dynamic demands requires pulling model requirements all the way down into the physical data movement layers.

- **Core Paradigm:** Co-designing hardware, network, communication libraries, kernels, and models together in one feedback loop. - **Workload Timeline:** - *Pre-2022:* Ranking and Recommendation (R&R) systems driving the bulk of infrastructure. - *2022:* Rapid scale-up of Generative AI training workloads. - *2025:* Emergence of agentic AI workloads introducing new traffic patterns and communication demands.

[01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) Overlapping Compute and Communication to Maximize Accelerator Efficiency

At its core, efficient AI serving is a distributed data problem: model weights, activations, and KV cache tokens exceed the memory limits of a single machine. To solve this, systems split work into compute (executing on the accelerator) and communication (aggregating and redistributing intermediate results across nodes). Because compute silicon represents the rarest and most expensive resource, the overarching goal is keeping both the compute cores and the network interfaces completely saturated at all times.

In traditional node execution, compute and communication happen sequentially: the system computes, then communicates, then computes again. This clean, linear dependency model leaves half the hardware idle at any given moment. To fix this, Meta restructures models and execution pipelines so data is prepared and sent across the network while the compute grid simultaneously processes the next slice of data. Overlapping these two phases allows more model serving output within the exact same unit of time.

- **Sequential Execution Flaw:** Compute $\rightarrow$ Communicate $\rightarrow$ Compute leads to ~50% hardware idleness. - **Goal:** Overlap compute and communication to keep both the accelerator grid and network links fully saturated. - **Mechanism:** Restructure model execution so data is transmitted as soon as it is produced, during concurrent compute operations.

[03:05](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=185s) Traditional Accelerator Bottlenecks and Host CPU Inefficiencies

Rajeev illustrates a simplified traditional accelerator architecture consisting of a host CPU and a processing accelerator. In typical setups, the host CPU acts as the central traffic controller: it generates compute kernels for the accelerator and network instructions for the Network Interface Cards (NICs), telling the NIC to pull data directly from accelerator memory. While placing the NIC close to accelerator memory prevents dragging data back and forth across a PCIe bottleneck, keeping the host CPU in the critical path for every single network message introduces substantial overhead and latency.

Furthermore, modern AI workloads increasingly blur the boundaries between compute and communication. Aggressive techniques like kernel fusion mix mathematical computations and network transfers within the same execution blocks. Keeping silicon future-proof requires relieving the host CPU from message orchestration while providing native hardware capabilities that support these fused communication patterns.

- **Traditional Pattern:** Host CPU issues both compute kernels to the accelerator and network commands to the NIC. - **PCIe Bypass:** Direct memory access between accelerator HBM and NICs reduces bus bottlenecks. - **Critical Path Problem:** Host CPU involvement in every network instruction introduces latency and overhead.

[04:26](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=266s) Introducing MTIA 300 and Hardware Architecture Deep-Dive

Wes Bland takes over to detail how Meta designed MTIA 300 (Meta Training and Inference Accelerator 300), which debuted in March 2024 as Meta's first custom silicon engineered specifically to handle Deep Learning Recommendation Models (DLRM) alongside Generative AI workloads. MTIA 300 introduces dedicated hardware communication offloading alongside its primary compute grid to eliminate host dependencies and compute core starvation.

The physical chip architecture features a large central grid of Processing Elements (PEs) flanked by banks of dedicated Message Engines (MEs) and High Bandwidth Memory (HBM). The Message Engines operate autonomously by receiving batched network instructions, driving message submission, tracking completions, and managing dependencies without involving the host CPU or consuming PE compute cycles. Built inside these message engines are "Near-Memory Compute" (NMC) units designed specifically for inline reduction operations (such as data accumulation in `AllReduce` collectives) directly as data arrives from peer chips.

- **Launch Date:** MTIA 300 introduced in March 2024. - **Target Workloads:** Deep Learning Recommendation Models (DLRM) and Generative AI training/inference. - **On-Chip Architecture:** - Central grid of Processing Elements (PEs). - High Bandwidth Memory (HBM) stacks on the chip margins. - Dedicated Message Engines (MEs) handling queue submission, completion tracking, and dependency management. - Near-Memory Compute (NMC) units within MEs performing inline reduction operations (e.g., `AllReduce` accumulations) off the main compute grid. - **Networking Specs:** Two banks of 6 Broadcom RDMA NICs providing up to 1.2 TB/s aggregate bandwidth (typically configured to 1.0 TB/s for optimal energy and operational efficiency).

[07:22](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=442s) Hardware Offloading and Stream-Based Queuing Efficiency

In traditional setups, the host CPU injects network messages step-by-step and the accelerator grid executes mathematical reductions for collective communication, taking precious execution cycles away from matrix multiplications (GEMMs). MTIA 300 fundamentally shifts this approach by defining the entire communication pattern up front and offloading execution entirely to the hardware Message Engines and Near-Memory Compute units.

MTIA 300 achieves this through a streaming work paradigm similar to NVIDIA CUDA. Applications enqueue work batches into streams on the host CPU, which copies instructions into the accelerator’s HBM. The host CPU is immediately freed to queue future work while the Message Engines execute tasks from HBM as they reach the front of the queue. The host is only notified upon full task completion.

- **Offloaded Architecture:** Communication patterns defined entirely ahead of execution; MEs execute autonomously without host intervention. - **Streaming Model:** Enqueueing work on the host CPU is decoupled from execution on the chip, allowing 100% pipeline overlapping. - **Resource Preservation:** Near-Memory Compute units execute reduction operations, keeping 100% of PE compute cores free for matrix multiplication.

[08:55](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=535s) Benchmarking Overlap Performance: MTIA 300 vs. NVIDIA H100

To prove the efficacy of hardware message offloading, Meta ran head-to-head benchmarking tests overlapping matrix multiplication compute with collective communication tasks simultaneously. In these benchmarks, a score of 100% represents perfect overlap efficiency—meaning simultaneous compute and communication take no longer to complete than running the compute operation alone in isolation.

The benchmark results revealed a stark performance difference between MTIA 300 and standard GPUs. While NVIDIA H100 GPUs experienced an efficiency degradation down to 70%–80% of peak during simultaneous compute and communication (due to compute core starvation driving network traffic and inline arithmetic), MTIA 300 sustained approximately 98% efficiency. Because MTIA 300 offloads scale-up traffic and inline reduction arithmetic to dedicated Message Engines and Near-Memory Compute hardware, its main compute grid remains entirely unstarved.

- **Benchmark Definition:** 100% efficiency score = simultaneous compute + communication executes in the same duration as standalone compute. - **NVIDIA H100 GPU Efficiency:** Drops to **70%–80%** of peak performance during simultaneous compute/communication due to core contention. - **MTIA 300 Efficiency:** Maintains **~98%** efficiency owing to dedicated Message Engines and NMC hardware.

``` Simultaneous Compute + Communication Efficiency -------------------------------------------------- NVIDIA H100 GPU: [========75%======== ] 70-80% MTIA 300: [========98%========== ] 98% -------------------------------------------------- ```

[09:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=592s) Large Message Bandwidth and Ultra-Low Latency PE Injection

MTIA 300 delivers high-bandwidth performance for large payloads, reaching approximately 94% of the theoretical roofline limit (achieving 1.0 TB/s bandwidth across 16 nodes utilizing combined scale-up and scale-out networks). However, because AI inference workloads are frequently latency-bound by small message transfers, Meta also implemented an alternative direct-injection path bypassing the Message Engines.

To minimize latency for small messages, Processing Element (PE) cores can directly construct Remote Direct Memory Access (RDMA) packets and push them into the desired NIC's queue. While consuming a PE compute cycle, this direct injection path drops single-message latency below 6 microseconds ($\mu\text{s}$). The Message Engines are still utilized to track message completions via counter updates, simplifying the workload on the PE cores and enabling tight kernel fusion of compute and network operations in latency-sensitive inference scenarios.

- **Large Message Bandwidth:** Reaches **~94% of roofline efficiency** (1.0 TB/s across 16 nodes). - **Small Message Latency:** Direct PE packet injection achieves **$<6\ \mu\text{s}$ single-message latency**. - **PE Direct Injection Trade-off:** Consumes a compute core briefly to format and enqueue RDMA packets, but drastically cuts latency for small-message, latency-critical inference workloads.

[11:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=712s) Flexible Data Center Networks and Network Fungibility

In typical AI data center deployments, accelerators within a single rack communicate over high-bandwidth links forming a "scale-up domain." Racks connect to each other over a secondary network layer termed the "scale-out domain." While large AI training runs require massive scale-out connectivity across hundreds of racks, AI inference workloads typically do not scale beyond a single rack. Equipping dedicated inference nodes with expensive, underutilized scale-out networking leads to poor capital expenditure and operational inefficiency.

MTIA 300 solves this through "network fungibility"—using identical Broadcom Ethernet NICs for both scale-up and scale-out traffic. Meta firmly maintains that standard Ethernet is both necessary and sufficient for modern AI scale, relying on ecosystem partners like Broadcom to optimize the underlying NIC hardware. Because the NICs are identical, the network topology can be dynamically reconfigured depending on the deployed workload.

- **Network Topologies:** - *Scale-Up Domain:* High-bandwidth interconnect between accelerators within the same rack. - *Scale-Out Domain:* Inter-rack network connecting multiple racks for multi-node scale. - **Standardization:** Built on standard Ethernet technology via Broadcom RDMA NICs. - **Default MTIA 300 Configuration:** - **10 NICs** allocated to Scale-Up inside the rack (optimized for DLRM high-bandwidth intra-rack communication). - **2 NICs** allocated to Scale-Out across racks. - **Fungible Reconfiguration:** For single-rack inference workloads, the 2 scale-out NICs can be repurposed as additional scale-up channels, increasing intra-rack bandwidth and eliminating wasted networking hardware.

[14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) Meta's Silicon Cadence and the Future of Co-Design

Rajeev Nair returns to present Meta’s silicon release roadmap, demonstrating how hardware cycles have accelerated over time. Initial iterations (MTIA 100 and 200) were specialized single-node inference chips built primarily for Ranking and Recommendation (R&R) models. MTIA 300 represents a major architectural leap as a unified silicon platform supporting both training and inference workloads.

Meta has revamped its silicon development processes to dramatically compress design cycles, enabling a release cadence of approximately two silicon generations launched into production every single year. Successive generations—including MTIA 450 and MTIA 500—are already well under development.

Rajeev concludes by emphasizing that MTIA is built as a PyTorch-native platform designed to eliminate friction for model developers, minimizing code porting and allowing researchers to focus on model innovation.

- **MTIA Generations:** - *MTIA 100 & 200:* Specialized single-node inference accelerators for R&R workloads. - *MTIA 300:* Unified architecture supporting both training and inference. - *MTIA 450 & 500:* Next-generation silicon currently under active development. - **Production Cadence Commitment:** Targeting approximately **two silicon generations entering production per year**. - **Software Integration:** Native PyTorch integration ensuring seamless model translation from software down to physical silicon.

<ElicitationsGroup message="Explore related technical aspects of Meta's MTIA architecture:"> <Elicitation label="Deep-dive into Near-Memory Compute (NMC) hardware architecture" query="Can you explain the hardware architecture of Near-Memory Compute (NMC) units in MTIA 300 and how they handle inline reductions?"/> <Elicitation label="Compare MTIA 300 scale-up vs scale-out network topologies" query="How does MTIA 300's network fungibility compare structurally to traditional InfiniBand or custom NVLink interconnect networks?"/> <Elicitation label="Detail MTIA's PyTorch integration and compiler stack" query="How does MTIA's PyTorch-native software stack lower workload execution onto MTIA 300's Message Engines and PE grid?"/> </ElicitationsGroup>

## Notable quotes > [00:04](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=4s) Co-designing hardware, network, communication libraries, kernels, and models together in one feedback loop. — Rajeev Nair

> [01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) Compute -> Communicate -> Compute leads to ~50% hardware idleness. — Rajeev Nair

> [01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) Overlap compute and communication to keep both the accelerator grid and network links fully saturated. — Rajeev Nair

> [03:05](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=185s) Host CPU involvement in every network instruction introduces latency and overhead. — Rajeev Nair

> [04:26](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=266s) Near-Memory Compute (NMC) units within MEs performing inline reduction operations (e.g., AllReduce accumulations) off the main compute grid. — Wes Bland

> [07:22](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=442s) Communication patterns defined entirely ahead of execution; MEs execute autonomously without host intervention. — Wes Bland

> [07:22](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=442s) Near-Memory Compute units execute reduction operations, keeping 100% of PE compute cores free for matrix multiplication. — Wes Bland

> [08:55](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=535s) 100% efficiency score = simultaneous compute + communication executes in the same duration as standalone compute. — Wes Bland

> [08:55](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=535s) MTIA 300 maintains ~98% efficiency owing to dedicated Message Engines and NMC hardware. — Wes Bland

> [09:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=592s) Direct PE packet injection achieves <6 µs single-message latency. — Wes Bland

> [11:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=712s) For single-rack inference workloads, the 2 scale-out NICs can be repurposed as additional scale-up channels, increasing intra-rack bandwidth and eliminating wasted networking hardware. — Wes Bland

> [14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) Targeting approximately two silicon generations entering production per year. — Rajeev Nair

## Mentioned Rajeev Nair, Wes Bland, Meta, MTIA 300, MTIA 100, MTIA 200, MTIA 450, MTIA 500, NVIDIA H100, Broadcom, PyTorch, DLRM, RDMA, Ethernet, @Scale

[Source](https://www.youtube.com/watch?v=RrOhr0XHLQ4)

中文简报

Meta 部署 MTIA 300 芯片实现 AI 通信解耦 通过将通信任务卸载至专用硬件消息引擎，Meta 的 MTIA 300 芯片实现了 98% 的重叠效率，消除了困扰标准 GPU 的核心资源争用问题。 Meta 工程师 Rajeev Nair 与 Wes Bland 概述了 Meta 的 AI 加速器协同设计理念，展示了如何通过整合芯片、网络、软件和模型来优化分布式 AI 基础设施。演讲介绍了 MTIA 300 加速器，详细阐述了其专用硬件消息引擎与近存计算（NMC）单元如何卸载通信和归约负载，从而维持近乎完美的计算效率。此外，演讲还展示了网络可替代性如何使相同的 Broadcom 以太网网卡在 scale-up 与 scale-out 拓扑之间动态适配，以满足推荐系统和生成式 AI 不断演进的需求。 Meta 强调在芯片、网络架构、通信库、算子和机器学习模型层面进行全栈协同设计，以应对从传统推荐系统（R&R）向生成式 AI 及智能体（agentic）工作负载的演进。 (0:04) 传统的“计算-通信”串行模式会导致约 50% 的硬件处于闲置状态；Meta 通过重组执行流水线，使计算与跨节点通信能够完全重叠，从而最大化硬件饱和度。 (1:40) Meta 于 2024 年 3 月推出 MTIA 300 加速器，配备自主消息引擎（Message Engines）与近内存计算（NMC）单元，用于卸载通信与归约操作，使处理核心完全专注于矩阵乘法计算。 (4:26) 在计算与通信同步重叠的基准测试中，MTIA 300 保持了约 98% 的效率，显著优于因计算核心争用导致效率下降至 70%–80% 的 NVIDIA H100 GPU。 (8:55) MTIA 300 采用标准博通以太网网卡实现“网络通用性”（network fungibility），不仅能根据负载灵活调配 Scale-Up 与 Scale-Out 端口，还支持低于 6 微秒的 PE 直接注入低延迟通信。 (11:52)

Reading summary

Meta is co-designing its custom silicon, networking, and software stack in a unified loop to resolve distributed AI communication bottlenecks. In this presentation, Rajeev Nair and Wes Bland outline how the MTIA 300 chip decouples networking from the host CPU and compute cores to achieve near-total overlap of computation and data transfer. Hardware-Software Co-Design for Shifting AI Workloads (0:04) According to Rajeev Nair, Meta's infrastructure shifted from ranking and recommendation models prior to 2022 to large-scale Generative AI training in 2022, and toward agentic workloads by 2025. Because these evolving workloads create disparate message sizes and traffic patterns, Meta abandoned the traditional practice of designing hardware, software, and networking in isolated stages. Instead, the company co-designs its physical silicon, network fabrics, communication libraries, and PyTorch kernels in a continuous feedback loop. This continuous integration directly targets distributed serving bottlenecks. As models exceed single-machine memory limits, sequential execution—computing, communicating, and then computing again—leaves roughly half the hardware idle at any moment. Meta restructures execution pipelines to overlap compute and communication, ensuring that network interfaces and compute cores stay continuously saturated. MTIA 300 Architecture and Autonomous Message Engines (4:26) In typical accelerator deployments, the host CPU acts as an intermediary, dispatching compute kernels to the accelerator and network instructions to network interface cards (NICs), which introduces latency in the critical path. Wes Bland explains that Meta addressed this in its MTIA 300 accelerator, introduced in March 2024 for Deep Learning Recommendation Models (DLRM) and Generative AI, by embedding dedicated Message Engines (MEs) and Near-Memory Compute (NMC) units directly on the chip alongside the central Processing Element (PE) grid. MTIA 300 employs a CUDA-like streaming model where the host CPU enqueues instructions into High Bandwidth Memory (HBM) ahead of time and delegates execution entirely to the hardware. The autonomous Message Engines manage queues, completions, and dependencies without host intervention. Concurrently, the NMC units execute inline mathematical reductions, such as AllReduce accumulations, as data arrives from peer chips, preserving 100% of the PE grid's capacity for matrix multiplication. Benchmark Efficiency and Dual-Path Latency Optimizations (8:55) Bland presents benchmark tests assessing simultaneous matrix multiplication compute and collective communication, where a 100% score indicates that overlapping operations complete in the same duration as standalone compute. Standard NVIDIA H100 GPUs degraded to 70%–80% efficiency due to compute core contention when driving network tasks and inline arithmetic. In contrast, MTIA 300 sustained approximately 98% efficiency because its compute grid remained insulated from communication overhead. For large payloads across 16 nodes, MTIA 300 reached about 94% of its theoretical roofline limit, achieving 1.0 TB/s aggregate bandwidth. To handle latency-sensitive AI inference characterized by small message transfers, Meta also implemented an alternative path where PE cores bypass the Message Engines to construct and push RDMA packets directly into NIC queues. While this direct-injection method consumes PE compute cycles, it reduces single-message latency below 6 microseconds while leaving completion tracking to the Message Engines. Network Fungibility and Rapid Silicon Iteration (11:52) To improve data center capital and operational efficiency, MTIA 300 implements network fungibility using standard Broadcom RDMA Ethernet NICs for both intra-rack (scale-up) and inter-rack (scale-out) connectivity. Meta's default configuration allocates 10 NICs to scale-up traffic and 2 NICs to scale-out traffic, but single-rack inference workloads can repurpose the 2 scale-out NICs into additional scale-up bandwidth, eliminating idle hardware. Nair notes that while earlier iterations (MTIA 100 and 200) were limited to single-node recommendation inference, MTIA 300 provides a unified platform for training and inference. Meta has compressed its hardware development timeline to target a production release cadence of approximately two silicon generations per year, with MTIA 450 and MTIA 500 under active development as PyTorch-native accelerators. Meta positions MTIA 300's hardware offloading and Ethernet fungibility as an effective blueprint for high-efficiency distributed AI, maintaining that standard Ethernet is sufficient for modern scale. However, the aggressive target of deploying two silicon generations annually places heavy ongoing demands on compiler stability and PyTorch software integration to continuously deliver hardware gains to production models.

精炼阅读摘要

Meta 基础设施团队提出，随着人工智能工作负载从传统推荐系统转向生成式 AI 及智能体架构，传统架构中的通信开销与主机 CPU 依赖已成为核心瓶颈；为此 Meta 推进了贯穿底层芯片、网络拓扑、通信库至模型调度的全栈协同设计，并通过自研加速器 MTIA 300 实现了计算与通信的高效重叠。 负载演变与计算通信重叠需求 (0:04) Meta 阐述了其基础设施研发的核心理念：全栈协同设计（Co-Design）。该方法打破了硬件先行、软件叠加、网络后置的传统割裂开发模式，将芯片制造、网络架构、通信库、算子内核与机器学习模型深度绑定在单一连续反馈闭环中。根据 Rajeev Nair 的梳理，Meta 基础设施承载的流量形态历经数次重大变迁：2022 年前以排序与推荐系统（R&R）为主导；2022 年经历生成式 AI 训练集群的大规模扩展；到 2025 年则演进出新型智能体（Agentic）工作负载。每一次范式转换都会颠覆消息体积、集合通信模式与网络延迟敏感度，因此必须将模型层面的数据流动特征直接映射到物理硬件传输设计中。 在分布式 AI 计算场景下，模型参数、激活值与 KV 缓存规模均超越单机内存极限，必须将任务拆分为加速器上的本地计算与跨节点的数据通信。传统串行模式采用“计算—通信—计算”的时序设计，导致算力核心在通信期间处于空闲状态，硬件闲置率接近 50%。Meta 协同设计的核心方向在于重组模型执行流水线，使加速器在处理当前数据切片的同时，前序切片已通过网络并行分发，借由计算与通信的完全重叠消除算力等待，从而在相同时间内最大化算力矩阵与网络链路的饱和度。 MTIA 300 消息引擎与近存计算 (4:26) 传统加速器架构通常依赖主机 CPU 承担通信调度中枢职责，由 CPU 分别向加速器派发计算任务、向网卡（NIC）下达网络指令，并指挥网卡从加速器显存中拉取数据。尽管网卡直连显存可绕开 PCIe 传输带宽瓶颈，但主机 CPU 介入每一条网络指令仍会产生显著的系统开销与延迟，难以适应算术计算与网络传输深度混叠的算子融合趋势。为此，Meta 于 2024 年 3 月推出了首款兼顾深度学习推荐模型（DLRM）与生成式 AI 任务的自研芯片 MTIA 300。 在物理硬件层面，MTIA 300 采用中央处理单元（PE）网格架构，芯片边缘集成高带宽内存（HBM）与两组共 12 颗博通 RDMA 网卡（聚合带宽最高 1.2 TB/s，常规能效配置为 1.0 TB/s）。其关键架构创新在于引入了专用的硬件消息引擎（Message Engine, ME）及内嵌的近内存计算（Near-Memory Compute, NMC）单元。消息引擎能够自主完成队列消息派发、状态跟踪与任务依赖管理，完全不占用主机 CPU 与核心 PE 网格周期；NMC 单元则专用于在数据到达时直接在片上完成诸如 AllReduce 累加等行内规约运算，彻底解放主算力网格用于矩阵乘法（GEMM）。 重叠效率实测对比与低延迟直接注入 (8:55) MTIA 300 引入了类似于 CUDA 流的任务编排机制，主机 CPU 仅需将作业批次写入加速器 HBM 队列即可立刻释放以调度后续任务，由硬件消息引擎自主拉取并推进执行，仅在整批任务完成时回传统知。在矩阵计算与集合通信并发的基准测试中（100% 效率代表并发执行耗时与单独计算耗时完全一致），NVIDIA H100 GPU 由于算力核心被网络管理与行内计算占用，效率衰退至 70% 至 80%；而 MTIA 300 凭借专用消息引擎与近存计算卸载，维持了约 98% 的重叠执行效率。 在传输性能层面，MTIA 300 处理大消息时能达到约 94% 的理论峰值性能，在包含 16 个节点的集群实测中实现了 1.0 TB/s 的有效带宽。针对单字生成等对延迟极度敏感的小消息推理场景，Meta 还开辟了一条绕过消息引擎的直接注入路径：允许 PE 计算核心直接格式化并向网卡注入 RDMA 报文，虽然短暂占用少量算力周期，但将单消息延迟压缩至 6 微秒以内，同时配合消息引擎维护计数器状态，为推理阶段极致的算子网络深度融合提供了底层支撑。 标准以太网可替代性与高速芯片迭代 (11:52) 在数据中心拓扑层面，Meta 主张标准以太网完全能够满足现代 AI 扩展需求，并在 MTIA 300 中提出“网络可替代性”（Network Fungibility）概念。该芯片对机架内加速器互连的纵向扩展（Scale-Up）域与跨机架的横向扩展（Scale-Out）域统一采用同构的博通以太网 RDMA 网卡。在 MTIA 300 的标准配置中，10 个网卡分配给机架内 Scale-Up（适配 DLRM 的高带宽需求），2 个网卡用于跨机架 Scale-Out。在无跨机架需求的单机架推理场景中，这 2 个 Scale-Out 网卡可动态重构为额外的 Scale-Up 链路，以此消除网络基础设施的资本沉没与闲置损耗。 在芯片迭代路线上，Meta 显著缩短了硬件研发周期。相较于早期专注于单节点推荐推理的 MTIA 100 与 200，MTIA 300 已升级为兼顾训练与推理的统一架构，而后续世代 MTIA 450 与 MTIA 500 亦处于研发进程中。Rajeev Nair 指出，Meta 的规划目标是在生产环境中维持每年投产约两代自研芯片的高频节奏，并在软件栈全面对齐原生 PyTorch 框架，以确保算法研究人员能够无缝调用底层芯片与通信硬件的加速能力。 Meta 通过 MTIA 300 确立了以硬件消息引擎卸载、近内存规约与标准以太网可替代性为核心的协同设计范式，证实了摆脱主机 CPU 约束对维持高重叠计算效率的关键价值。然而，该路线深度绑定了 Meta 自身独特的 DLRM 与生成式 AI 业务结构；随着智能体架构对小消息非规则通信需求的进一步提高，维持每年投产两代芯片的超高工程迭代强度能否在更广泛的大规模集群中保持持续的能效与成本优势，仍有待未来生产环境的大规模验证。

Detailed notes — zh

## Background 本内容由 @Scale 官方频道于 2026 年 9 月 3 日发布在 YouTube 平台。视频简介指出，Meta 的 Rajeev Nair 和 Wes Bland 深入探讨了如何通过硬件、软件与模型的深度协同设计（Co-Design）应对生成式 AI 带来的庞大算力与数据传输需求，并重点解析了自研芯片 MTIA 300 依托专用消息引擎实现通信卸载以及网卡资源可变通（Network Fungibility）的通信架构革新。 ## TL;DR 演讲系统阐述了 Meta 面对从推荐系统、生成式 AI 到智能体（Agentic）工作负载演进时，如何通过软硬件协同破除“计算与通信无法重叠”的传统瓶颈。Meta 自研的 MTIA 300 芯片通过片上集成专用消息引擎（Message Engines）与近内存计算（NMC），实现了高达约 98% 的极致算网重叠效率，同时针对小消息极低延迟场景支持 PE 核心直接注入网络的灵活权衡。此外，架构利用以太网 RDMA 网卡实现了单机架内 Scale-Up 与跨机架 Scale-Out 之间的网络可变通性（Network Fungibility），并在 PyTorch 原生生态和每年两代芯片的超高速迭代节奏下，为 Meta 全球数十亿用户的 AI 应用提供底层支撑。 ## Atmosphere & style 整场演讲洋溢着硬核务实、节奏紧凑的高性能计算工程技术质感。两位讲者配合默契，Rajeev 从宏观算力需求跃迁与协同设计哲学切入，Wes 则无缝衔接深入到芯片物理布局、寄存器级直推报文与微秒级延迟权衡的底层微架构细节，随后再次收敛至整体芯片路线图。现场没有晦涩抽象的概念包装，而是全方位展示直击痛点的生产级实测数据与架构取舍，向听众传递出顶级大厂在 AI 基础设施前沿快速迭代、敢于重塑底层物理通信范式的极客自信与技术张力。 ## Bulletin - [04:29](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=269s) Meta 推出自研训推一体 AI 加速器 MTIA 300，片上直接集成消息引擎（ME）与近内存计算单元（NMC），彻底将通信控制与算术规约从主计算核心中解耦卸载。 - [07:22](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=442s) 在计算与通信并发场景下，Nvidia H100 的有效重叠效率受资源抢占影响降至 70%~80%，而 MTIA 300 凭借硬件卸载机制实现了约 98% 的极高算网重叠效率。 - [12:46](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=766s) 通过引入“网络可变通性（Network Fungibility）”，MTIA 300 采用相同的商用以太网卡构建 Scale-Up 与 Scale-Out，单机架纯推理模式下可将预留的跨机架网卡动态复用为机架内通道。 - [14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) Meta 维持每年向数据中心导入约两代自研硅片的交付节奏，后续 MTIA 450 与 MTIA 500 已步入推进正轨，全系深度集成原生 PyTorch 生态以降低迁移成本。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=4s) 协同设计（Co-Design）是应对 AI 算力与通信瓶颈的核心范式，必须将底层芯片、互联网络、通信库与上层模型置于统一定制闭环之中。 - [00:04](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=4s) Meta 内部工作负载经历三代演进：早期推荐与排序（R&R）到 2022 年 GenAI 训练扩展，再到 2025 年智能体工作负载全面爆发，流量特征与延迟要求被彻底重构。 - [01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) 传统加速器架构中“计算-通信-计算”的时间串行执行会导致灾难性资源闲置，提升模型服务吞吐的关键在于实现计算与通信的无损重叠。 - [01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) 传统系统拓扑中 Host CPU 依然处于每条通信消息的关键路径上，下发与调度开销成为大规模集群扩展的主要拖累。 - [04:29](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=269s) MTIA 300 核心物理布局集成 PE 计算阵列、两侧 HBM 以及 12 个 Broadcom RDMA 网卡，生产环境下启用 1.0 TB/s 的有效聚合网络带宽。 - [04:29](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=269s) 消息引擎（ME）可自主解析批量网络指令并追踪依赖，其内置的近内存计算单元（NMC）能在数据传输时直接执行 AllReduce 规约，不挤占通用计算核心。 - [07:22](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=442s) MTIA 采用流式处理范式解耦任务排队与底层硬件执行，在矩阵乘法与集合通信完全并发测试中跑出了高达约 98% 的重叠效率。 - [09:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=592s) 针对大消息的大规模传输，MTIA 300 在 16 节点集群规模下实现了高达 94% 的 Roofline 理论峰值性能。 - [09:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=592s) 为应对生成式 AI 推理对小包低延迟的严苛要求，Meta 探索打破消息引擎路径，转而由 PE 核心在寄存器级别直推 RDMA 报文并由 ME 追踪计数。 - [11:21](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=681s) 通过软硬件协同与计算通信算子融合（Fused Kernel），小消息通信延迟被进一步压缩至 6 微秒以下。 - [11:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=712s) 机架内部的 Scale-Up 域享有全集群最高带宽与最低延迟，而跨机架的 Scale-Out 域横向扩展则受限于相对较低的网络互联带宽。 - [12:46](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=766s) 以太网在 MTIA 互联中被证明既必要又充分，网络可变通性通过 10:2 的网卡配置既满足 DLRM 高通信需求，又能动态重构全量支持单机架推理。 - [14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) MTIA 300 标志着 Meta 首度实现训推一体的自研芯片架构，取代了以往仅针对单节点推理的 MTIA 100/200 系列。 - [15:55](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=955s) Meta 坚持 PyTorch 原生集成哲学，旨在彻底抹平异构硬件适配摩擦，为全球数十亿用户的实际生产服务提供底层算力支撑。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=4s) 联合设计：应对爆发式 AI 工作负载的核心范式 演讲由来自 Meta 的 Rajeev Nair 开场。他首先指出，本次分享的核心主旨可以用一个词概括——**“联合设计（Co-Design）”**。在 Meta 为其自研 AI 加速器打造从硅片底层一直到模型服务（Model Serving）的完整通信技术栈过程中，绝不能采用“先做硬件、再做软件、最后把网络生硬拼凑在末端”的割裂模式，而是必须将底层硬件、网络互联、通信库、计算内核（Kernels）以及上层模型全部置于同一个协同迭代的闭环之中进行统一设计。高效 AI 服务的未来完全维系于这种全栈协同的理念。 随后，Rajeev 阐述了驱动这一架构变革的根本原因：Meta 内部工作负载的演进速度超乎想象。在生成式 AI 爆发之前，Meta 底层基础设施的主要压力来源于排序与推荐系统（Ranking and Recommendations, R&R）。到了 2022 年，生成式 AI（GenAI）训练迎来了极其迅猛的规模扩展；而至 2025 年，智能体工作负载（Agentic Workloads）又作为一种全新的算力需求形态全面涌现。每一次业务架构的跃迁，都彻底改写了网络流量的形态特征、消息大小、集合通信模式（Collective Patterns）以及对延迟的敏感度。要跟上这一演进节奏，唯一的路径便是实施深度协同设计，将上层模型的真实运行需求直接下沉映射到底层数据搬运机制之中。 - **工作负载演进轨迹**：早期以推荐与排序（R&R）为主导 $\rightarrow$ 2022 年生成式 AI 训练规模急剧扩展 $\rightarrow$ 2025 年智能体（Agentic）工作负载全面爆发。 - **流量特征变动**：消息尺寸分布、集合通信交互拓扑、端到端延迟与吞吐敏感度发生根本性变化。 --- ### [01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) 计算与通信重叠：破解端到端性能瓶颈 Rajeev 剖析了 AI 运行时的底层本质挑战：现代大模型的权重、激活值和 Token 等海量数据，已经根本无法容纳在单一服务器节点之中。因此，分布式系统的构建聚焦于两大核心能力——其一是利用加速器完成核心计算，其二则是通过高效通信实现数据的聚合与再分发。算力是整个系统中最为稀缺昂贵的资源，理想状态下必须让计算引擎与互联网络全天候保持饱和运转。然而在传统的单节点工作模式中，往往呈现出“计算 $\rightarrow$ 通信 $\rightarrow$ 计算”这种严格的时间串行关系。这种清晰的阶段依赖关系会导致灾难性的资源闲置——在任意时刻，系统中几乎总有一半的硬件处于空转等待状态。 解决这一难题的核心法宝在于**重叠（Overlap）**。工程师必须对模型架构与执行流进行深度重构，使得当硬件正在对某一批次数据执行计算时，另一批次已就绪的数据能够同时在网络中传输，从而使计算芯片与网络织网时刻处于饱和负载状态，在相同单位时间内压榨出更多的模型服务吞吐。“算网重叠就是这场博弈的全部核心（That overlap is a whole game）。” 在剖析传统典型硬件系统拓扑时，Rajeev 指出其通常由 Host CPU 与加速器构成：Host CPU 负责为加速器生成计算内核（Compute Kernels），同时也负责为网络接口生成网络指令，驱使网卡（NIC）直接从加速器内存中拉取数据。虽然这种架构特意让网卡靠近加速器内存以避免穿透 PCIe 总线造成带宽拖累，但**Host CPU 依然处于每条消息传输的关键路径（Critical Path）之上**。加之当前混合计算与通信的融合内核（Fused Kernels）被越来越激进地采用，计算与通信的界限已高度模糊，硬件底层必须保持足够的前瞻性与灵活性以应对全新的通信范式。 ```mermaid flowchart TD accTitle: Traditional Host CPU Bottleneck in Accelerator Communication accDescr: Diagram contrasting traditional host-in-the-loop communication against ideal computation and network overlap. subgraph Host_Domain["主机端 (Host CPU)"] H1["生成计算内核 (Compute Kernels)"] H2["生成网络控制指令 (Network Instructions)"] end subgraph Accel_Domain["加速器端 (Accelerator)"] A1["计算核心 (Compute Grid)"] A2["片上高带宽内存 (HBM)"] end subgraph Net_Domain["互联网络 (Network)"] N1["网卡 (NIC) 直接访存"] end H1 -->|"下发工作队列"| A1 H2 -->|"关键路径控制与通知"| N1 A2 <-->|"DMA 读写数据"| N1 A1 <-->|"运算读写"| A2 ``` --- ### [04:29](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=269s) MTIA 300 架构解密：专为深度学习负载定制的消息引擎与片上拓扑 随后，在 Meta 负责 MTIA 通信库开发的资深高性能计算专家 Wes Bland 接过话题。针对 Rajeev 指出的传统架构痛点，Meta 于 2024 年 3 月正式推出了专为深度学习与推荐模型定制的自研芯片——**MTIA 300**（Meta Training and Inference Accelerator 300）。深度学习推荐模型（DLRM）对通信吞吐与协同有着极为严苛的要求，MTIA 300 正是 Meta 首款在提供强劲算力的同时，在硬件层面专门引入通信卸载（Communication Offloading）机制的定制芯片。 Wes 详细解构了 MTIA 300 的物理布局与硬件设计创新。芯片核心由大面积的处理单元阵列（Processing Elements, PE Grid）居中构成，两侧配备了海量的高带宽内存（HBM），而最引人注目的革新在于芯片上直接集成了专用的**消息引擎（Message Engines, MEs）**与两组共计 12 个 Broadcom RDMA 网卡。 ```mermaid flowchart LR accTitle: MTIA 300 Physical On-Chip Layout accDescr: Internal layout of MTIA 300 showing HBM, Message Engines with Near-Memory Compute, PE grid, and RDMA NIC banks. subgraph Memory_Left["高带宽内存"] HBM_L["HBM 存储库"] end subgraph Engines_Left["左侧消息引擎组"] ME_L["Message Engines + NMC"] end subgraph Compute_Core["核心计算网格"] PE["PE 处理单元阵列 (Processing Elements)"] end subgraph Engines_Right["右侧消息引擎组"] ME_R["Message Engines + NMC"] end subgraph Memory_Right["高带宽内存"] HBM_R["HBM 存储库"] end subgraph NIC_Subsystem["网络接口子系统"] NIC1["6x Broadcom RDMA NICs"] NIC2["6x Broadcom RDMA NICs"] end HBM_L <--> ME_L ME_L <--> PE PE <--> ME_R ME_R <--> HBM_R NIC1 --- ME_L NIC2 --- ME_R ``` - **消息引擎（Message Engines）**：能够在完全不打扰 Host CPU 和主计算核心的情况下，直接解析批量网络指令，自主驱动网络任务提交、监控完成事件并精准追踪数据依赖关系。 - **近内存计算单元（Near-Memory Compute, NMC）**：内置于消息引擎内部。它能够在数据参与网络通信时直接执行归约（Reduction）等算术操作。这一机制在处理像 `AllReduce` 这样的集合通信时展现出惊人的效率，能够一边接收来自对端节点的数据，一边由专用近存硬件完成实时累加，无需侵占通用 PE 核心。 - **集成网卡配置**：芯片集成了两组各 6 个 Broadcom RDMA 网卡，聚合带宽理论上限高达 1.2 TB/s；但在生产实践部署中，出于功耗与能效比的最优平衡考量，Meta 通常将其配置为启用 1.0 TB/s 的有效聚合带宽。 --- ### [07:22](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=442s) 极致通信卸载：流式范式与 98% 算网重叠效率对比 进入软件运行机制层面，Wes 深入解析了 MTIA 300 如何通过消息卸载革新软硬件协同模型。在传统加速器中，由 Host CPU 向网络直接注入消息不仅引入了额外的指令传输与事件往返信令延迟，且在集合通信进行数据规约时，通常必须强制抢占加速器自身的通用计算核，挤占如矩阵乘法等密集运算的宝贵资源。而在 MTIA 300 方案中，上层应用可以预先将完整的通信模式定义完毕并下发给加速器，随后 Host CPU 彻底退出控制回路，直至全部任务执行完成才接收回调通知。 在系统软件层面，MTIA 采用了类似 CUDA 的**流式处理范式（Streaming Paradigm）**： 1. 宿主机应用首先将一系列操作挂入一个或多个工作流（Streams）； 2. 批量指令被移交并加载到片上 HBM 中形成工作队列； 3. 一旦指令排到队列前端即刻由加速器硬件自主执行。 这种将“任务排队”与“底层运行”彻底解耦的机制，使得 Host CPU 上的流水线指令准备能够与加速器芯片内部的运算执行实现绝对的并行并发。 为了验证该架构的硬核优势，团队针对“矩阵乘法计算”与“集合通信操作”完全并发重叠运行的极端场景进行了严苛的基准测试。Wes 给出了一项极具震撼力的对比数据：在测试中，100% 的基准线代表两个任务在同时并发运行时所消耗的总时间，与它们各自单独运行的时间完全一致（即达到理想的无损重叠）。 - **行业主流标杆（Nvidia H100 GPU）**：由于其网络驱动或规约运算不可避免地要分食计算核心与内部总线资源，当两类任务完全重叠时，整体并发有效效率显著下跌至峰值的 **70% 到 80%** 之间。 - **Meta MTIA 300**：得益于硬件级独立的消息引擎（ME）与近内存计算（NMC）承担全部 Scale-up 流量调度与内联算术归约，完全没有发生通用计算资源的饥饿争抢，测试中保持了高达 **约 98%** 的并发重叠效率，同时加速了前向计算与互联通信。 --- ### [09:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=592s) 应对生成式 AI 冲击：从吞吐优化向低延迟直推的演进抉择 对于大规模数据的通信，MTIA 300 展现出了极其强大的带宽利用率。无论是在节点内的 Scale-up 互联还是跨节点的 Scale-out 织网中，面对大尺寸数据包传输时，在 16 节点的集群规模下，MTIA 300 能够跑出高达 **94% 的 Roofline 理论峰值性能**（基于 1 TB/s 的配置基准）。基准曲线清晰地显示出，随着单次消息数据量的持续放大，系统迅速逼近理论物理极限。 然而技术浪潮的演进并未止步于原定方案。Wes 坦言，在 MTIA 300 从最初立项设计到实际规模化投产部署的周期内，生成式 AI 的大爆发彻底颠覆了业务对底层系统的诉求。这款芯片虽然最初是为 DLRM 推荐模型的训练而定制，但在现实部署中必须急速扩展去适配多样的生成式 AI 推理任务——而推理场景对吞吐的要求往往退居次席，对**极低端到端延迟（Latency-Sensitive）**的要求则变得极其苛刻。 为了攻克小消息尺寸下的延迟瓶颈，Meta 探索打破常规的消息引擎路径，转而实现了一种**由 PE 核心直接注入网络流量（Direct Injection from PE Core）**的执行模式： - **微架构路径选择**：允许计算核心（PE）直接在寄存器级别构建 RDMA 流量报文，并瞬间压入指定网卡的硬件工作队列（Work Queue），消除任何中间代理调度开销。 - **混合状态追踪**：在这一路径下，消息引擎（ME）并不负责调度发起，而是退居幕后仅负责追踪底层消息的传输完成状态。由于 PE 计算核通常是批量注入大批微小消息，借由 ME 维护轻量级计数器，PE 核心只需极少代价读取计数即可判定一批消息是否全部交付。 - **工程权衡与代价（Trade-off）**：这种针对低延迟的激进设计并非毫无成本，它直接消耗并锁定了宝贵的计算核心来负责封包与网络写入。这种根据通信规模在“吞吐型专用引擎卸载”与“极速延迟型算力核心直发”之间的抉择，成为硬件在多变 AI 时代保持弹性的核心技术抓手。 ### [11:21](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=681s) 小消息延迟优化与算子融合权衡 在网络通信设计中，向网络注入数据的方式直接决定了延迟与吞吐的表现。针对某些特定负载，将计算操作与通信操作融合（fuse）到同一个 Kernel 中虽然会对整体带宽造成一定的妥协，但在端到端执行效率上却是一个极具价值的权衡方案。 通过这种深度优化的软硬件协同手段，Meta 成功将单条消息的通信延迟压低到了 6 微秒（6 µs）以下。该优化专门针对小消息（small messages）的极低延迟场景进行打磨，使得通信层在面对小包突发流量时能够提供接近极限的响应速度，极大减少了计算核心等待数据同步的空闲开销。 - 单条消息传输延迟已成功降低至 6 微秒以下。 - 采用计算与通信算子融合（Fused Kernel）策略，在特定负载下用微小的带宽妥协换取极低的端到端执行延迟。 --- ### [11:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=712s) 数据中心 Scale-Up 与 Scale-Out 网络拓扑 理解 AI 加速器网络的适应性，需要从典型的数据中心网络拓扑切入。在一个标准的机架（rack）中，插有多块刀片服务器（blades），每块刀片承载一个或多个加速器芯片，并通过机架内的交换机互联。机架内部这一组共享最高互联带宽的加速器芯片集合被称为 **Scale-Up 域**。由于这一组加速器规模最小、物理距离最短，芯片之间可以享受到全集群中最高密度的通信带宽。 跨机架之间的互连则依赖另一层独立的网络交换结构，称为 **Scale-Out 网络**。它允许作业规模按需横向扩展，特别是在超大规模的模型训练场景中，工程师只需不断追加机架即可构建庞大的计算拓扑。然而，跨机架的 Scale-Out 带宽通常显著低于机架内的 Scale-Up 带宽。因此在设计底层通信算法时，必须精心编排通信流水线（pipelining）并实现流量均衡，以最大化全网吞吐。 ```mermaid flowchart TD accTitle: 数据中心网络分层拓扑 accDescr: 展示机架内 Scale-Up 域与跨机架 Scale-Out 互联结构 subgraph ScaleOutNetwork ["Scale-Out 网络层"] SW_SO["Scale-Out 交换机"] end subgraph Rack1 ["机架 1 (Scale-Up 域)"] SW_SU1["Scale-Up 交换机"] B1["刀片 / 加速器群"] SW_SU1 --- B1 end subgraph Rack2 ["机架 2 (Scale-Up 域)"] SW_SU2["Scale-Up 交换机"] B2["刀片 / 加速器群"] SW_SU2 --- B2 end SW_SO === SW_SU1 SW_SO === SW_SU2 ``` - **Scale-Up 域**：机架内部加速器互连，共享最低延迟与最大专用带宽。 - **Scale-Out 域**：通过跨机架网络水平扩展，支持超大模型分布式训练作业。 - **架构挑战**：在跨机架带宽受限的约束下，通信算法必须做流水线编排与负载均衡来榨干吞吐。 --- ### [12:46](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=766s) 网络可变通性（Network Fungibility）与 MTIA 300 架构 在许多传统架构中，Scale-Up 和 Scale-Out 要么共用相同的物理互联网络，要么采用两套完全割裂的独立硬件，两者各有优劣。然而在现实业务中，绝大多数 AI 推理（Inference）工作负载规模并不会超出单个机架。如果在此时部署庞大且高成本的专用 Scale-Out 网络，会造成巨大的硬件资产闲置与能耗浪费。 Meta 在 MTIA 300 架构中提出了突破性的解决方案——**网络可变通性（Network Fungibility）**。MTIA 300 在 Scale-Up 和 Scale-Out 通信上采用完全相同的网络接口卡（NIC），从而让同一套物理网络资产能根据生产任务灵活重构。Meta 明确指出："以太网（Ethernet）对于此类互联既是必要的，也是充分的（necessary and sufficient）"。通过与博通（Broadcom）等生态伙伴紧密合作，深度优化商用 NIC 等成熟组件，Meta 致力于推动 AI 算力服务演变为一种普惠且低成本的行业标准。 在默认的硬件拓扑分配中，MTIA 300 系统为 Scale-Up 分配 10 张网卡，为 Scale-Out 分配 2 张网卡。这种 10:2 的配比完美适配了 DLRM（深度学习推荐模型）等生产负载——这类模型最极致的高带宽密集通信恰好集中在机架内部。更为关键的是，该网络具备弹性重构能力：在不需要大规模 Scale-Out 的纯推理场景下，原本预留给 Scale-Out 的 2 张网卡可以直接反哺给 Scale-Up 流量，作为机架内的额外通信通道，从而使单机架推理的可用网络带宽进一步大幅拉升。 | 配置模式 | Scale-Up NIC 数量 | Scale-Out NIC 数量 | 适用场景与技术优势 | |:--- |:--- |:--- |:--- | | **生产默认配置** | 10 张 NIC | 2 张 NIC | 高度适配 DLRM 推荐模型，在满足机架内海量通信的同时保留横向扩展能力 | | **推理/小规模重构模式** | 最大 12 张 NIC（复用 Scale-Out 通道） | 动态调减 | 将全部网卡资源聚焦于机架内部，彻底消除跨机架闲置，最大化小规模与推理吞吐 | --- ### [14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) 芯片迭代节奏与 Meta 自研算力路线图 随后由 Rajeev Nair 接过话筒，深入剖析 Meta 硬件路线图与软硬件协同设计机制。回顾发展历程：早期的 MTIA v1（MTIA 100）和 MTIA 200 专为单节点推理设计，主攻推荐系统相关负载；而全新的 MTIA 300 则实现了跨越式的代际跃升——它是 Meta 首款同时支撑模型训练与推理的统一芯片架构（"One chip that now does both training and inference"）。 传统芯片研发项目往往漫长而充满不确定性。Meta 正在通过自研流程与工具链的变革，极大压缩硅片迭代周期。凭借前代沉淀的架构经验，团队正在以每年向生产环境交付约两代自研芯片（"roughly two silicon generations into production every year"）的惊人节奏全速推进。目前，后续型号 MTIA 450 与 MTIA 500 的研发与流片推进工作均已步入正轨。这一惊人的产品化速度在 Meta 官方工程博客（当年三月份发布的技术长文）中有详细披露。 - **代际跨越**：MTIA 100/200 专注单节点推理；MTIA 300 跃升为集训推于一体的通用 AI 芯片。 - **研发节奏**：打破传统芯片长周期束缚，保持每年向数据中心导入约两代芯片的超高交付频次。 - **后续规划**：MTIA 450 与 MTIA 500 已在加速研发流程中。 --- ### [15:55](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=955s) 软硬件协同设计理念与 PyTorch 原生生态 总结全场演讲，Rajeev 强调，面对通信卸载、延迟优化、网络弹性以及芯片代际演进等纷繁复杂的技术细节，如果只能提炼一个核心准则，那就是**协同设计（Co-Design）**。高效 AI 服务的未来，必然建立在硬件、底层系统软件与上层 AI 模型本身的三位一体深度融合之上——"那正是芯片与软件交汇的地方，也是当下最激动人心的创新正在发生的地方。" Meta 并非闭门造车，而是选择与整个开源及产业生态伙伴并肩共建。MTIA 系统从诞生第一天起就确立了 **PyTorch 原生（PyTorch native）** 的软件设计哲学。团队的核心诉求极其纯粹：彻底抹平异构算力的适配摩擦，让广大的模型开发者不再把精力浪费在繁琐的底层硬件迁移与移植（porting）上，而是全身心投入到构建下一代模型与应用中。 最终，正是这套软硬协同演进的自研 AI 算力底座，在 Meta 家族全系应用与服务背后稳定运行，将前沿 AI 技术源源不断地输送给全球数十亿真实用户。Rajeev 在演讲尾声表达了对投身这一技术浪潮的自豪与兴奋，并热情邀请对自研通信体系与芯片架构感兴趣的业界同仁保持交流。 - **核心宗旨**：协同设计是破解 AI 算力供需瓶颈的终极答案，驱动芯片架构紧随前沿模型特征迭代。 - **生态承诺**：坚持 PyTorch 原生集成，最大程度降低 AI 工程师从研发到落地的工程摩擦。 - **服务规模**：支撑 Meta 家族（Family of Apps）核心产品，为数十亿全球终端用户提供实时 AI 动力。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=4s) 联合设计（Co-Design） — Rajeev Nair > [01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) That overlap is a whole game — Rajeev Nair > [01:40](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=100s) Host CPU 依然处于每条消息传输的关键路径（Critical Path）之上 — Rajeev Nair > [09:52](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=592s) 由 PE 核心直接注入网络流量（Direct Injection from PE Core） — Wes Bland > [12:46](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=766s) 以太网（Ethernet）对于此类互联既是必要的，也是充分的（necessary and sufficient） — Wes Bland > [12:46](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=766s) 网络可变通性（Network Fungibility） — Wes Bland > [14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) One chip that now does both training and inference — Rajeev Nair > [14:25](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=865s) roughly two silicon generations into production every year — Rajeev Nair > [15:55](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=955s) 那正是芯片与软件交汇的地方，也是当下最激动人心的创新正在发生的地方。 — Rajeev Nair > [15:55](https://www.youtube.com/watch?v=RrOhr0XHLQ4&t=955s) PyTorch 原生（PyTorch native） — Rajeev Nair ## Mentioned Meta, Rajeev Nair, Wes Bland, MTIA 300, MTIA 100, MTIA 200, MTIA 450, MTIA 500, Broadcom, Nvidia H100, PyTorch, CUDA, DLRM ## Video description This presentation details how Meta is addressing the massive compute and data demands of modern generative AI workloads by co-designing hardware, software, and models. The talk highlights the communications architecture of MTIA 300, Meta's custom-designed AI training and inference accelerator featuring dedicated message engines to efficiently offload communication workloads. By decoupling communication from the primary compute grid, the system minimizes resource contention and substantially drives up both compute and network efficiency. Additionally, the speakers explore how "network fungibility" allows the same network interface cards (NICs) to flexibly adapt to both scale-up and scale-out configurations depending on production needs. Ultimately, the presentation demonstrates how a tight co-design loop across infrastructure layers provides a scalable, future-ready roadmap for Meta’s rapidly evolving AI demand. Learn more about the @Scale conferences here: https://atscaleconference.com/ [Source](https://www.youtube.com/watch?v=RrOhr0XHLQ4)

Source quotes

[0:04] Rajeev Nair: 联合设计（Co-Design） [1:40] Rajeev Nair: That overlap is a whole game [1:40] Rajeev Nair: Host CPU 依然处于每条消息传输的关键路径（Critical Path）之上 [9:52] Wes Bland: 由 PE 核心直接注入网络流量（Direct Injection from PE Core） [12:46] Wes Bland: 以太网（Ethernet）对于此类互联既是必要的，也是充分的（necessary and sufficient） [12:46] Wes Bland: 网络可变通性（Network Fungibility） [14:25] Rajeev Nair: One chip that now does both training and inference [14:25] Rajeev Nair: roughly two silicon generations into production every year [15:55] Rajeev Nair: 那正是芯片与软件交汇的地方，也是当下最激动人心的创新正在发生的地方。 [15:55] Rajeev Nair: PyTorch 原生（PyTorch native）

Source quotes

[0:04] Rajeev Nair: Co-designing hardware, network, communication libraries, kernels, and models together in one feedback loop. [1:40] Rajeev Nair: Compute -> Communicate -> Compute leads to ~50% hardware idleness. [1:40] Rajeev Nair: Overlap compute and communication to keep both the accelerator grid and network links fully saturated. [3:05] Rajeev Nair: Host CPU involvement in every network instruction introduces latency and overhead. [4:26] Wes Bland: Near-Memory Compute (NMC) units within MEs performing inline reduction operations (e.g., AllReduce accumulations) off the main compute grid. [7:22] Wes Bland: Communication patterns defined entirely ahead of execution; MEs execute autonomously without host intervention. [7:22] Wes Bland: Near-Memory Compute units execute reduction operations, keeping 100% of PE compute cores free for matrix multiplication. [8:55] Wes Bland: 100% efficiency score = simultaneous compute + communication executes in the same duration as standalone compute. [8:55] Wes Bland: MTIA 300 maintains ~98% efficiency owing to dedicated Message Engines and NMC hardware. [9:52] Wes Bland: Direct PE packet injection achieves <6 µs single-message latency. [11:52] Wes Bland: For single-rack inference workloads, the 2 scale-out NICs can be repurposed as additional scale-up channels, increasing intra-rack bandwidth and eliminating wasted networking hardware. [14:25] Rajeev Nair: Targeting approximately two silicon generations entering production per year.

Original transcript

[0:04] Hi everyone, thanks for being here. [0:07] Today we are unpacking the journey of [0:11] building the full communication stack [0:14] silicon to model serving for metas a [0:17] accelerators. [0:19] The one word to leave with is core [0:22] design. [0:24] Not hardware followed by software. Not [0:27] networking bolter at the end. Hardware [0:31] network coms library kernels and models [0:35] designed all together in one loop. [0:39] The future of efficient AI serving is [0:43] core design. [0:45] Let me ground us in why that matters [0:49] with how fast our workloads have [0:52] evolved. [0:54] Before Gen AI, our world was mostly [0:58] ranking and recommendations RNR that [1:02] drove the infra. 2022, Gen AI training [1:07] scales up fast. [1:09] By 2025, [1:11] agentic workloads arrive as a whole new [1:14] source of demand. [1:17] Every one of these shifts change the [1:19] shape of traffic, message sizes, [1:22] collective patterns, latency, [1:24] sensitivity, etc. [1:27] It's that the only way to keep pace is [1:31] code design, pulling the model's needs [1:34] all the way back down into how we move [1:38] data. [1:40] Let's make that concrete. What actually [1:43] happens during an AI run? [1:47] At its core, an efficient serving is [1:50] challenged with data that won't fit on [1:53] one machine. [1:55] Weights, activations, tokens, and all. [1:59] So, we build systems that do two things [2:02] well. Compute, usually on an [2:06] accelerator, and communications that [2:10] aggregate and redistribute results. [2:14] Compute is our scarcest resource. We [2:17] want both compute and network to stay [2:20] busy all the time. In a node, [2:24] compute, communicate, [2:27] compute, strictly sequential over time. [2:31] Clean dependencies [2:33] and half your hardware idle at any [2:36] moment. What we need is overlap. [2:40] Restructure the model. So some data is [2:44] ready to send while you compute on other [2:47] data. Compute chip and the network are [2:51] saturated [2:52] more serving in the same unit of time. [2:56] That overlap is a whole game. Let's look [3:00] at the hardware meant to deliver it. [3:05] Here is a simplified view of most [3:07] systems. A host CPU and an accelerator. [3:12] The host generates the compute work for [3:15] the accelerator call those compute [3:17] kernels. [3:20] Then the host generates work for the [3:23] network interfaces called those network [3:25] instructions telling the nick to pull [3:29] data directly from accelerator's memory. [3:32] A deliberate choice. The nick sit closer [3:35] to the accelerator and reads memory [3:38] directly without dragging the data. [3:41] across something like PCIe adding [3:43] additional bottlenecks [3:46] but the host is still in critical path [3:48] for every message. [3:50] As we discussed before boundaries [3:53] between compute and communications are [3:56] blurry. Fuse kernels mix compute and [3:59] communications aggressively. [4:02] Our job is to keep the silicon future [4:05] proof to support these new communication [4:08] patterns. [4:10] Hi, I am in Meta source networking team [4:13] supporting the communications team for [4:17] MTIE. With that, let me hand to Wes and [4:21] see how we solve some of these important [4:23] challenges in MTA300 [4:26] meta training and influence accelerator. [4:29] >> Thanks Rajie. Hi, I'm Wes. I've been [4:32] working in communications for high [4:34] performance computing and AI for a [4:36] couple of decades and I'm on the team [4:38] that develops the communication library [4:39] for MTIA. [4:41] So we looked at the chip design that [4:43] Rejie just talked about and we saw some [4:45] opportunities to improve things that are [4:47] specific for our workloads and that led [4:49] us to introduce MTI300 this past March. [4:52] Deep learning recommendation models or [4:54] DRM require significant communication [4:57] capabilities and MTI300 is our first [5:00] chip specifically designed for these [5:02] types of workloads. We've done that by [5:04] adding communication offloading in [5:05] addition to the computational power. [5:08] In the picture you see here, you can see [5:10] what the physical chip looks like. And [5:12] now I'd like to show you some of the [5:13] specific differences between this and [5:15] similar accelerators. [5:18] So here you can see a cartoon of a [5:20] system that looks similar to what Reggie [5:22] previously showed, but it represents one [5:24] of the major hardware additions for [5:26] MTI300 that enables better overall [5:28] network performance. So we've added [5:31] message engines on the chip that are [5:33] able to take advantage of a batch of [5:34] instructions and quickly execute them [5:38] without involving the host or the [5:39] compute course. This message engine [5:42] drives submission to the network, [5:44] monitors completion, and tracks [5:45] dependencies. [5:47] We've recently published multiple papers [5:49] and blog posts describing the [5:50] architecture of of our MTI300 chip in [5:53] far more detail and I invite you to [5:55] refer to those if you want to get a [5:57] better understanding of all the hardware [5:58] components involved. [6:01] So this shows more detail about the chip [6:03] overall. You can see a large grid of [6:05] processing elements in the middle with [6:07] banks of message engines that I just [6:09] talked about and high bandwidth memory [6:10] on the sides. Internally, the message [6:14] engines contain a component called the [6:15] near-memory compute, which allows us to [6:17] perform reduction operations on data [6:19] being used for communication. So, this [6:22] is particularly effective for collective [6:23] operations like all reduce where we get [6:26] dedicated compute hardware that can [6:28] accumulate data as it arrives from [6:29] peers. We also have two banks of six [6:33] Broadcom RDMA nicks that provide up to [6:35] 1.2 two terabytes of aggregate bandwidth [6:38] though we often configure the system [6:39] with only one terabyte enabled for [6:41] efficiency. [6:43] We'll talk a bit more about those in a [6:45] minute. [6:47] So now that we understand what the new [6:48] hardware looks like, let's see how we're [6:50] able to leverage it. First, we'll talk [6:53] about how we accelerate our networking [6:54] and therefore improve our overall [6:56] application performance and total cost [6:58] of ownership by taking advantage of [7:00] communication offloading. [7:02] Second, we'll see how we introduced [7:04] optionality via fungeible networking [7:06] that enables us to adapt our custom [7:08] silicon to handle different kinds of [7:10] workloads. Finally, Rejieve will give us [7:13] a view of how the code designes that [7:14] we're able to achieve between our teams [7:16] at Meta enables us to bring the designer [7:18] production models all the way back into [7:20] the silicon. [7:22] So, first let's talk about message [7:24] offloading. We talked about how [7:26] traditional accelerator design tends to [7:28] drive traffic by having this host CPU [7:31] inject messages directly into the [7:33] network, but this adds latency for [7:35] moving around instructions and there's [7:37] signaling when messages are ready or to [7:39] be sent or they've arrived. There are [7:41] also times such as during communication [7:44] patterns like all reduce where we need [7:46] to perform arithmetic operations on the [7:47] data that's being communicated and this [7:50] usually consumes compute cores on the [7:52] accelerator. [7:54] For MTI300 the message engines on the [7:56] accelerator itself and that allows us to [7:59] define the entire communication pattern [8:02] before it begins and then we give it to [8:04] the accelerator to execute. At that [8:06] point the host CPU is no longer involved [8:08] until it's notified of completion. [8:11] Similarly, we can use the near-memory [8:13] compute to avoid consuming compute cores [8:15] that would otherwise be used for other [8:17] operations like matrix multiplication. [8:19] So, this works efficiently because we [8:21] operate in a streaming paradigm similar [8:23] to CUDA where applications in Q work [8:26] into one or more streams on the host and [8:28] then they're handed over to the [8:29] accelerator to be executed. At that [8:31] point, the instructions are copied into [8:33] HBM to be executed when they reach the [8:35] front of the work queue. [8:38] Separating out queuing the work from [8:40] running it allows us to overlap queuing [8:43] instructions on the host CPU with [8:45] simultaneous execution on the [8:46] accelerator. [8:49] All of this enables us to use the [8:51] hardware much more efficiently than [8:53] other hardware of a similar generation. [8:55] So we ran benchmarks that overlap matrix [8:58] multiplication compute with collective [9:00] communication operations at the same [9:01] time. In this graph, a score of 100% [9:04] would mean that the compute and the [9:05] communication operations would complete [9:08] in the same time when they're running [9:09] together as they would when they're [9:11] running individually. So this means [9:12] we're effectively offloading work and [9:15] therefore overlapping operations in the [9:17] way that Rejieve discussed earlier. What [9:19] we found is that while Nvidia H100 GPUs [9:23] would see the overall simultaneous [9:25] efficiency drop to around 70 to 80% of [9:28] peak, we were able to remain at about [9:30] 98% due to the fact that we have [9:33] separate hardware where we can execute [9:34] all this. What that means is that [9:38] because we don't use any of the compute [9:39] cores to drive either scale up traffic [9:42] or the inline arithmetic, we don't [9:44] starve resources for the simultaneous [9:46] compute and that speeds up both the [9:48] compute and the communication. [9:52] So all the work we've talked about [9:54] improves bandwidth for larger messages. [9:56] And you can see in the figure on top [9:58] that we're able to take advantage of [9:59] both the scale up and the scale out [10:01] networks to implement efficient [10:03] communication for large data. And we see [10:06] performance at about 94% of roof line [10:08] which would be at 1 tabyte per second [10:10] for 16 nodes. Keep in mind that higher [10:14] is better here particularly as you move [10:16] to larger data sizes on the right side [10:18] of the graph. However, there are times [10:20] where we're latency bound and we want to [10:22] optimize for small message performance. [10:25] As we've already said, in the time [10:26] between initially designing MTI300 and [10:29] its deployment, generative AI training [10:31] and inference workloads dramatically [10:34] changed the landscape of what we [10:35] required our systems to do. While this [10:38] chip was originally intended for DRM [10:40] model training, we needed to be able to [10:41] adapt it to changing workloads and [10:44] optimize for cases like inference, which [10:46] are often latency sensitive. [10:49] One of the ways we worked on optimizing [10:51] for latency is by injecting network [10:53] traffic directly from a PE core. This [10:56] allows the PE to construct an RDMA [10:58] traffic packet and insert it into the [11:01] work Q of the desired nick. We use the [11:04] MEES to track completion of messages to [11:07] simplify the requirements on the PE [11:09] course which typically would batch [11:10] together many of these operations and [11:12] only require a counter know when the [11:14] messages are done. [11:16] Obviously, this has the trade-off of [11:18] consuming a compute core while the [11:20] message is being constructed and [11:21] injected into the network. But for [11:23] certain types of workloads where we may [11:25] want to fuse compute operations together [11:27] with communication in a single kernel, [11:30] this is actually a good trade-off even [11:31] though it might not be the best for [11:33] overall bandwidth. So, by doing this [11:36] work, we're able to bring down the [11:37] latency of a single message to below 6 [11:39] micros, which you can see in the figure [11:41] on the bottom here. In this instance, [11:44] we're optimizing for low latency on [11:46] small messages, which means that lower [11:48] is better the further to the left of the [11:50] graph you get. [11:52] To talk about other ways we could adapt, [11:54] it's helpful to quickly look at the [11:56] design of a typical data center network [11:58] using a very simplified diagram. You [12:01] have a rack full of blades with one or [12:04] more accelerators in each of them that [12:06] are all interconnected via a switch. We [12:08] call the group of accelerators within [12:10] the same rack scale up domain. The [12:12] machines in this group will usually have [12:14] the most bandwidth available to them [12:16] because it's being shared by the [12:17] smallest set of accelerators. [12:20] Between racks, we also have connectivity [12:22] through a separate switch that's called [12:23] the scale out. This allows us to scale [12:25] the job size to be as large as we need, [12:28] typically for larger training runs by [12:30] just adding more racks and they're added [12:33] to the scale up switch. the bandwidth [12:35] between racks may be less which means [12:37] that when we're designing our [12:38] communication algorithms we want to do [12:40] our best to balance and pipeline [12:42] communication in a way that will [12:43] maximize overall throughput. [12:46] In some systems these two networks can [12:48] be the same interconnect. In others they [12:50] could be two different networks. There [12:53] are benefits and drawbacks to either [12:54] solution of course, but we know that an [12:57] AI inference workload typically does not [12:59] scale beyond a single rack today, which [13:02] means that a large scaleout network is [13:04] an inefficient deployment of resources [13:06] and we would want to do something else [13:08] for that type of workload. What we'd [13:10] like is to be able to adapt our network [13:12] to the workload by focusing all of our [13:14] resources on the scale out communication [13:17] instead. [13:19] For MTI300, we're able to get the best [13:22] of both worlds here because we use the [13:23] same mix for both scale up and scale out [13:26] communication. [13:27] This means that we can easily [13:29] reconfigure a system to adapt to our [13:30] needs. So, we have taken the position [13:33] that Ethernet is necessary and [13:35] sufficient for this. We have ecosystem [13:37] partners optimize the portions that [13:39] they're good at, such as the Broadcom [13:40] Neck, and they help us advance [13:42] technologies to eventually make AI [13:44] serving a standard that is inexpensive [13:46] for everyone. Our default configuration [13:49] is to use 10 nicks for scale up and two [13:51] for scale out communication. This [13:53] achieves a good balance for DLRM [13:55] workloads where we can optimize for [13:57] messages within the rack or models tend [13:59] to do most high bandwidth communication. [14:02] With this configuration, we're still [14:04] able to achieve large scale deployments [14:06] through the scaleout network when [14:08] needed. However, it also allows us to [14:10] reconfigure where it makes sense to [14:12] focus on scale up communication or even [14:14] use the scale out network as an [14:16] additional channel for scale up traffic. [14:19] This increases the overall available [14:21] bandwidth for smaller scale workloads [14:23] like inference. [14:25] So, next reje will talk a bit more about [14:27] our hardware roadmap and how the [14:28] codeesign work that we're doing allows [14:31] us to tie back feedback from the models [14:33] all the way down to the hardware. [14:35] >> Thanks. [14:37] Same timeline [14:38] now with our products on it. Initial MTI [14:42] versions [14:44] 100 and 200 were built for single node [14:48] inference for Iron R. MTA 300 is the [14:53] leap. One chip that now does both [14:56] training and inference. [14:59] Each generation banks on past learnings [15:03] so we ship the next faster. [15:07] Some general, some specialized. [15:10] Silicon programs are typically difficult [15:13] and lengthy. We are compressing the [15:16] timelines and improved our processes to [15:20] launch successive generations faster [15:23] into production. Moreover, we made it [15:27] fun to evolve. [15:29] Work on 450 and 500 are well underway. [15:35] And thus the cadence hopes we are [15:38] committed to launching roughly two [15:41] silicon generations into production [15:43] every year. Deep dive on all of this on [15:48] our meta engineering blog post from this [15:52] past March. [15:55] Let me bring it all together. [15:59] We have covered a lot road maps, [16:02] offloading techniques, latency [16:05] optimizations, [16:06] network fungibility, [16:08] etc. If you take one thing away, [16:13] the future of efficient AI serving is [16:16] core design. [16:19] It's the tight integration between [16:21] hardware, software, and the models [16:24] themselves. [16:26] That intersection where silicon meets [16:28] the software is exactly where the most [16:32] exciting innovation is happening right [16:35] now. [16:37] And we are not doing this alone. We [16:40] build alongside industry partners which [16:42] means all of you. We are deeply [16:46] committed to that collaborative spirit. [16:48] A fully integrated system [16:51] takes all our expertise combined. We [16:55] have built MTIA as a PyTorch native [16:58] system and our philosophy is simple. We [17:02] want to remove the friction for you the [17:06] model developers. This means less time [17:10] porting and more time doing what you do [17:14] best, building the future. Ultimately, [17:19] we are proud to build the infrastructure [17:22] that brings AI to billions of people [17:25] through our family of apps and services. [17:29] It's an exciting journey [17:31] and I'm glad to share it with you all. [17:37] Thank you for listening. If this works [17:40] is of interest to you, please reach out [17:43] to one of us. Have a great day.