Meta Redesigns Storage Architecture to Eliminate GPU Training Stalls

Meta重构存储架构以消除GPU训练停滞 Meta采用$O(1)$直接访问数据平面和全球级分层缓存取代传统的跨区域查找系统，以避免前沿AI训练中每小时数百万美元的GPU闲置损失。 为支持超大规模AI模型训练并避免代价高昂的GPU停顿，Meta对其存储架构进行了根本性重构，从传统的$O(N)$多跳查找转向支持客户端直连存储流式传输的$O(1)$扁平元数据架构。通过部署区域闪存存储，并实现基于智能数据预取和按需填充的全球级分层缓存，Meta将数据集摄取时间缩短了90%。展望未来，Meta正在着手解决诸如网卡（NIC）带宽翻倍等硬件瓶颈，持续优化检查点机制，并推动其存储技术栈适配大规模推理工作负载。 前沿AI模型的训练数据规模已增长了10个数量级，并扩展至多模态输入和AI生成内容。 (0:01) 在一段120秒的采样快照中，GPU约有20%的时间因等待存储而停滞，在大规模数据中心中每小时造成数千万美元的损失。 (0:01) Meta的存储架构从2024年基于HDD/TLC的块存储演进至2025年的QLC闪存，并计划在2026年前在Tectonic上优化对象存储API。 (0:01) 单台主机上的存储延迟突增会导致GPU停顿并级联波及整个同步集群，因此极低的长尾延迟（Pmax）至关重要。 (4:00) Meta用O(1)扁平统一架构查找取代传统的O(N)多跳数据库查找，从而消除了元数据瓶颈。 (6:00)

Meta engineering leaders Sidharth Bajaj and Venkatraghavan Srinivasan argue that modern frontier AI workloads require abandoning traditional distributed storage designs in favor of power-efficient, flash-backed architectures with $O(1)$ metadata lookups and planet-scale tiered caching to prevent multimillion-dollar GPU stalls. GPU Synchronization and the Cost of Tail Latency (0:01) Frontier AI model training data has expanded by 10 orders of magnitude while transitioning from text to multimodal formats like audio and long-form video, alongside an influx of synthetic data. Because model training synchronizes thousands of GPUs in parallel execution loops, storage performance dictates overall computational efficiency. In a 120-second production snapshot, Meta observed GPUs stalled roughly 20% of the time across five separate events while waiting on data delivery. Bajaj claims that in Manhattan-sized data centers running hundreds of thousands of GPUs, this 20% idle rate wastes tens of millions of dollars every hour in unutilized compute. To keep GPUs saturated, host CPU cores prefetch batch N+1 while the GPU processes batch N. However, because modern training loops enforce synchronization barriers across clusters, a latency spike on even a single host stalls its GPU, cascading idle time across every other synchronized node. AI infrastructure is therefore exceptionally sensitive to tail latency ($P_{\max}$), making deterministic read performance a strict operational requirement. Transitioning from Legacy Proxies to Direct Flash Streaming Meta's legacy storage stack for its Family of Apps was built for $/GB efficiency, relying on multi-hop $O(N)$ regional and global lookups, central API server data proxies, and hard disk drives. This model proved unusable for AI training, which requires millisecond-level access to high-IOPS flash storage. In response, Meta collapsed its metadata layers into an $O(1)$ unified schema and shifted byte streaming directly to a client SDK reading from regional flash nodes, bypassing API proxies. Meta deployed QLC flash globally in 2025 and optimized its Object API over Tectonic block storage in 2026. This redesign was driven directly by data center power constraints. Because data centers operate under fixed power caps, every kilowatt of electricity drawn by storage hardware directly subtracts from the power available to run GPUs. To absorb extreme IOPS spikes without over-provisioning hardware, Meta implemented an in-memory Read Plan Cache for metadata alongside a peer-to-peer Hot Object Cache carved out of unutilized GPU host RAM, which achieves an average 80% hit rate in production. Planet-Scale Tiered Caching Accelerates Research Iteration While massive multi-month pre-training runs justify manual dataset replication to local clusters, the majority of AI training workloads finish within hours to a single day. Manually copying multi-terabyte or petabyte datasets across global regions often took hours or days, stalling researchers. With frontier model release cadences accelerating from roughly 104 weeks in 2020–2022 to approximately 4 weeks by 2026, cross-region data movement created an unacceptable bottleneck to iteration velocity. To decouple data location from compute, Meta architected a global tiered caching system modeled on operating system page caching. Immutable datasets are written once to global blob storage and hydrated transparently across three tiers: local GPU host memory (L1), local host NVMe (L2), and regional flash-backed blob storage (L3). The client SDK pairs this hierarchy with an immediate data loader prefetch and an asynchronous background prefetch that looks ahead several minutes. Srinivasan reports that this on-demand hydration model reduced dataset ingestion latencies by roughly 90%, establishing it as Meta's default production workflow. Hardware Bottlenecks, Checkpointing, and the Inference Shift (19:22) Looking forward, Bajaj highlights three evolving infrastructure bottlenecks. First, modern hardware upgrades—such as moving from NVIDIA GB200 to GB300 systems—double network interface card (NIC) bandwidth while host CPU capacity remains largely flat. Storage software must become substantially more efficient to saturate these network interfaces without starving training pipelines of host CPU cycles. Second, as model parameter counts expand, checkpoint sizes grow exponentially, requiring high-throughput, non-blocking storage pipelines to save intermediate states without stalling GPUs. Finally, Meta is extending these storage architecture principles beyond training into large-scale AI inference, which introduces distinct low-latency and throughput demands across global infrastructure. Meta's experience demonstrates that AI storage cannot be evaluated solely on traditional capacity metrics, but must be co-designed around synchronization tail latencies, power budgets, and research velocity. An open engineering challenge remains optimizing host storage software efficiency so that CPU bottlenecks do not prevent systems from saturating rapidly doubling network interface bandwidths as hardware advances.

Meta 工程师 Sidharth Bajaj 与 Venkatraghavan Srinivasan 指出，前沿 AI 模型训练数据量激增了 10 个数量级，传统存储架构导致的高尾部延迟会让 GPU 面临约 20% 的停顿损耗；为此 Meta 重构了存储底座，通过扁平元数据、直连数据面与多级缓存，兼顾硬件利用率与研究迭代速度。 AI 工作负载驱动的存储经济学与停顿危机 (0:01) 在前沿大模型的训练中，数据规模膨胀了 10 个数量级，数据形态也从纯文本转向音频、长短视频混合，且合成数据占比快速攀升。这一演进对底层存储提出了严峻挑战。在数以万计 GPU 并行同步训练的场景下，GPU 高度依赖 CPU 预取批次数据；一旦单台宿主机发生存储延迟抖动，受制于同步屏障，整个集群的计算节点都会被迫等待。Meta 的实测抽样显示，GPU 约有 20% 的时间处于停顿等待数据的状态。在曼哈顿规模的数据中心内，这种闲置每小时会造成数千万美元的直接损失，同时还将前沿模型的迭代周期严重拖慢。 告别传统架构：扁平元数据与客户端直连 Meta 原有的对象与块存储体系针对社交应用（如 Instagram、Facebook）设计，依赖跨地域多跳数据库查询完成寻址（复杂度为 O(N)），且数据流经中心化 API 服务器代理。这种架构不仅延迟过高，而且以每字节成本（$/GB）与多区域灾备为核心目标，无法满足 AI 场景对峰值 IOPS、写一次读多次（WORM）一致性以及数据中心严苛电力配额（kW/GPU）的要求。为此，Meta 将底层的元数据检索重构为 O(1) 的扁平统一架构，并通过客户端 SDK 获取“读取计划”，绕过代理直接向区域闪存节点流式拉取数据。此外，系统引入了读取计划缓存，并利用宿主机未分配的内存构建热对象缓存（P2P 架构），在生产中实现了约 80% 的缓存命中率。 行星级分层缓存与按需水合机制 针对全球分布式集群中跨地域复制数据集耗时数小时至数天、严重阻碍日常短期训练（数小时到一天内完成）的问题，Meta 提出了类似操作系统页面缓存的行星级分层缓存架构。全局对象存储被视作不可变的统一事实来源，计算节点按需进行透明的“数据水合”（Hydration）。整个缓存层级分为 L1（本地 GPU 宿主机内存）、L2（本地 NVMe 盘）与 L3（区域级闪存）。结合数据加载预取与面向未来几分钟训练流的后台异步预取，该方案让数据加载接入延迟大幅降低了约 90%，研究人员无需再手动搬运海量数据集即可就地开展实验。 演进路线与未来硬件瓶颈挑战 (19:22) Meta 的存储底座演进经历了自 Tectonic 块存储升级文件 API、2025 年全球部署 QLC 闪存，到 2026 年基于 Tectonic 全面优化对象 API 路径的历程。展望未来，硬件架构的迭代（例如从 GB200 跨越至 GB300）使网络网卡（NIC）带宽翻倍，但宿主机 CPU 算力增长停滞，存储软件必须在极低 CPU 开销下跑满网络极限。与此同时，随着模型参数规模膨胀，高频大体量模型检查点（Checkpointing）的高效落盘，以及低延迟、高并发的大规模 AI 推理场景，正成为 Meta 存储团队亟待攻克的新瓶颈。 Meta 通过解耦元数据与直连数据面、构建行星级分层缓存，显著压低了数据接入延迟并缓解了 GPU 停顿；然而随着硬件网卡带宽翻倍而主机 CPU 算力受限，如何在不耗尽 CPU 资源的前提下饱和利用网络，并同时支撑更庞大的检查点写入与推理负载，仍是该架构持续演进中的核心考验。

## Background 该视频由 @Scale 频道于 2026 年 7 月 7 日发布在 YouTube 平台上，记录了来自 Meta 的 Sidharth Bajaj 与 Venkatraghavan Srinivasan 在 @Scale 大会上的技术分享。视频简介指出，随着模型能力指数级增长与数据集规模暴增，前沿模型的迭代周期已压缩至数周，演讲深入解析了 Meta 如何重构其存储架构，以攻克优化 GPU 利用率与最大化研发迭代速度这两大核心挑战。 ## TL;DR 面对 AI 数据规模十倍暴涨及多模态演进，Meta 发现传统存储架构的长尾延迟会导致数十万张 GPU 集群面临每小时数千万美元的算力空转损失。为此，团队彻底重构了底层对象存储 API，打破传统 $O(N)$ 的跨区域元数据长链路，实现了扁平单层的 $O(1)$ 极速寻址，并通过客户端 SDK 去代理化实现闪存直连。针对突发流量与研究员跨区搬迁数据的繁重摩擦，Meta 引入了 Read Plan Cache、利用 GPU 闲置内存的 Hot Object Cache，并借鉴操作系统虚拟内存理念构建了全球到芯片的多级分层缓存与透明按需灌库体系。这一系列架构革新将数据准备等待时间骤降 90%，在保证高 GPU 利用率的同时，极大释放了前沿模型的科研迭代速度。 ## Atmosphere & style 整场演讲呈现出顶尖工业界架构师务实、硬核且极具全局视野的技术气场。Sidharth 的开场宏大而警醒，直接用数千万美元的算力闲置与被极限压缩的模型发布周期点明工程痛点，迅速建立起高维度的紧迫感；随后 Venkat 则以抽丝剥茧的白板式剖析深入微观 I/O 流水线，在理性克制的工程推导中展现出推倒旧系统重构的果断魄力。两位讲者节奏紧凑、配合默契，将冷冰冰的分布式存储与狂飙突进的 AI 算力军备竞赛紧密缝合，让听众深刻感受到超大规模底层工程所特有的力量感与精妙感。 ## Bulletin - [01:20](https://www.youtube.com/watch?v=xc_BxLsijWA&t=80s) Meta 揭示 AI 算力集群中由存储引发的严重效率瓶颈：单卡 120 秒监控中发生 5 次计算停顿导致约 20% 时间闲置；在数十万张 GPU 的超大集群规模下，算力空转折合每小时损失高达数千万美元。 - [09:43](https://www.youtube.com/watch?v=xc_BxLsijWA&t=583s) 针对 AI 训练中高并发与强同步（All-Reduce / Barrier）引发的长尾延迟级联崩溃，Meta 彻底重构对象存储 API，通过单层扁平元数据将寻址复杂度从 O(N) 压缩至 O(1)，并剥除中转代理实现客户端 SDK 直连闪存节点。 - [10:57](https://www.youtube.com/watch?v=xc_BxLsijWA&t=657s) 为平抑极端突发 I/O 尖峰，Meta 引入专为高 QPS 设计的 Read Plan Cache，并利用 GPU 宿主机闲置内存构建点对点 Hot Object Cache，在生产环境中达到约 80% 的缓存命中率。 - [14:46](https://www.youtube.com/watch?v=xc_BxLsijWA&t=886s) 借鉴操作系统分页与分层缓存思想，Meta 推出跨越全球 Blob、区域 Flash 到 GPU 内存的多级分层存储与透明按需灌库机制，辅以两级预取完全掩盖跨区延迟，将研究员数据准备时延骤降 90%。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=xc_BxLsijWA&t=1s) AI 数据领域正经历三大变革：训练数据规模在极短时间内跃升 10 倍、模态从纯文本跨越至音视频高吞吐介质、数据来源由人类生成向 AI 合成数据快速演变。 - [01:20](https://www.youtube.com/watch?v=xc_BxLsijWA&t=80s) 前沿模型的重大研发迭代周期已从 2020-2022 年的 104 周被极端压缩至 2026 年的 4 周，研发迭代速度（Research Velocity）成为与硬件成本同等重要的战略核心。 - [02:49](https://www.youtube.com/watch?v=xc_BxLsijWA&t=169s) Meta 存储体系底座为横向扩展的 Tectonic 分布式块存储，历经 2024 年文件 API 优化、2025 年超大规模 QLC 部署后，2026 年重点全面重构对象存储 API（Blob 存储）。 - [04:03](https://www.youtube.com/watch?v=xc_BxLsijWA&t=243s) AI 训练具有大规模并发与全局同步屏障两大特征，任一单节点因存储链路毛刺引发的预取延迟，都会通过集合通信同步导致整个集群协同空转，因此尾延迟指标必须严控至 P_max（100分位）。 - [06:39](https://www.youtube.com/watch?v=xc_BxLsijWA&t=399s) 传统面向消费级应用的 Blob 架构依赖代理网关和多次跨地域元数据寻址，形成高耗时的 O(N) 调用链，与闪存时代 AI 训练毫秒级高频访问形成不可调和的矛盾。 - [08:05](https://www.youtube.com/watch?v=xc_BxLsijWA&t=485s) AI 存储的权衡空间发生彻底重塑：不再盲目追求抵御单区离线的跨地域强同步复制，且随着数据中心电力逼近物理极限，存储系统每多消耗一千瓦电力便意味着减少一千瓦宝贵的 GPU 算力。 - [08:05](https://www.youtube.com/watch?v=xc_BxLsijWA&t=485s) AI 训练数据集具备典型的 WORM（单次写入、多次读取）静态特征，允许采用更宽松的一致性语义以换取极致的吞吐性能。 - [09:43](https://www.youtube.com/watch?v=xc_BxLsijWA&t=583s) 新架构将分散的元数据系统整合为扁平统一单层模型（Flat Unified Schema），将 Blob 寻址直接降为单次 O(1) 查找，带来断层式性能阶跃。 - [10:20](https://www.youtube.com/watch?v=xc_BxLsijWA&t=620s) 数据面彻底剥离中间 API 代理网关，将路由与流控逻辑下沉至客户端 SDK，由客户端获取 Read Plan 后直接与存储节点建联拉取数据流，极大消除网络抖动与 CPU 内存拷贝开销。 - [10:57](https://www.youtube.com/watch?v=xc_BxLsijWA&t=657s) 通过 Read Plan Cache（分布式内存元数据缓存）与 Hot Object Cache（GPU 宿主机 P2P 内存缓存）双层防护，系统在抵御流量尖峰的同时实现了 80% 的生产缓存命中率。 - [12:04](https://www.youtube.com/watch?v=xc_BxLsijWA&t=724s) 全球分散算力导致研究员此前必须手动跨区域拷贝庞大数据集，导致数小时乃至数天的等待阻塞，成为严重拖慢研发迭代的核心摩擦点。 - [13:49](https://www.youtube.com/watch?v=xc_BxLsijWA&t=829s) 业务负载呈现两极分化：持续数周的大规模预训练适合提前搬迁数据以追求极致本地性能，而占绝大多数的中小型长尾训练更迫切需要按需访问的研发敏捷性。 - [14:46](https://www.youtube.com/watch?v=xc_BxLsijWA&t=886s) Meta 将全球分布式存储抽象为“行星级计算机的硬盘”，构建了从全球 Blob、区域 Flash（L3）到宿主机 NVMe（L2）与内存（L1）的多级分层缓存体系。 - [14:46](https://www.youtube.com/watch?v=xc_BxLsijWA&t=886s) 通过紧邻批次流水线预取与向前预测数分钟的后台深度预取（Background Prefetch）双级协同机制，系统在透明跨区拉取数据时完全掩盖了跨区网络延迟，算力利用率不降反升。 - [17:29](https://www.youtube.com/watch?v=xc_BxLsijWA&t=1049s) 按需透明灌库落地后，端到端数据准备等待时延骤降 90%，并迅速替代传统手动复制成为 Meta 内部模型训练的标准主流范式。 - [17:29](https://www.youtube.com/watch?v=xc_BxLsijWA&t=1049s) 面向未来，硬件网卡（NIC）带宽成倍激增而 CPU 资源受限，Meta 存储团队下一步将攻坚低 CPU 开销下跑满网络极限、检查点（Checkpointing）写优化以及大规模在线推理存储。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=xc_BxLsijWA&t=1s) AI 数据变革的三大趋势与存储的战略地位 在本次 @Scale 大会的“AI 存储蓝图”（AI Storage Blueprint）演讲开场中，来自 Meta 的 Sidharth Bajaj 携手同事 Venkatraghavan Srinivasan，共同探讨了如何为快速演进的 AI 工作负载彻底重构存储底层架构。Sidharth 首先带领大家回顾了近年来驱动 AI 技术爆发的数据层深刻演变。他指出，数据领域正在经历三个维度的剧烈变革：其一是数据规模的指数级膨胀，用于训练前沿大模型的数据量在极短时间内实现了整整一个数量级（10倍）的惊人增长。 其二是数据模态的全面拓展。在前沿模型探索的起步阶段，训练集几乎完全以纯文本数据为主；而如今的训练数据集已经演变成包罗万象的多模态世界，涵盖音频、长视频以及短视频，使得底层数据管道的吞吐压力成倍飙升。其三则是数据生成来源的历史性交替：过去整个互联网的数据底座完全由人类生成内容（Human-generated content）构成，虽然目前人类内容依然占据核心，但 AI 生成的内容（AI-generated content / 合成数据）正在以极快的速度迎头赶上，两者之间的界限正迅速弥合。这三大趋势的交汇，直接将底层存储推到了决定前沿模型研发成败的最前线。 - **数据规模**：前沿模型训练集规模呈现 10 倍（一个数量级）的跨越式激增。 - **数据模态**：从早期的单一纯文本，跨越至音频、短视频、长视频等高吞吐多模态数据。 - **内容来源**：以人类生成内容为主导的格局正在被爆发式的 AI 生成内容迅速逼近追平。 --- ### [01:20](https://www.youtube.com/watch?v=xc_BxLsijWA&t=80s) GPU 利用率黑洞与研发迭代速度的较量 存储对于 AI 算力集群究竟有多重要？Sidharth 展示了一张记录某台 GPU 在 120 秒真实运行窗口内的性能监控快照。令人震惊的是，在仅仅两分钟的周期内就出现了多达 5 次停顿（Stalls），累计导致约 20% 的时间里 GPU 完全处于闲置摸鱼状态。经过底层深度剖析发现，这些昂贵的算力核心并不是在进行复杂推理计算，而仅仅是在空等存储系统把数据喂过来。 ```mermaid flowchart LR accTitle: GPU 闲置引发的成本与研发双重损失 accDescr: 展示存储延迟导致的 GPU 停顿如何向上传导为巨额资金浪费与模型交付周期的拖延。 subgraph Stalls["120秒性能快照分析"] A["120秒监控窗口"] --> B["5 次计算停顿"] B --> C["20% 时间闲置等数据"] end subgraph Impact["集群级链式后果"] C --> D["数十万 GPU 集群: 每小时损失数千万美元"] C --> E["研发迭代滞后: 前沿模型发布从 104 周被压缩至 4 周"] end ``` 单个 GPU 闲置 20% 听起来似乎尚可忍受，但 Sidharth 提醒现场观众注意规模效应的放大：“当你建造的是一个规模堪比整个曼哈顿、容纳了数十万张 GPU 的巨型数据中心时，这 20% 的算力停顿意味着每小时就有高达数千万美元白白蒸发（tens of millions of dollars lost every hour）。”除了肉眼可见的真金白银损失，更为致命的是时间与机会成本：由于 GPU 在苦等数据，研究人员无法快速拿到实验反馈。在 2020 至 2022 年间，前沿模型的重大升级周期大约是 104 周（两年）；而到了 2026 年，全新前沿模型的推出周期已经被极限压缩到了区区 4 周。在这种一日千里的研发节奏下，研发迭代速度（Research Velocity）的权重与基础设施的硬件成本同样生死攸关。 - **算力闲置率**：120 秒实测窗口内发生 5 次中断，GPU 整体停顿等待时间占比高达约 20%。 - **财务资金冲击**：在数十万张 GPU 规模的超级数据中心里，20% 的闲置折合每小时损失高达数千万美元。 - **科研迭代周期**：重大前沿模型的演进迭代周期从 2020-2022 年的 104 周急剧收窄至 2026 年的 4 周。 --- ### [02:49](https://www.youtube.com/watch?v=xc_BxLsijWA&t=169s) Meta 存储架构演化历程：从 Tectonic 到对象 API 重构 为了让大家理解 Meta 存储团队当前的战略突破，Sidharth 梳理了团队从 2024 年至今的技术演进路线图。在整个 Meta 存储体系的最底层，是名为 Tectonic 的分布式水平扩展块存储层（Block Layer），其硬件介质囊括了传统机械硬盘（HDD）、TLC 固态硬盘等多种存储介质。 回溯技术脚步，团队在 2024 年的大会上分享了如何针对 Llama 系列模型训练对文件 API（File API）路径进行深度优化；到了 2025 年，团队向业界公布了全球规模最大的企业级 QLC 固态硬盘生产环境部署实践。而本次 2026 年的分享，则聚焦于最新技术突破：如何在底层块存储之上彻底重构并优化对象 API（Object API / Blob 存储）路径，以全面承接 Meta 新一代前沿模型的训练需求。演讲的核心攻坚目标凝练为两点：**最大限度榨干 GPU 利用率以降低硬性成本**，以及**最大限度释放 AI 研究员的研发迭代速度**。随后，Sidharth 将演讲交接给深耕 Meta Blob 存储团队的软件工程师 Venkatraghavan Srinivasan。 - **底座基础（Tectonic）**：横向可扩展的分布式块存储层，融合 HDD、TLC 等异构硬件介质。 - **2024 年演进**：大幅改造文件 API（File API），全方位支撑 Llama 模型训练流水线。 - **2025 年演进**：实现全球顶级规模的 QLC SSD 生产级落地部署。 - **2026 年突破**：全面重构基于块存储之上的对象存储 API，支撑新一代前沿多模态大模型。 --- ### [04:03](https://www.youtube.com/watch?v=xc_BxLsijWA&t=243s) AI 训练流水线：预取重叠与尾延迟的连锁崩溃效应 Venkat 接过麦克风，直接将视角切入模型训练的微观数据流。从宏观上讲，每个 GPU 都可以被抽象为一个全速运转的数据处理核心：它从存储系统中持续拉取批次数据，运行矩阵计算与梯度反向传播，并周期性地将模型权重状态作为检查点（Checkpoint）回写到持久化存储。 这一看似常规的 I/O 流水线在 AI 超大规模场景下带来了前所未有的工程挑战。其一是**极致并发性**（Massive Parallelism），成百上千乃至数十万张 GPU 在完全相同的时刻执行读写；其二是**严格的同步性**（Synchronization），GPU 集群在执行数据加载、前向传播或检查点保存时往往存在全局同步屏障（Barrier / All-Reduce）。这种“大规模并发 + 步调高度协同”的致命组合，会在存储层引发极端的突发式（Bursty）吞吐洪峰，对系统 IOPS 提出了苛刻要求，且各训练阶段多变的 I/O 访问模式更是让传统存储雪上加霜。 ```mermaid flowchart TD accTitle: 数据预取重叠失效与多 GPU 跨节点同步级联等待 accDescr: 展示当 CPU 预取延迟超出 GPU 计算时长时，GPU 2 发生停顿，并因分布式集合通信导致正常的 GPU 1 共同停顿。 subgraph GPU1["GPU 1（预取正常）"] G1_Compute["计算 Batch N"] --> G1_Wait["等待集群同步..."] C1_Prefetch["CPU 预取 Batch N+1: 耗时 < 计算时长 (正常)"] end subgraph GPU2["GPU 2（遭遇存储长尾延迟）"] G2_Compute["计算 Batch N"] --> G2_Stall["GPU 挂起停顿 (等待数据)"] C2_Prefetch["CPU 预取 Batch N+1: 存储毛刺延时超标!"] --> G2_Stall G2_Stall --> G2_Done["迟滞完成"] end G2_Done -.-> G1_Wait G1_Wait --> Sync["全局步调同步完成 (算力大面积浪费)"] ``` 为了隐藏数据加载耗时、实现计算与 I/O 的完全重叠（Compute-IO Overlap），训练框架通常利用宿主机的 CPU 核心进行后台数据预取（Prefetching）——当 GPU 忙于计算当前批次数据时，CPU 已经提前在网络和存储端拉取下一个批次。只要预取下一个批次的耗时小于当前批次的 GPU 计算时长，GPU 就能始终处于饱和运转状态。然而，一旦某台节点（如 GPU 2）因存储链路波动或尾延迟导致预取超时，该节点上的 GPU 就会立刻陷入停顿。更为严重的是，由于分布式训练中 GPU 之间存在强烈的同步依赖，原本计算完毕且预取正常的 GPU 1 也不得不停下脚步等待 GPU 2 追赶进度。Venkat 强调了一个至关重要的结论：“对于 AI 工作负载而言，存储可预测的稳定延迟必须一路严防死守到最极端的尾延迟（$P_{\max}$ / 100分位）。” - **AI 训练 I/O 特征**：涵盖大规模并发（数十万张卡同频读写）与严格的跨机同步步调。 - **重叠机制（Overlap）**：依赖 CPU 并行预取机制，前提是数据拉取耗时严格小于 GPU 批次计算时长。 - **长尾连锁反应**：任一单节点上的存储高延迟毛刺，都会通过集合通信同步屏障迅速演变为全集群的算力空转。 - **核心优化指标**：传统服务关注的 $P_{50}$ 或 $P_{99}$ 远远不够，AI 场景必须消除直至 $P_{\max}$ 的极值毛刺。 --- ### [06:39](https://www.youtube.com/watch?v=xc_BxLsijWA&t=399s) 传统 Family of Apps 架构痛点：$O(N)$ 跨区域元数据开销 为了讲清楚这次重构的破局点，Venkat 剖析了此前支撑 Meta 家族应用（Family of Apps，即 Facebook、Instagram 等核心消费级产品）的长达数年的传统 Blob 对象存储架构。在传统架构下，一次客户端的 `GET` 读取请求通常需要经历曲折的生命周期：客户端首先请求 API 代理服务器，API 服务器为了将一个对象名称（Blob Name）解析为具体的存储物理地址，必须依次发起一连串的元数据寻址，其中既包含本区域内的元数据检索，还频繁穿插着跨地域（Global/Cross-region）的元数据服务调用。只有在拼凑出完整的物理存储地址后，API 服务器才会向底层的 Tectonic 块存储发起读取，并将字节流代理中转回客户端。 ```mermaid flowchart LR accTitle: 传统对象存储与 AI 新架构 GET 请求路径对比 accDescr: 对比传统架构中繁琐的多次跨域元数据查询和代理中转，与新架构下 O(1) 元数据寻址及直连数据流。 subgraph Legacy["传统架构: 代理中转与 O(N) 寻址"] Client1["Client"] --> API["API 代理网关"] API --> Meta1["区域元数据服务"] API --> Meta2["全球跨区元数据"] Meta2 --> Block1["Tectonic 块存储"] Block1 --> API --> Client1 end subgraph AIBlueprint["AI 存储新蓝图: SDK 直连与 O(1) 扁平寻址"] Client2["Client (内嵌 SDK)"] --> FlatMeta["扁平统一元数据 (O(1))"] Client2 --> Block2["Tectonic 块存储 (直连 Flash 数据面)"] end ``` 这种机制在消费级移动互联网时代运行良好，但在闪存时代的 AI 场景下却暴露出致命缺陷。其核心症结在于：对象名到物理地址的映射变成了一个复杂度为 $O(N)$ 的长调用链，且部分查询不可避免地要横跨跨洋或跨国数据中心。这一历史包袱的成因在于系统长年累月的“有机演化”，多层业务堆叠导致不同的存储子系统为了维护自身状态，各自引入了独立的元数据依赖。当 AI 训练作业需要以毫秒甚至微秒级的高频访问闪存介质时，动辄耗费数十上百毫秒的跨系统、跨区域元数据查找，在性能要求面前是“根本无法接受的致命伤（a total non-starter）”。 - **传统请求链路**：Client $\rightarrow$ API Server $\rightarrow$ 多次本地及全球跨区元数据寻址 $\rightarrow$ Tectonic 块存储 $\rightarrow$ API Server 代理回传 Client。 - **复杂度瓶颈**：对象解析呈现 $O(N)$ 复杂度依赖，长链路调用带来巨大延迟开销。 - **根本冲突**：AI 数据面要求毫秒级的闪存极速访问，与传统跨地域网络往返（RTT）的元数据寻址形成不可调和的矛盾。 --- ### [08:05](https://www.youtube.com/watch?v=xc_BxLsijWA&t=485s) 系统权衡的彻底转移：从高可用成本到算力能效比 不仅是访问延迟，整个系统设计所面临的物理约束和权衡空间（Trade-off Space）也发生了全盘位移。Venkat 提纲挈领地从可靠性、成本效率、能耗以及一致性四个核心维度，对传统应用场景与 AI 训练场景进行了对比反思： 在**可靠性与持久性（Reliability & Durability）**方面，传统架构极致追求极高的高可用性与容灾能力，核心指标是抵御整个数据中心区域离线并对终端用户零感知。为了达成这种极致鲁棒性，系统默认对元数据和数据体进行全球跨地域强同步复制；然而在 AI 训练中，这种跨大区复制带来的高昂延迟不仅没有帮助，反而直接拖垮了训练吞吐。在**成本与能效（Efficiency & Power）**方面，传统应用极度看重每字节存储成本（Dollar per byte），底层严重依赖机械硬盘（HDD）以及复杂冷热分层体系；但在 AI 场景下，对海量 IOPS 的刚性需求使得全闪存（Flash）成为不二之选。更重要的是，在数百亿美元的 GPU 集群账单面前，存储本身的硬件成本几乎可以忽略不计。 与此同时，**数据中心的电力瓶颈（Power Constraint）**成为全新悬顶之剑。Venkat 指出，当代 AI 超级数据中心的电力供应已逼近物理极限，“你在存储设备上多消耗的每一千瓦功率，都是原本可以用来驱动 GPU 核心的宝贵电力”。此外在**一致性语义（Consistency Semantics）**层面，传统业务要求强一致性和复杂的事务操作；而海量 AI 数据集绝大多数具备“单次写入、多次读取”（Write Once, Read Many / WORM）的静态特性，完全可以接受更加宽松的一致性语义，从而为系统激进简化换取极致吞吐扫清了障碍。 | 设计维度 | 传统 Family of Apps 架构 | AI 专用存储新架构 | |:--- |:--- |:--- | | **延迟与性能** | 容忍数十毫秒长尾，偏好吞吐削峰 | 要求亚毫秒级极致低延迟，严控至 $P_{\max}$ 长尾 | | **可靠性与容灾** | 默认全球/跨地域多副本复制，抵御单区崩溃 | 摒弃默认跨区长链复制，优先保障计算局部性与低延迟 | | **硬件介质与成本** | 追求极致每字节成本（Dollar/byte），以 HDD 分层为主 | 追求超高 IOPS，全面拥抱闪存；相对 GPU，存储成本权重极低 | | **能耗约束** | 追求单位存储功耗经济性 | **数据中心电力被 GPU 瓜分，存储每消耗一千瓦即少一千瓦算力** | | **一致性模型** | 严苛的数据读写一致性与复杂状态变更 | 针对 WORM（单次写入多次读取）数据集采用宽松一致性语义 | --- ### [09:43](https://www.youtube.com/watch?v=xc_BxLsijWA&t=583s) 全新对象存储蓝图：$O(1)$ 扁平元数据与客户端去代理化 面对如此深刻的场景变迁，修修补补已无济于事，唯一的出路是推倒重构一套统一的新蓝图。Venkat 展示了 Meta 团队针对 AI 负载量身定制的重构方案，并介绍了其中最关键的三大结构性飞跃（当前分享深入阐述了前两项）： 其一是**重塑元数据层：实现 $O(1)$ 常数级极速寻址**。团队彻底推翻了以往由于系统有机演进而在各子系统之间杂乱分散的元数据存储，将其全盘收拢压缩为一个扁平化的统一单层模式（Flat Unified Schema）。无论对象深埋何处，从 Blob 名称到物理存储地址的解析都被拍平为纯粹的单次 $O(1)$ 查找，相比传统架构呈现出断层式的性能阶跃（Step Function Improvement）。 其二是**数据面彻底去代理化（Eliminate the Proxy）**。在过去的架构中，API 代理网关不仅是网络瓶颈，还会引发二次内存拷贝与额外转发抖动。新架构直接砍掉了数据平面中的中转代理层，将路由、解析及流控逻辑直接下沉并推入客户端 SDK（Client-side SDK）。当训练作业客户端发出 `GET` 读取请求时，请求会被本地进程内的 SDK 直接截获，随后通过精简的控制流直接与底层存储端点建联拉取数据。 - **元数据架构收敛**：打破部门孤岛与服务状态碎片，统一元数据为单层扁平模式，寻址开销从 $O(N)$ 降至 $O(1)$。 - **数据面客户端去代理**：彻底剥除独立 API 代理服务节点，由内嵌客户端的智能 SDK 拦截并主导端到端直连。 - **核心成效**：极大压缩 I/O 链路的软件栈开销，直接将极速闪存硬件潜能转化为训练集群的有效吞吐。 ### [10:20](https://www.youtube.com/watch?v=xc_BxLsijWA&t=620s) 直连存储与新架构基石 在重构后的读路径中，客户端首先将读取请求转化为向 API 服务器发送的 `get read plan`（获取读取计划）请求。API 服务器随后返回响应，其中包含精确的底层存储地址（Storage Address）。获取到该地址后，客户端即可绕过中间代理，直接从存储节点（Storage Nodes）拉取数据流。这种直连拉取模式带来了多维度的技术优势：不仅实现了极高的吞吐量和超低延迟，而且具有极高的能源与算力效率。 最终，该基础架构被设计为区域化（Regionally）部署，将数据与元数据紧密放置在与 GPU 相同的区域内。同时，整套软件栈保持得足够精简敏捷，能够根据 Meta 的业务实际需要，在全球或各个本地区域灵活铺设。所有这些模块的有机结合，构成了支撑 Meta AI 训练的基础底座。 ```mermaid flowchart LR accTitle: Client Direct Read Path Architecture accDescr: Demonstrates how a client interacts with the API server to get storage addresses and streams bytes directly from storage nodes. subgraph ControlPlane["控制平面"] API["API Server"] end subgraph DataPlane["数据平面"] Client["Client / Data Loader"] Storage["Storage Nodes"] end Client -- "1. Get Read Plan" --> API API -- "2. Storage Address" --> Client Client -- "3. Direct Stream" --> Storage ``` --- ### [10:57](https://www.youtube.com/watch?v=xc_BxLsijWA&t=657s) 应对突发流量：Read Plan Cache 与 Hot Object Cache 坚实的底座搭建完成之后，团队开始攻克训练场景中极端且不可预测的突发性读写流量（Bursty Reads and Writes）。为此，系统引入了两个关键的缓存组件： 第一个组件是 **Read Plan Cache（读取计划缓存）**。读取计划负责将高层级的 Blob 名称映射到具体的物理存储地址。Read Plan Cache 本质上是一个分布式内存存储系统（Distributed In-memory Store），专门缓存 Blob 名称到存储地址的映射关系。它能够以极低的时延处理极高并发的每秒查询数（QPS），在流量高峰到达底层元数据引擎之前，有效吸收并削平这些尖峰。 第二个组件是 **Hot Object Cache（热对象缓存）**。工程团队注意到，在实际的模型训练期间，GPU 宿主机（GPU Host）上的系统内存并不总是处于 100% 满负荷状态。因此，团队从宿主机内存中划分出一小部分，将其改造为一个分布式点对点（P2P）的热对象缓存。在实际生产环境中，Hot Object Cache 在各类负载下普遍展现出了高达 80% 的缓存命中率。这两项机制的双剑合璧，使系统在读取路径上从容抵御突发流量的冲击。 - **Read Plan Cache**：分布式内存级映射缓存，专为高 QPS、低延迟设计，专门用于吸收元数据查找尖峰。 - **Hot Object Cache**：利用 GPU 宿主机闲置内存构建的点对点（P2P）分布式缓存，在生产环境中达到约 80% 的命中率。 --- ### [12:04](https://www.youtube.com/watch?v=xc_BxLsijWA&t=724s) 边际攻坚与研究员面临的研发摩擦 完成了基础架构重构与缓存优化后，团队解决了大约 80% 到 90% 的性能瓶颈。然而，正如软件工程中经典的法则所言，“最后的 10% 往往需要耗费极漫长的时间（the last 10% takes forever）”，实际推进也是如此。为了最终冲过终点线，团队在整套技术栈中实施了一系列密集的工程微调与底层优化。AI 工作负载引发的权衡空间位移，迫使 Meta 从根本上推倒并重塑了存储基石，才得以在维持高 GPU 利用率的同时满足训练吞吐。 攻克了单集群读性能后，聚光灯转向了另一个核心诉求：**如何最大化研究速度（Research Velocity），进而驱动业务的创新步伐**。 在 Meta，AI 研究员的标准工作流通常包括：从不同来源整理并精选数据集，持久化保存到 Blob 存储；选定一个运行训练任务的目标区域；将海量数据集手动跨区拷贝到对应 GPU 算力所在的区域；运行训练任务；任务结束后分析输出指标、调整参数或数据，再开始新一轮实验。 这一看似常规的流程在现实中却给研究员带来了巨大的摩擦： 1. **陷入算力寻址决策**：研究员必须花心思介入底层的物理基础设施决策，挑选目标区域。 2. **漫长的数据搬迁阻塞**：跨区域复制庞大的数据集往往耗时数小时甚至数天，在这期间，研究员完全处于被阻塞的状态，只能被动“干等存储系统完成传输”。 --- ### [13:49](https://www.youtube.com/watch?v=xc_BxLsijWA&t=829s) 痛点剖析：大模型与长尾任务的需求分歧 为什么研究员不得不经历这些繁琐的搬迁流程？根本原因在于 GPU 算力是全球稀缺资源，Meta 的计算资源并非集中在单一大型中心，而是分散在遍布全球的多个算力集群中。从传统的高性能计算角度看，为了实现最高性能，通常要求数据和元数据必须与 GPU 部署在同一个区域。因此，研究员才被迫在启动训练前把数据集复制到特定数据中心。 然而，深入观察生产实践后，团队发现了显著的特征分化： - **超大规模预训练（Large-scale Pre-training）**：任务往往会连续运行数周甚至数月，这类任务对本地极致性能的诉求最强，提前搬迁数据是划算的。 - **绝大多数中小型训练任务**：生命周期通常在几小时到一天内结束。对于这部分占绝对多数的长尾任务，**快速迭代的敏捷度远比追求榨干最后一滴存储性能更重要**。 --- ### [14:46](https://www.youtube.com/watch?v=xc_BxLsijWA&t=886s) 借鉴 OS 思想：分层缓存与透明按需灌库 AI 训练数据具备显著的“一次写入，多次读取”（Write Once, Read Many）特征。研究员最理想的体验是：**只需将数据摄入一次，就能在全球任何算力节点无视物理区域边界、随时按需消费**。 为了实现这一目标，架构团队将全球分布式存储系统抽象类比为一个“行星级计算机（Planet-scale Computer）”中的硬盘，并从操作系统的经典虚拟内存与缓存设计中汲取灵感。正如 Linux 进程访问磁盘文件时，操作系统会透明地将数据逐级向上加载到内存 Page Cache，再进入各级 CPU 缓存直至 L1 缓存一样，团队构建了一套跨越全球到 GPU 芯片的多级分层缓存体系（Tiered Caching Solution）： ```mermaid flowchart TD accTitle: Tiered Caching Architecture accDescr: Hierarchical data hydration flow from global blob storage to regional tiers and GPU host memory. subgraph GlobalLevel["全球持久化层 (Permanent Source of Truth)"] GlobalBlob["Global Blob Storage (Planet-Scale Disk)"] end subgraph RegionalLevel["区域级缓存 (L3)"] FlashBlob["Regional Blob Storage (Flash-backed L3 Tier)"] end subgraph GPUHostLevel["GPU 宿主机缓存 (L1 & L2)"] NVMe["Host NVMe (L2 Tier)"] HostMem["Host Memory (L1 Tier)"] GPU["GPU Core / HBM"] end GlobalBlob -->|"按需透明灌库 (On-demand Hydration)"| FlashBlob FlashBlob --> NVMe NVMe --> HostMem HostMem --> GPU ``` 当某区域的训练任务发起远程数据访问时，Data Loader 配合客户端 SDK 会自动将数据从全球 Blob 存储透明灌库（Transparent Hydration）到本地 L3 闪存，再层层提拉至 GPU 宿主机内存中。 这种远程加载模式看似增加了访问延迟，但团队通过两种关键的预取（Prefetching）机制完美掩盖了网络开销： 1. **Data Loader 预取**：在 GPU 计算处理当前批次数据时，利用流水线并发预取紧邻的下一批次数据。 2. **后台深度预取（Background Prefetch）**：Data Loader 向前预测，识别出作业在未来数分钟内将要消费的数据集合，主动下发预取任务。 通过两级协同预取，系统在隐藏跨区延迟的同时，始终维持住了高水平的 GPU 算力利用率。 --- ### [17:29](https://www.youtube.com/watch?v=xc_BxLsijWA&t=1049s) 收益落地：摄入时延骤降 90% 与基础设施下一步 这一套按需透明灌库体系上线后，取得了极其亮眼的生产成效： - **数据准备时延骤降 90%**：全工作流中的摄入与等待时间大幅削减，彻底解除了此前长达数小时或数天的等待。 - **演进策略**：在生产环境中，系统依然兼容预先复制数据的传统路径与按需自动灌库路径，但“按需灌库”已迅速成为各团队日常训练的默认主流方式。 在技术演讲的结尾，Sidharth Bajaj 进行了全盘总结，并前瞻了 Meta 存储的演进重点： - **数据规模暴增 10 个数量级**：过去十年间，训练前沿基础模型所需的数据量暴涨了 10 个数量级，这种史无前例的吞吐需求要求全栈重塑。 - **让存储触达网络极限**：随着 GPU 硬件的极速迭代，从 GB200 到 GB300，网卡（NIC）带宽直接翻倍，然而主机端的 CPU 资源却没有对应增长。存储层必须深入优化，以在低 CPU 开销下持续打满最新硬件的高速网卡带宽。 - **检查点（Checkpointing）优化**：模型与状态数据爆炸使得 Checkpoint 越来越庞大、触发频率越来越高，写路径的存储架构必须持续改造。 - **全面攻坚推理工作负载（Inference Workloads）**：本次演讲主要聚焦于训练期存储，而大规模在线推理的存储性能优化，已成为 Meta 正在全力应对的下一代核心挑战。 ## Notable quotes > [01:20](https://www.youtube.com/watch?v=xc_BxLsijWA&t=80s) 当你建造的是一个规模堪比整个曼哈顿、容纳了数十万张 GPU 的巨型数据中心时，这 20% 的算力停顿意味着每小时就有高达数千万美元白白蒸发（tens of millions of dollars lost every hour）。 — Sidharth Bajaj > [04:03](https://www.youtube.com/watch?v=xc_BxLsijWA&t=243s) 对于 AI 工作负载而言，存储可预测的稳定延迟必须一路严防死守到最极端的尾延迟（P_max / 100分位）。 — Venkatraghavan Srinivasan > [06:40](https://www.youtube.com/watch?v=xc_BxLsijWA&t=400s) 根本无法接受的致命伤（a total non-starter） — Venkatraghavan Srinivasan > [08:05](https://www.youtube.com/watch?v=xc_BxLsijWA&t=485s) 你在存储设备上多消耗的每一千瓦功率，都是原本可以用来驱动 GPU 核心的宝贵电力 — Venkatraghavan Srinivasan > [09:43](https://www.youtube.com/watch?v=xc_BxLsijWA&t=583s) 断层式的性能阶跃（Step Function Improvement） — Venkatraghavan Srinivasan > [12:04](https://www.youtube.com/watch?v=xc_BxLsijWA&t=724s) 最后的 10% 往往需要耗费极漫长的时间（the last 10% takes forever） — Venkatraghavan Srinivasan > [13:49](https://www.youtube.com/watch?v=xc_BxLsijWA&t=829s) 快速迭代的敏捷度远比追求榨干最后一滴存储性能更重要 — Venkatraghavan Srinivasan > [14:46](https://www.youtube.com/watch?v=xc_BxLsijWA&t=886s) 只需将数据摄入一次，就能在全球任何算力节点无视物理区域边界、随时按需消费 — Venkatraghavan Srinivasan ## Mentioned Meta, Sidharth Bajaj, Venkatraghavan Srinivasan, @Scale, Tectonic, Llama, Facebook, Instagram, Family of Apps, GB200, GB300 ## Video description The rapid, exponential growth in model capabilities and training dataset sizes over the last few years has accelerated AI innovation. New frontier models are now being released in a matter of weeks, down from months just a year ago. This pace makes reliable, consistent fast access to storage essential for managing both the speed and cost of development. This presentation will detail how we evolved Meta's Storage Architecture to overcome two key hurdles: optimizing GPU Utilization and maximizing Research Velocity. Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=xc_BxLsijWA)

[1:20] Sidharth Bajaj: 当你建造的是一个规模堪比整个曼哈顿、容纳了数十万张 GPU 的巨型数据中心时，这 20% 的算力停顿意味着每小时就有高达数千万美元白白蒸发（tens of millions of dollars lost every hour）。 [4:03] Venkatraghavan Srinivasan: 对于 AI 工作负载而言，存储可预测的稳定延迟必须一路严防死守到最极端的尾延迟（P_max / 100分位）。 [6:40] Venkatraghavan Srinivasan: 根本无法接受的致命伤（a total non-starter） [8:05] Venkatraghavan Srinivasan: 你在存储设备上多消耗的每一千瓦功率，都是原本可以用来驱动 GPU 核心的宝贵电力 [9:43] Venkatraghavan Srinivasan: 断层式的性能阶跃（Step Function Improvement） [12:04] Venkatraghavan Srinivasan: 最后的 10% 往往需要耗费极漫长的时间（the last 10% takes forever） [13:49] Venkatraghavan Srinivasan: 快速迭代的敏捷度远比追求榨干最后一滴存储性能更重要 [14:46] Venkatraghavan Srinivasan: 只需将数据摄入一次，就能在全球任何算力节点无视物理区域边界、随时按需消费

[1:00] Sidharth Bajaj: When you're building a data center which is the size of Manhattan... that 20% stall is about tens of millions of dollars lost every hour. Like just let that sink in—tens of millions of dollars every hour lost while your GPUs are waiting for your storage. [9:00] Sidharth Bajaj: AI workloads essentially shifted the trade-off space enough for us to rethink the entire foundations... power efficiency matters a lot for AI workloads. Data centers are becoming increasingly power constrained, and every kilowatt of power that you actually spend on storage is a kilowatt that you cannot spend on GPUs.

[0:01] Hey everyone. Welcome to AI storage [0:04] blueprint session. I'm Sidharth and this [0:07] is my colleague Venkat. We're going to [0:09] walk you through how we've reimagined [0:11] the storage infrastructure for the [0:14] ever-evolving [0:16] AI workloads. [0:18] I hope you all have a great time. [0:20] All right, so let's take a step back and [0:22] kind of like just look at how the data [0:24] space have evolved over the years. [0:27] So there's three main broad categories [0:30] of changes that we've seen. I think the [0:32] first is the amount of data. There's [0:34] been a 10 [0:36] order magnitude increase in the amount [0:38] of data that is now being used to train [0:40] the frontier models. [0:42] The second change we've seen is when we [0:44] started out on the frontier model [0:46] training, it was all about text. Now [0:49] that's changed to not only be text, but [0:52] also involve audio, long-form video, [0:55] short-form video. [0:56] Um [0:58] which has increased the amount of data. [1:00] The third change we've seen is when we [1:01] started at when we started on this [1:03] journey, it was a lot more [1:05] human-generated content, which is still [1:07] the case. But it is very closely being [1:10] caught up by AI-generated content. So [1:13] those are like three broad categories of [1:15] changes that we're seeing in storage [1:18] play out. [1:20] So why does storage matter for the AI [1:23] workloads? So when we looked at GPUs, [1:26] and this is a snapshot on the screen of [1:28] 120 seconds on the performance of in [1:31] GPU. What we realized that there were [1:33] about five stalls, which is about 20% of [1:36] the time the GPUs were just waiting, not [1:39] doing anything. And then we when we look [1:42] deeply into that, what we've realized is [1:44] they were waiting for the data. [1:46] Um so now you say, "Okay, 20% not that [1:50] bad." But when you're building data [1:52] center which is the size of Manhattan, [1:54] right, hundreds and thousands of GPUs, [1:56] that 20% stall [1:59] is about tens of millions of dollars [2:01] lost every hour. [2:03] Like just let that sink tens and [2:05] millions of dollars every hour lost [2:08] while your GPUs are waiting for your [2:10] storage. So that's just the financial [2:12] implications. [2:14] The second order problem that we've seen [2:16] which is [2:17] an opportunity cost because by while [2:20] it's taking longer for your GPUs to [2:22] produce the results they're looking for [2:24] because they're waiting for data is your [2:26] researchers are not able to move fast. [2:29] And what's the impact? I think when we [2:31] look at 2020 2022 when the initial [2:33] models came out, you know, major upgrade [2:35] was about 104 weeks and fast forward to [2:38] 2026 that's about four weeks a new [2:41] frontier models coming. So the research [2:43] velocity is super super important just [2:46] as much as the cost. [2:49] So I think we want to kind of been going [2:51] on this journey with you all from 2024 [2:53] to now. So kind of wanted to just give a [2:55] high-level overview of what storage at [2:57] Meta looks like and kind of take it from [2:59] there. So at the bottom layer for us is [3:02] tectonic which is a horizontally [3:03] scalable block layer consider consisting [3:06] of HDD TLCs and other technologies. [3:09] In 2024 [3:11] we kind of walked everyone here through [3:14] the file API and how we [3:16] improve those in order to support the [3:19] llama workloads. Then in 2025 we kind of [3:21] walked the group through what we did and [3:24] how did we deploy QLCs globally as one [3:26] of the largest production deployment of [3:29] QLC technology. [3:31] And today what we're going to do is how [3:32] we've optimized the object API path on [3:35] top of our block layer to support and [3:39] enable the upcoming models at Meta. [3:43] So I I think like a broad theme for [3:45] today is going to be two problems that [3:46] we're going to talk about. How do how [3:48] are we maximizing the GPU utilization [3:51] for cost and the second is how are we [3:54] maximizing the research velocity. So I [3:56] think to walk through the details of [3:58] both of these on what we've been doing, [3:59] my colleague Venkat's going to join us. [4:01] Venkat, the stage is all yours. [4:03] >> Thanks, Siddharth. [4:05] Hey everyone. Um my name is Venkat. I'm [4:08] a software engineer working with the [4:10] blob storage team at Meta. [4:12] So today I will be deep diving into some [4:14] of the topics that Siddharth started [4:15] with. Um so let's jump right in and talk [4:18] about the story of how we maximize GPU [4:20] utilization. [4:22] Um I want to start with a little bit of [4:23] background about AI workloads in [4:25] general. Um let's talk model training. [4:28] At a high level, you can think of every [4:30] GPU as a data processing core that's [4:32] continuously reading data from storage, [4:35] like doing some processing, and then [4:36] periodically checkpointing its state [4:37] back to storage. [4:39] So what's unique and challenging about [4:40] this workload for storage? [4:43] Two things. One, like this operation [4:45] happens in parallel across hundreds and [4:47] thousands of GPUs. [4:49] And second, these GPUs typically [4:51] synchronize among themselves before [4:53] performing operations such as data [4:55] loading or checkpointing. [4:56] So the combination of massive [4:58] parallelism and the synchronized reads [4:59] and writes is what makes it challenging [5:01] for storage. [5:03] The resultant bursty reads and writes [5:05] and create a demand for higher IOPS. [5:08] In addition to this, the IO patterns [5:10] vary a lot across these workloads, which [5:12] is another challenge for storage. [5:16] All right. So with that context in mind, [5:18] let's deep dive a little bit into data [5:20] loading and see how latency can impact [5:22] GPU utilization. [5:24] So the picture you see on the screen [5:25] shows two GPUs loading data from [5:27] storage. [5:28] As we discussed earlier, the GPUs are [5:30] continuously reading data from storage [5:32] and processing them in batches. [5:34] Now, for maximum compute IO overlap, the [5:38] CPU cores on these GPU machines are [5:41] prefetching the data that is needed by [5:43] GPUs next while the GPUs are processing [5:45] the previous batches of data. [5:48] Now, as long as the time it takes to [5:50] prefetch the next batch is within the [5:53] time it takes for the GPUs to process [5:54] the previous fresh batches of data, the [5:57] GPUs are never going to be stalled. [5:59] So, as you can see in picture one, like [6:01] GPU one here, so this is the case, like [6:04] the latencies are well within bounds, [6:06] GPU one is never stalled. [6:08] In the case of GPU two though, you can [6:10] see that there are a couple of instances [6:11] where storage latency was high, and as a [6:14] result the GPU was stalled. [6:16] Now, because these GPUs synchronize [6:18] among themselves, like uh [6:20] GPU one also waits ends up waiting for [6:23] GPU one to finish, like resulting in [6:25] overall like wastage of GPU cycles [6:27] across the board. [6:29] Now, the key takeaway here is that uh [6:31] predictable latencies from storage [6:33] matter all the way up to Pmax for AI [6:36] workloads. [6:39] All right, next let's talk about the [6:41] legacy block storage architecture which [6:43] used to serve the Meta family of apps. [6:46] Uh when I talk about family of apps, [6:47] this is the standard Instagram, [6:49] Facebook, all of the standard Meta [6:50] products. Uh this picture here shows the [6:53] life of a get request. Uh [6:55] when the client sends a get request, the [6:57] request arrives at the API server. [6:59] The first thing that the API server does [7:01] is translate a blob name to a storage [7:04] address. [7:05] To achieve that, it does a series of [7:06] metadata lookups, some regional and some [7:09] global. [7:10] At the end of it, it has the storage [7:11] address, which it sends to the tectonic [7:13] block layer, and streams the bytes from [7:16] the block layer back to the client. [7:19] Now, this works pretty well for the [7:21] family of apps, so what's the problem [7:23] for AI workloads? [7:25] So, the main problem is that mapping a [7:26] blob name to a storage address is an [7:29] order of n operation, where some of the [7:31] lookups we do cross regions. [7:34] Why is that the case? [7:35] Uh this is because like this [7:37] architecture evolved organically, and [7:40] over the years we've added many layers. [7:42] Um some of these layers ended up taking [7:45] their own dependency for managing their [7:46] state for metadata. [7:49] With AI workloads which have millisecond [7:51] access to data on flash, these lookups [7:54] across different services, some of them [7:55] global, are a total non-starter. [7:59] So, clearly latency is a problem with [8:01] this architecture. Is that the only [8:03] problem? [8:05] Let's take a more holistic view into [8:07] different design constraints and [8:08] variables. [8:10] Uh [8:11] as you can see, many variables have [8:12] changed for AI workloads. Uh [8:15] We just talked about performance and [8:16] latency. [8:17] Let's talk a bit about [8:19] uh reliability and durability. [8:22] So, the traditional architecture is [8:23] highly optimized for high availability [8:26] and durability. [8:27] Uh what do I mean by that? Like, it's [8:29] designed to survive scenarios such as [8:30] like regions going down with zero impact [8:33] to users. [8:34] But to achieve that, uh like both data [8:37] and metadata are replicated globally by [8:38] default. [8:40] With AI workloads, this design choice [8:42] hurts rather than helps with [8:43] performance. [8:45] Now, moving on to efficiency, uh the [8:48] traditional architecture is highly [8:49] optimized for cost. [8:51] Uh it is backed by hard drives, and [8:53] there is various layers of tearing [8:55] optimizing for dollar per byte. [8:57] With AI workloads, the demand for high [8:59] IOPS necessitates flash. And in [9:01] addition, the cost of storage is [9:03] typically negligible relative to the [9:05] cost of GPUs. [9:07] On the other hand, power efficiency [9:09] matters a lot for the AI workloads. [9:11] Data centers are becoming increasingly [9:13] power constrained, and every kilowatt of [9:15] power that you actually spend on storage [9:17] is a kilowatt that you cannot spend on [9:18] GPUs. [9:21] Finally, talking a bit about consistency [9:22] semantics, um AI many of the AI data [9:25] sets are write once, read many, and can [9:28] benefit from relaxed consistency [9:29] semantics. Whereas, this isn't always [9:31] the case with [9:33] uh traditional workloads. [9:35] In in summary, the trade-off space has [9:37] shifted enough for us to rethink the [9:39] entire architecture into one. [9:43] So, this brings to the picture you see [9:44] here. At a high level, uh we made three [9:48] major changes. First, [9:50] we collapse the metadata that's [9:52] distributed and spread across different [9:53] services into a single flat unified [9:56] schema. [9:57] This helps achieve order of one look up [9:59] to map a blob name to a storage address, [10:01] which is a step function improvement [10:03] relative to the traditional [10:04] architecture. [10:07] Second, we eliminate the proxy in the [10:09] data plane and push logic into the [10:11] client-side SDK. [10:13] Now, with this new design, when the [10:15] client sends a get request, the request [10:18] is intercepted by the SDK and it [10:20] translates that to a get read plan [10:22] request to the API server. [10:24] The API server then returns back a get [10:26] read plan response containing the [10:28] storage address, and the client is now [10:30] able to stream bytes directly from the [10:32] storage nodes. [10:33] This helps with high throughput, low [10:35] latency, as well as power efficiency. [10:39] Lastly, this architecture is now [10:41] deployed regionally with data and [10:43] metadata co-located in the same region [10:44] as the GPUs. [10:46] Now, this stack itself is lean enough to [10:48] be deployed regionally or globally as [10:50] the need arises. [10:52] The combination of all of this makes up [10:54] the new foundation for us. [10:57] Now the foundation's in place, we are [11:00] ready to tackle the problem of bursty [11:01] reads and writes. [11:03] To solve this problem, we introduced two [11:05] new components. One is what we call a [11:07] read plan cache. [11:09] Uh as we just discussed, read plan [11:11] refers to the metadata that maps a blob [11:13] name to a storage address. [11:15] And read plan cache is nothing but a [11:17] distributed in-memory store that stores [11:19] the mappings of blob names to storage [11:21] addresses. [11:22] It is able to serve very high QPS with [11:24] very low latency and effectively absorb [11:26] the spikes. [11:28] The second component we added was a hot [11:30] object cache. [11:32] Uh during model training, we noticed [11:34] that the memory on the GPU host is not [11:36] always 100% utilized. So, we carve out a [11:39] small portion of the memory on the GPU [11:40] host to uh [11:42] repurpose that as a distributed [11:44] peer-to-peer hot object cache. [11:46] Now, in production, like we typically [11:48] observe about 80% cache hit rate um on [11:51] the hot object cache across different [11:52] workloads. [11:54] So, now the combination of these two [11:56] mechanisms allows us to efficient [11:57] effectively deal with like bursty reads [12:00] uh in the read path. [12:04] All right. So, what what we discussed so [12:07] far got us 80 to 90% of the way, but as [12:10] you all know, the last 10% takes [12:12] forever, and it was no different for us. [12:15] We ended up having to do a bunch of [12:16] optimizations across the stack to get us [12:18] to the finish line. [12:20] Uh this slide here lists a few of these [12:23] optimizations, but by no means [12:24] exhaustive. [12:28] To summarize, AI workloads essentially [12:30] shifted the trade-off space enough for [12:32] us to rethink the entire foundations, [12:34] and the new foundations that we just [12:36] talked about uh allows us to effectively [12:38] serve AI workloads with reasonable [12:40] amount of GPU utilization. [12:45] Next, let's talk about the story of how [12:47] we maximize research velocity um and in [12:49] turn the speed of innovation. [12:54] So, this workflow that you see here is a [12:57] typical workflow for a researcher at [12:58] Meta. [13:00] Let's say you want to start a job. [13:02] Uh you begin by curating data sets from [13:04] different sources and persisting them in [13:06] blob storage. [13:07] Next, you pick a region on which you [13:10] want to train run your training job on. [13:13] Then, you copy your data sets onto the [13:15] region where you want to run your job [13:17] on. [13:18] And after the job finishes, you analyze [13:20] the outputs, you tweak some parameters, [13:22] data sets, and you're ready to start [13:24] again. [13:25] So, this seems like a pretty [13:26] straightforward workflow. What are the [13:28] parts that slow down researchers [13:30] specifically? [13:33] First, researchers are in the business [13:35] of picking a target region. [13:37] And second, researchers copy data sets [13:40] across regions, which depending upon the [13:42] size of the data sets can take hours to [13:44] days, during which time they're simply [13:46] blocked waiting for storage. [13:49] But why do researchers have to do these [13:51] operations in the first place? [13:53] As many of you here know, GPUs are [13:55] scarce, and so like Meta has many [13:58] different clusters spread across the [13:59] globe as opposed to one cluster in a [14:02] single region. [14:04] And for performance reasons, it's more [14:05] optimal to have the data and metadata [14:08] co-located in the same region as the [14:09] GPUs. [14:10] And it is precisely for this reason that [14:12] researchers copy the data sets onto the [14:14] region where they want to start their [14:15] job on. [14:17] That said, in practice, what we observed [14:20] was this level of performance matters [14:22] the most for large-scale pre-training [14:24] jobs, which span for weeks to months. [14:27] But the vast majority of the jobs don't [14:29] fall into that category. [14:31] They finish within hours to a day, and [14:33] for these jobs, the speed of iteration [14:35] matters a lot more than the [14:37] maximum performance you can get out of [14:39] storage. [14:41] So how can we do better for these [14:42] workloads? [14:46] So we exploit the fact that the data [14:48] these data sets are write once, read [14:50] many. [14:51] And ideally, the experience that [14:53] researchers are looking for is they're [14:55] able to ingest the data once and consume [14:58] them anywhere irrespective of any [14:59] regional boundaries. [15:01] So how could we achieve that? [15:04] What if we think of storage as a disk in [15:06] a planet-scale computer and borrow ideas [15:09] from the operating system world? [15:12] As most of you here know, when a process [15:14] Linux process tries to access a file [15:16] from the disk, the operating system [15:18] transparently hydrates the data from the [15:20] disk to the page cache in memory to the [15:22] various CPU caches all the way up to the [15:25] L1 cache. [15:26] Could we do something similar? [15:30] The answer turned out to be yes, and it [15:32] got us to the picture you see here. [15:34] At a very high level, we leverage the [15:37] blob storage, global blob storage stack, [15:39] along with the resources on the GPU [15:41] host, memory and NVMe, to build [15:44] a tiered caching solution. [15:47] This, along with smart prefetching [15:48] techniques, allows us to effectively [15:50] like solve our problems. [15:53] So, going back to the analogy we just [15:54] talked about previously, [15:56] you can think of the [15:58] disk as the global blob storage in the [16:00] picture here. [16:02] So, researchers ingest the data once, [16:04] and this becomes a permanent source of [16:06] truth. [16:08] Now, zooming onto a specific region, you [16:10] can think of the memory on the GPU host [16:12] as the L1 tier, [16:14] the NVMe as the L2 tier, and the [16:17] regional blob storage, which is backed [16:18] by flash, as the L3 tier. [16:21] Now, if any of if one job tries to [16:24] access data from a specific region, [16:27] remotely, [16:28] the data loader, along with the [16:30] client-side SDK, is able to [16:32] transparently hydrate the data from the [16:35] global blob storage to the L3 tier, and [16:37] all the way up to the GPU memory. [16:40] All right, this sounds great on paper, [16:42] but aren't we like undoing some of the [16:44] latency optimizations we just talked [16:46] about previously? [16:48] Uh Naively speaking, the answer is yes, [16:50] but like we mitigate that with two kinds [16:53] of prefetching techniques. [16:55] One is the data loader prefetch, which [16:57] we discussed earlier, where the data [16:59] loader prefetches the next batches of [17:01] data, [17:02] while the GPU processes the previous [17:04] batch of data. [17:05] The second type of prefetch is what we [17:08] call a background prefetch, where [17:10] the data loader looks ahead and [17:11] identifies the data which is needed over [17:13] the course of next few minutes, and [17:15] issues prefetch to proactively prefetch [17:18] those. [17:20] The combination of these mechanisms [17:21] allows us to effectively hide latencies [17:25] and keep the GPU utilization high. [17:29] All right, so [17:31] like moving on to some results, this [17:33] table here shows the ingestion latencies [17:37] before and after the on-demand hydration [17:39] we just talked about. [17:41] As As you can see, there is about like [17:43] 90% decrease in ingestion times across [17:46] the board. [17:48] Today in production, we continue to [17:49] support both the on-demand hydration [17:52] technique we just talked about as well [17:53] as the traditional way of copying data [17:56] sets before the job starts. [17:58] And the on-demand hydration technique is [18:00] becoming increasingly the default [18:02] going forward. [18:04] All right, that's mostly what I had to [18:06] cover from a technical perspective. I'm [18:08] going to hand off back to Sid for some [18:10] closing thoughts. [18:12] >> All right, thanks Venkat for all the [18:14] details. [18:15] All right, let's get on to some key [18:17] takeaways here. Modern AI workloads are [18:21] data hungry. Like as I said, in the past [18:23] decade, the amount of data that has [18:25] increased to train the frontier model is [18:27] 10 order magnitude. Just huge amount of [18:30] data. [18:31] The second thing, the storage [18:33] performance is super critical if you [18:35] want to have the highest possible [18:36] utilization on the GPUs and to allow [18:39] your researchers to move as fast as they [18:41] can. [18:44] Now, the AI workloads are so different [18:47] from any workloads that we've seen and [18:49] they're growing at a, you know, just an [18:51] unbelievable rapid pace that we've just [18:54] had to reimagine and re-architect our [18:56] whole stack. And that's what we've been [18:58] done and that's the journey we've been [19:00] taking you along for the past couple of [19:01] years. [19:03] Now, as Venkat was also saying, now the [19:05] GPUs are can be located anywhere in the [19:07] world. The data is being produced [19:09] everywhere in the world. So, [19:12] so such things as like tiered caching, [19:14] pre-fetching, and on-demand hydration [19:16] becomes really important from storage [19:18] perspective. So, these are like the key [19:20] takeaways from the talk. [19:22] And, you know, this is the work as I [19:24] said, it's fast evolving. We're moving [19:26] very fast, and we continue to innovate [19:28] in this field. So, there's a lot more [19:29] work we're doing [19:31] right now as well as over the next few [19:34] months and years. [19:35] Uh one is scaling the storage to network [19:37] limits. Now, what do I mean by that? [19:39] Like, I think as each generation of GPUs [19:41] are coming, the NIC is increasing. When [19:44] you go from GB200 to GB300, the NICs [19:47] almost double. But, your resources such [19:50] as CPU haven't changed at all. So, that [19:52] means in storage, we got to work with [19:54] that and make sure we're utilizing the [19:56] NICs on these greatest and the newest of [19:58] GPUs all the time. [20:01] Now, one of the key bits that we talked [20:02] about was the data loading path, which [20:04] is one half of the story. There's also [20:06] optimiz- optimizations of the [20:08] checkpointing. [20:09] As the data grows more, the checkpoints [20:11] are bigger, they're more frequent, and [20:13] there's another more optimizations and [20:15] more changes we have to make to our [20:17] stack. [20:18] And, you know, as all of you would have [20:19] caught on, like, all we talked about [20:21] today was training. But, then inference [20:23] workloads are here, and that's the next [20:25] challenge that we are working on at Meta [20:27] on how do we optimize for the upcoming [20:29] challenge with the inference workloads. [20:31] So, thank you, everyone. I'd like to get [20:33] Venkat back here for some questions.