Meta Normalizes AI Training Data to Avert Exabyte Capacity Crunch

youtube

Decoupling long user sequences from monolithic training rows into an optimized key-value store reduced storage costs tenfold, averting an infrastructure crisis that consumed 10% of Meta's global data center power.

AI-assisted summary of How Meta Scaled AI Training Storage via Data Normalization | Sarang Masti and Weiran Liu from Meta — original source. Check the original for full context.

Source published: 2026-06-19

Key ideas

Adopting long user sequences expanded Meta's AI training data to tens of exabytes across 10+ data centers, consuming 10% of the company's total power budget and forcing severe downsampling. (1:21)
To counter extreme duplication from logging identical user histories in every recommendation row, Meta normalized its training data by storing lightweight pointer keys in warehouse tables and moving raw sequences into ZippyDB. (6:22)
Engineers resolved major early throughput and latency bottlenecks in ZippyDB and RocksDB by bypassing the write path via offline SSTable generation, adding parallel multi-scans with I/O coalescing, and pushing down schema-aware projection and truncation. (10:52)
The optimized Sequence Storage system increased per-server read throughput fivefold (from 200 MB/s to over 1 GB/s) and scaled maximum sequence lengths by 40x (from 2,000 to 80,000 events) across more than 30 production recommendation models. (17:25)

Detailed notes

## Background Presented at the @Scale Conference and published on YouTube on June 19, 2026, this technical talk features Meta engineers Sarang Masti and Weiran Liu detailing how Meta scaled AI training storage for its recommendation systems. The presentation description highlights how shifting to rich user sequences inflated feature footprints by 10x to 100x, causing massive duplication in legacy relational tables, and how a co-designed indexing and storage architecture cut sequence storage costs tenfold.

## TL;DR To improve recommendation quality, Meta transitioned from condensed feature summaries to long user interaction sequences, inadvertently triggering a storage crisis that ballooned data footprints across more than 10 data centers and consumed 10% of the company's power budget. The team solved this by normalizing the training data: retaining lightweight pointer keys in warehouse tables while offloading raw sequences into ZippyDB. When initial iterations introduced severe read amplification and join latency, Meta implemented offloaded SSTable generation in Spark and Presto, added parallel multi-scan and I/O coalescing to RocksDB, and pushed projection and truncation down into the storage engine. These optimizations boosted per-server read throughput fivefold, expanded supported sequence lengths from 2,000 to 80,000 events across 30+ production models, and paved the way for lifetime user sequence modeling.

## Atmosphere & style The presentation carries the polished, authoritative, and deeply technical tone characteristic of high-stakes infrastructure conferences. Liu and Masti deliver a measured, pragmatic narrative that transitions from an urgent macro-level capacity crisis into granular systems-level post-mortems, speaking with candid transparency about early design iterations that initially fell short on cost. The talk feels like an insider masterclass in systems engineering, maintaining a disciplined focus on bottlenecks, hardware mechanics, and the end-to-end co-design required to make exabyte-scale ML viable.

## Bulletin - [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) Adopting long user sequences expanded Meta's AI training data to tens of exabytes across 10+ data centers, consuming 10% of the company's total power budget and forcing severe downsampling. - [06:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=382s) To counter extreme duplication from logging identical user histories in every recommendation row, Meta normalized its training data by storing lightweight pointer keys in warehouse tables and moving raw sequences into ZippyDB. - [10:52](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=652s) Engineers resolved major early throughput and latency bottlenecks in ZippyDB and RocksDB by bypassing the write path via offline SSTable generation, adding parallel multi-scans with I/O coalescing, and pushing down schema-aware projection and truncation. - [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) The optimized Sequence Storage system increased per-server read throughput fivefold (from 200 MB/s to over 1 GB/s) and scaled maximum sequence lengths by 40x (from 2,000 to 80,000 events) across more than 30 production recommendation models.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=5s) Prior training storage optimizations like columnar compression, row/column reordering, and model-aware placement proved insufficient once model features grew 100x larger. - [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) Long chronological user sequences drastically improve recommendation quality compared to static summary features, making sequence length directly correlated with model performance. - [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) The rapid growth of user sequences overwhelmed data center capacity, forcing engineers to cut training data retention down to 30 days and sacrifice annual seasonality learning. - [04:06](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=246s) The legacy training architecture snapshotted all features inline within self-contained rows, causing massive duplication because a user's sequence changes minimally between frequent recommendations. - [06:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=382s) Meta adapted relational database normalization by keeping primary rows on warm HDD storage with sequence ID pointers while offloading raw sequences into ZippyDB. - [08:46](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=526s) V1 of Sequence Storage was as expensive or more costly than duplicated storage because ZippyDB was tuned for OLTP rather than high-throughput batch AI training reads. - [10:52](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=652s) Continuous row-by-row writes in RocksDB created deep LSM trees and aggressive background compactions, causing severe read amplification and starving flash bandwidth. - [10:52](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=652s) Using Spark and Presto to generate pre-compacted, sorted SSTables offline allowed ZippyDB to bypass memtables and WALs, achieving a single-level LSM layout with roughly one flash I/O per read. - [13:44](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=824s) RocksDB's lack of native multi-scan originally forced serial range scans whose latencies stacked cumulatively, creating pipeline stalls for GPU training workers. - [13:44](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=824s) Implementing parallel multi-scans and I/O coalescing in RocksDB grouped adjacent disk block reads, reducing media IOPS and bounding batch latency to the single slowest scan. - [15:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=922s) Because raw key-value storage is schema-agnostic, training workers initially had to download full sequence blobs over the network before filtering unneeded attributes on client CPUs. - [15:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=922s) Pushing feature projection and length truncation down into ZippyDB cut transmitted network data and GPU client parsing loads to about one-third of their previous volume. - [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) The project demonstrates that modern AI training storage requires end-to-end co-design across offline data pipelines, core storage engines, and GPU ingestion layers. - [19:28](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1168s) Meta's future sequence storage roadmap targets scaling sequence length by another 10x to capture lifetime user history (hundreds of thousands of events) and generalizing the normalized architecture to all ML features.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=5s) Scaling Challenges and Meta's Traditional Storage Optimizations

Weiran Liu introduces the talk alongside his colleague Sarang Masti, setting the stage for how Meta built "Sequence Storage" to scale training data pipelines for next-generation recommendation systems. Across the recommendation systems industry, storing training data has evolved into a massive, exabyte-scale bottleneck. Recommendation models require exabyte-scale data to achieve high predictive accuracy, making storage scalability a primary operational priority across Meta's infrastructure.

Over recent years, Meta heavily funded and developed storage infrastructure to stay ahead of this rapid growth. Prior optimizations focused on four primary techniques: columnar compression, row reordering, column reordering, and model-aware placement. By actively exploiting similarities across rows and columns alongside observed access patterns, engineering teams compressed data much tighter and minimized unnecessary physical copies. These techniques performed remarkably well for classic recommendation workloads consisting primarily of standard dense and sparse features—until an entirely new modeling paradigm broke the existing assumptions.

- **Primary classical optimization levers**: Columnar compression, row reordering, column reordering, and model-aware physical placement. - **Goal of classical techniques**: Exploit data similarities and common model access patterns to minimize redundant physical data copies.

---

[01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) The Rise of User Sequences and the Exabyte Storage Crisis

The new data paradigm that "changed everything" is the adoption of long user sequences. Rather than relying on static, summarized snapshots, user sequences capture a user's raw chronological interaction history spanning anywhere from several months to years. This provides an information-dense representation that bundles detailed user behaviors, deep content-understanding data, and fine-grained contextual signals. Because of this extreme information density, user sequences are 100x larger (or more) than traditional sparse and dense features.

Over the past three years, Meta pivoted its recommendation modeling to rely heavily on user sequences for both digital advertising and organic content recommendations. Meta published multiple papers demonstrating that sequence length is directly correlated with model quality: the longer the user sequence history provided, the better the recommendation quality became.

```mermaid flowchart TD accTitle: Transition from Traditional Features to Sequence Data accDescr: Diagram illustrating how the adoption of 100x larger user sequences drove training storage into tens of exabytes across data centers.

subgraph Traditional["Traditional Modeling"] A["Dense & Sparse Features\n(Condensed Summaries)"] --> B["Warehouse Relational Tables\n(Self-contained Rows)"] end

subgraph ParadigmShift["User Sequence Paradigm"] C["Months-to-Years of User History\n(100x+ Feature Size)"] --> D["Storage Explosion\nTens of Exabytes"] end

Traditional -.->|"Shift to Sequences (2023+)"| ParadigmShift D --> E["Capacity Saturation\n(10+ DCs, 10% Power Budget)"] E --> F["Aggressive Downsampling &\nLost Seasonal Context"] ```

However, this modeling success created a severe storage crisis. By 2023, user sequences became the single dominant driver of training storage growth, overwhelming previous columnar compression and data placement efficiencies. Training data expanded to tens of exabytes across more than 10 global data centers, consuming an astonishing 10% of Meta's total power budget. Because of hard capacity limits in the data centers, teams were forced into aggressive workarounds: - Engineers had to continuously downsample incoming logs, discarding valuable training samples upfront before they could ever be used. - Data retention windows in many pipelines shrank to just 30 days, completely wiping out long-term historical learning such as annual seasonality patterns. - Experimenting with newer data formats became excruciatingly slow, requiring expensive and time-consuming generation of massive raw data volumes.

Recognizing that the old approaches were no longer viable, the team set out to re-architect training data generation from first principles, focusing on eliminating duplication through normalization.

---

[04:06](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=246s) Analyzing Training Rows and Root Causes of Data Duplication

Sarang Masti takes over to break down the mechanics of the existing AI training stack and pinpoint why user sequences caused storage footprints to balloon. In Meta's legacy architecture, a "training row" is logged during online inference, capturing an exact, self-contained snapshot of all ML features evaluated at recommendation time, combined with an outcome label indicating whether the user interacted with the recommendation. Collections of these rows over specific time windows make up a "training table." During training, a preprocessing worker reads these rows, reconstructs the training samples, and streams them into GPUs.

This legacy model favored fully self-contained rows where every feature was snapshotted inline. The design had clear historical advantages: sample reconstruction was trivial and fast, as the preprocessor only needed to perform a single, sequential read of a row with zero external lookups.

When features consisted of tiny, highly condensed summaries, this trade-off made sense. But analyzing training tables across the company revealed massive data duplication. Because a user's behavior history changes very little—or not at all—between two recommendation requests issued minutes or hours apart, subsequent training rows for the same user repeatedly logged identical feature states. Once features shifted to massive user sequences that were 100x larger, duplicating that history inline on every single interaction quickly broke the economics of the warehouse.

---

[06:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=382s) Sequence Storage: Normalizing Training Tables via ZippyDB

To solve the duplication explosion, Meta adapted classical relational database normalization. Instead of storing massive user sequences inline within every self-contained training row, the data is split across two separate systems:

1. **Normalized Training Table**: The primary training rows remain in the data warehouse on cost-effective, warm hard-disk drive (HDD) storage. However, the sequence features are stripped out and replaced with compact, lightweight pointer keys referencing unique user sequence IDs. 2. **User Sequence Table**: The large, raw user interaction sequences are extracted and written to a dedicated, high-performance indexable key-value store, keyed by user sequence ID.

```mermaid flowchart LR accTitle: Sequence Storage Normalized Architecture accDescr: Architecture showing training rows on HDD warehouse referencing sequences in ZippyDB, joined by a training preprocessor for GPU ingestion.

subgraph Storage["Normalized Storage Layer"] TT["Training Table\n(Warm HDD Warehouse)\nRows with Lightweight Keys"] ST["User Sequence Table\n(ZippyDB Key-Value Store)\nKey: Sequence ID | Value: Sequence"] end

subgraph Compute["Training Pipeline"] TP["Training Preprocessor\n(Connector Join Engine)"] GPU["AI Training GPUs"] end

For the indexed storage layer, Meta leveraged ZippyDB, its internal distributed key-value store, designed to handle high-volume point lookups. While this decoupled architecture immediately slashed duplicate bytes, it introduced a new systems bottleneck: reconstruction complexity. Instead of simple streaming scans, the training preprocessor must now perform real-time joins, reading rows from the warehouse and firing off point lookups into ZippyDB to assemble training batches on the fly. Normalization is only economical if this join overhead does not introduce runaway server costs or GPU starvation.

---

[08:46](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=526s) Early Inefficiencies and Storage Bottlenecks

The first deployed iteration of Sequence Storage fell short of performance and cost targets. In fact, Masti notes that V1 was so inefficient that the total infrastructure footprint ended up being "as costly or if not more" expensive than the naive approach of storing bloated, duplicated rows in the warehouse.

In hindsight, this friction was predictable: ZippyDB was originally built and tuned for standard OLTP-style application workloads, not high-bandwidth batch AI training ingestion. Similarly, the training preprocessor had never been designed to orchestrate high-concurrency join mechanics at this scale. To make Sequence Storage viable, Meta had to re-engineer both sides of the pipe, focusing heavily on reducing the cost per gigabyte-per-second ($/GB/s) of read throughput delivered by the storage servers.

The team diagnosed two primary server-side bottlenecks that had to be addressed: - **Severe read amplification**: Training access patterns caused storage nodes to read significantly more physical data than the logical payload requested, throttling effective throughput. - **Elevated read latency during index joins**: Latencies incurred while resolving sequence pointers into ZippyDB caused pipeline stalls that left expensive GPU accelerators underutilized.

[10:52](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=652s) Offloaded SSTable Generation and Compaction

Populating the initial key-value storage engine using conventional row-by-row insertions triggered continuous, aggressive background compactions inside RocksDB. These background compactions consumed an enormous share of database server resources, specifically flash write bandwidth and CPU cycles, putting them into direct competition with read-serving workloads. Furthermore, the row-by-row write path produced a deep Log-Structured Merge-tree (LSM tree). Reading a single user sequence required traversing multiple levels of SSTable files from the top level down to the bottom, causing severe read amplification on the underlying flash media. Because substantial resources were tied up servicing writes and each read generated multiple flash I/O operations, overall read throughput on the storage servers was severely constrained.

To eliminate this bottleneck, the team introduced an offloaded SSTable compaction and optimization pass. In ZippyDB, they added an ingestion API capable of directly loading pre-generated SSTables from external systems, completely bypassing the write-ahead logs (WAL), memtables, and the CPU-intensive row-by-row insertion pipeline. They updated offline query engines—specifically Apache Spark and Presto—to natively read and write RocksDB SSTable files. This enabled offline data pipelines to periodically compact, sort, optimize, and index sequence data outside the live ZippyDB serving cluster. Once the final SSTables are generated, they are ingested into ZippyDB to replace older files, flattening the database layout into an LSM structure optimized purely for read performance.

```mermaid flowchart LR accTitle: Offloaded SSTable Generation Pipeline accDescr: Spark and Presto process raw data and produce SSTables that are directly loaded into ZippyDB, bypassing the write-ahead log.

subgraph Offline["Offline Processing Pass"] A["Raw Sequence Data"] --> B["Spark & Presto Pipelines"] B --> C["Compacted & Indexed SSTables"] end

subgraph Serving["ZippyDB Storage Engine"] C -->|"Direct SSTable Ingestion API"| D["Single-Level LSM Tree"] D -->|"~1 Flash I/O per Read"| E["Training Clients"] end ```

- Row-by-row writes caused severe I/O amplification across multiple LSM tree levels, consuming CPU and flash bandwidth needed for read queries. - Bypassing the server-side memtable and WAL via direct SSTable loading freed up server-side write budgeting. - Externalizing SSTable creation allowed Spark and Presto to produce pre-indexed, compact files. - The resulting single-level LSM tree achieved roughly "one IO per read" directly from flash.

---

[13:44](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=824s) RocksDB Parallel Multi-Scan and I/O Coalescing

When looking into read latency, the team found that read scans across user sequences were executing serially. Because RocksDB lacked native multi-scan support, ZippyDB processed batches of range scans one after the other. Under this serial scan model, the latency for querying a batch of user sequences equaled the aggregate sum of every individual scan latency. This elevated latency posed a direct risk of training GPU QPS regressions, as high-performance accelerators frequently stalled while waiting for sequential batch fetches to complete across the network.

To resolve this, the engineers implemented a native multi-scan capability within RocksDB. When a batch of reads arrives through the new multi-scan API, RocksDB inspects the requested ranges to discover adjacent disk blocks and identify I/O coalescing opportunities. Whenever two read requests touch neighboring physical blocks, RocksDB coalesces them into a single continuous scan instead of issuing two distinct operations. Once coalesced ranges are determined, the engine issues those I/O requests in parallel across the underlying media. Consequently, media IOPS dropped significantly, and overall batch latency collapsed from the cumulative sum of all scans to "only decided by the single slowest scan", boosting GPU utilization.

---

[15:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=922s) Pushdown Optimizations: Feature Projection and Length Truncation

A third major efficiency barrier stemmed from data over-fetching. Traditional key-value engines like RocksDB and ZippyDB treat keys and values as opaque byte blobs without any schema awareness. However, a user sequence is not arbitrary data; it is structured as a time series of discrete interaction events (effectively a list of structs), where each event contains numerous individual attributes. In practice, specific recommendation models only care about events within defined historical time windows and only need a small subset of the attributes associated with each event. Without schema awareness in the storage layer, the entire sequence had to be shipped over the network to the GPU training hosts, where the clients performed filtering in user space. This created a dual penalty: high network bandwidth usage and substantial client-side CPU overhead to parse and discard unwanted data.

To eliminate this transfer waste, the team made the ZippyDB query engine schema-aware, enabling it to deserialize user sequence values on demand directly on the storage server. With schema inspection enabled, they implemented two server-side pushdown optimizations: 1. **Feature projection:** Deconstructs the decoded key-value structs and discards unrequested event attributes before building the network response. 2. **Length truncation:** Filters the event time series to retain only the specific time window required for a given training sample.

By pushing attribute projection and sequence truncation down into ZippyDB, the total volume of data transmitted over the network and parsed by the GPU training workers was reduced to roughly one-third of its original footprint.

---

[17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) Production Results and Key Architectural Takeaways

Implementing these storage-layer optimizations produced dramatic efficiency gains across Meta's recommendation infrastructure. Storage server read throughput jumped by roughly 5x, expanding from an initial 200 MB/s to over 1 GB/s per server node, drastically lowering hardware footprint and operational costs. Furthermore, the maximum supported sequence length increased from 2,000 events to 80,000 events—an almost 40x leap—without imposing significant storage cost penalties. As of the presentation, the normalized sequence storage platform operates in production serving more than 30 distinct recommendation models across Meta's product families.

Sarang Masti summarized three high-level lessons learned from redesigning the training storage stack: - **Training data scale is a critical, persistent bottleneck:** Meta has reached the ceiling of affordable relational storage, with high-capacity sequence features threatening to expand data footprints by 5x to 10x. - **Normalization is the only viable path forward:** When features were compact, aggregated summary statistics, storing duplicated snapshots alongside training labels was an acceptable trade-off to optimize sequential I/O. For rich, raw sequences, data duplication is economically non-viable. - **Modern AI/ML storage requires full-stack co-design:** Capturing the benefits of data normalization required end-to-end modifications spanning offline preprocessing (Spark/Presto), internal storage engine mechanics (RocksDB and ZippyDB), and model ingestion pipelines on the GPU clients.

---

[19:28](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1168s) Future Roadmap: Lifetime Sequences and Canonical Feature Storage

Meta's sequence storage roadmap focuses on two primary initiatives. The first is scaling sequence length by another order of magnitude. While the production system reliably handles 80,000 events per user, the team's long-term North Star is a 10x expansion to capture comprehensive "lifetime user history". Pushing into hundreds of thousands of events per user introduces new latency, hardware reliability, and cost hurdles, prompting active exploration into advanced data compression, smarter encodings, and hierarchical storage tiering across different physical media.

The second initiative involves expanding the architecture beyond user sequences into a generalized feature platform. Meta intends to apply the normalized indexing infrastructure built for sequence storage to item features, content metadata, and dense embedding tables. The ultimate vision is establishing a single canonical, normalized storage architecture for all machine learning training data across Meta.

## Notable quotes > [00:05](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=5s) Across the recommendation systems industry, storing training data has evolved into a massive, exabyte-scale bottleneck. — Weiran Liu

> [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) The new data paradigm that "changed everything" is the adoption of long user sequences. — Weiran Liu

> [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) Training data expanded to tens of exabytes across more than 10 global data centers, consuming an astonishing 10% of Meta's total power budget. — Weiran Liu

> [08:46](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=526s) as costly or if not more — Sarang Masti

> [10:52](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=652s) one IO per read — Sarang Masti

> [13:44](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=824s) only decided by the single slowest scan — Sarang Masti

> [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) Training data scale is a critical, persistent bottleneck — Sarang Masti

> [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) Normalization is the only viable path forward — Sarang Masti

> [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) Modern AI/ML storage requires full-stack co-design — Sarang Masti

> [19:28](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1168s) lifetime user history — Sarang Masti

## Mentioned Sarang Masti, Weiran Liu, Meta, ZippyDB, RocksDB, Apache Spark, Presto, @Scale Conference

## Video description Training data for Meta's recommendation systems was entirely stored in Data Warehouse, structured as relational tables where each row captures labels and snapshotted features at the point of recommendation.

New modeling techniques, such as learning from user sequences and multi-modality, has led to a 10-100x increase in feature size, making the training data increasingly cost-prohibitive due to high duplication. The same user's features are stored repeatedly for every recommendation request, with highly popular content features being duplicated potentially over a million times.

We present a co-designed data and infrastructure in order to address the scaling challenge. By moving features out of training samples into a high-performance indexing storage and implementing model access pattern-aware pushdown optimizations, we have achieved a 10x storage cost reduction for the largest feature: long user sequences.

Want to learn more about the @Scale Conference? Visit our website here: atscaleconference.com

[Source](https://www.youtube.com/watch?v=ZyAoUSmbntI)

中文简报

Meta通过规范化AI训练数据化解EB级容量危机将长用户序列从单体训练行中解耦并移至优化后的键值存储，使存储成本降低了十倍，化解了占用Meta全球数据中心10%电力的基础设施危机。为提升推荐质量，Meta从压缩特征摘要转向长用户交互序列，这无意中引发了一场存储危机，导致数据占用量在10多个数据中心急剧膨胀，并消耗了公司10%的电力预算。团队通过对训练数据进行规范化解决了这一问题：在数仓表中保留轻量级指针键，同时将原始序列卸载至ZippyDB。当下游初版方案引入严重的读放大和连接延迟时，Meta在Spark和Presto中实现了卸载式SSTable生成，在RocksDB中添加了并行多重扫描与I/O合并，并将投影和截断下推至存储引擎。这些优化将每台服务器的读取吞吐量提升了五倍，在30多个生产模型中将支持的序列长度从2,000个事件扩展至80,000个事件，并为用户生命周期全序列建模铺平了道路。采用长用户序列使Meta的AI训练数据在10多个数据中心膨胀至数十EB，消耗了公司总电力预算的10%，并迫使团队进行严重的下采样。 (1:21) 为应对在每条推荐行中记录相同用户历史所导致的极端重复，Meta通过在数仓表中存储轻量级指针键并将原始序列移至ZippyDB，对其训练数据进行了规范化。 (6:22) 工程师通过离线SSTable生成绕过写入路径、添加具有I/O合并功能的并行多重扫描，以及下推感知模式的投影与截断，解决了ZippyDB和RocksDB早期严重的吞吐量和延迟瓶颈。 (10:52) 优化后的Sequence Storage系统将每台服务器的读取吞吐量提升了五倍（从200 MB/s提升至1 GB/s以上），并在30多个生产推荐模型中将最大序列长度扩展了40倍（从2,000个事件增加到80,000个事件）。 (17:25)

Reading summary

Meta redesigned its AI recommendation training infrastructure by transitioning from fully self-contained, duplicated data rows to a normalized storage architecture. According to Meta engineers Weiran Liu and Sarang Masti, decoupling massive user sequence data into an optimized key-value store solved an exabyte-scale storage crisis while expanding model capacity fortyfold. Sequence Adoption and the Exabyte Crisis (1:21) Traditional recommendation architectures relied on compact dense and sparse feature summaries, which Meta optimized using columnar compression, data reordering, and model-aware placement. However, recommendation performance improved significantly when Meta shifted to rich user sequences—chronological interaction histories spanning months to years. Because these sequences bundle detailed behavior, content understanding, and contextual signals, they are at least 100 times larger than legacy features. By 2023, sequence data became the primary driver of training storage growth, expanding data footprints to tens of exabytes across more than 10 global data centers and consuming 10% of Meta's total power budget. Hardware limits forced infrastructure teams into severe operational compromises: engineers continuously downsampled incoming training logs and shrank data retention windows to just 30 days, entirely wiping out long-term signals like annual seasonality. Normalizing Pipelines with ZippyDB (6:22) Analyzing training tables revealed that inline snapshotting caused extreme data duplication. In legacy systems, each training row logged during inference captured a self-contained feature state. Because a user's behavioral history changes minimally between requests issued minutes or hours apart, consecutive training rows repeatedly duplicated identical, large sequence features. To solve this, Meta adapted relational database normalization into a system called Sequence Storage. Primary training rows remain in data warehouses on cost-effective hard-disk drives but replace large sequence payloads with lightweight sequence IDs. The full user sequences are offloaded into ZippyDB, Meta's distributed key-value store. However, the initial deployment proved as expensive as storing duplicated rows because ZippyDB was designed for OLTP workloads rather than batch AI ingestion, suffering from severe read amplification and join latencies that risked stalling training GPUs. Resolving Storage Amplification and Latency (10:52) Making Sequence Storage economical required a full-stack redesign of the storage and ingestion pipeline. Populating RocksDB row-by-row created deep LSM trees and high background compaction overhead. Meta eliminated this by modifying offline Apache Spark and Presto pipelines to pre-generate and compact SSTables externally. Ingesting these pre-built SSTables directly into ZippyDB bypassed write-ahead logs and memtables, flattening the database structure to achieve approximately one physical flash I/O per read. To address read latency during batch joins, the team introduced parallel multi-scans and physical I/O coalescing inside RocksDB. Instead of serial scans where latency accumulated additively, coalescing adjacent blocks reduced media operations and capped batch latency to the duration of the single slowest scan. Finally, Meta implemented schema-aware pushdown optimizations in ZippyDB: storage nodes perform attribute projection and length truncation on the server, cutting data transmitted across the network to roughly one-third of its original volume. Production Deployment and Unified Future Architecture (17:25) These optimizations increased storage server read throughput by roughly 5x, growing from 200 MB/s to over 1 GB/s per node. The platform also expanded maximum supported sequence lengths from 2,000 events to 80,000 events without severe cost penalties. In production, the system powers training pipelines for more than 30 distinct recommendation models. Meta's sequence roadmap targets another tenfold expansion to capture full lifetime user histories of hundreds of thousands of events, requiring further research into hierarchical storage tiering and advanced compression. Beyond sequence features, Meta plans to extend this normalized key-value approach to item features, content metadata, and dense embedding tables to establish a single canonical ML feature store. Meta's experience demonstrates that continuing to scale high-capacity AI models depends on database normalization and full-stack systems co-design rather than storing duplicate snapshots. However, scaling user histories by another order of magnitude to capture lifetime behaviors presents unresolved latency, hardware reliability, and cost hurdles that require further tiering across storage media.

Detailed notes — zh

## Background 该视频由 Meta 工程师 Sarang Masti 与 Weiran Liu 于 2026 年 6 月 19 日在 @Scale Conference 官方平台发布并分享。视频主要探讨了 Meta 推荐系统在引入长用户序列与多模态技术后，特征体积暴涨 10 至 100 倍导致传统数仓存储成本失控的背景下，团队如何通过数据规范化（Data Normalization）与协同软硬件架构优化实现 10 倍存储降本的工程实践。 ## TL;DR 面对长用户序列引发的数十 EB 训练数据爆炸和高昂的数据冗余成本，Meta 改变了过去将特征完全内嵌于自包含训练行的模式，转而采用类似关系型数据库的“数据规范化”思想，将长序列解耦存入高性能分布式 KV 存储 ZippyDB 中。针对初始方案遭遇的读放大、高读取延迟与数据过度拉取等系统瓶颈，团队通过离线外部生成单层 SSTable 绕过 Compaction、在 RocksDB 层面实现并发多段扫描（Multi-Scan）与 I/O 合并，以及在存储服务端实施特征投影与时间窗口截断的计算下推优化。最终，单台存储服务器读吞吐提升约 5 倍（突破 1 GB/s），所支持的用户序列长度跃升至 8 万个事件，强力支撑了全公司 30 多个主流推荐模型。 ## Atmosphere & style 整场演讲呈现出典型的大厂硬核工程分享风格，严谨务实且条理极其清晰。两位主讲人在宏观数据瓶颈与微观存储内核调优之间流畅交接，既有剖析早期版本在生产环境“性能滑铁卢”时的坦诚与复盘，也有攻克 RocksDB 底层 I/O 放大时的专注与自信。聆听过程充满技术张力，生动展现了超大规模分布式系统在遭遇算力与物理存储极限时，工程师如何通过跨层次的全栈协同创新杀出重围。 ## Bulletin - [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) Meta 推荐模型引入的长用户序列使特征体积膨胀 100 倍以上，导致训练数据规模达到数十 EB 并消耗公司 10% 的数据中心电力预算，迫使工程团队放弃完全自包含行的传统冗余存储模式。 - [06:20](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=380s) 团队借鉴关系数据库设计推行训练数据规范化，将长用户序列解耦至基于 ZippyDB 的高性能键值存储，在数仓训练表中仅保留轻量级指针并在训练预处理阶段执行运行时索引联结（Index Join）。 - [12:23](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=743s) 存储层通过 Spark/Presto 外部化离线构建单层 SSTable 绕过 LSM-tree 压缩与写日志，结合 RocksDB 原生并发 Multi-Scan 和模式感知计算下推，成功将单机读取吞吐量提升 5 倍至 1 GB/s 以上，网络传输体积缩减约三分之二。 - [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 该规范化存储系统已在生产环境支持 Meta 内部 30 多个核心推荐模型，支持的最大序列长度从 2,000 个事件扩展至 80,000 个事件，未来目标是再扩 10 倍以支持用户全生命周期历史建模。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=5s) 传统的列式压缩、行/列重排以及感知模型放置等优化手段，已无法应对基于长用户序列的新型推荐模型所带来的爆炸性数据增长。 - [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) 用户序列记录了数月乃至数年的原始交互历史，虽然显著提升了广告与自然内容的推荐质量，但也带来了超过传统特征 100 倍的数据密度膨胀。 - [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) 数十 EB 规模的训练数据导致严重的系统瓶颈，包括被迫对样本进行降采样、数据保留窗口压缩至 1 个月从而丢失季节性信号，以及特征探索迭代成本过高。 - [04:06](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=246s) 旧架构采用完全自包含的训练行快照，导致相邻请求间长序列特征被海量重复记录，热门内容特征甚至被复制上百万次。 - [06:20](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=380s) 数据规范化方案将长序列特征剥离至独立的 ZippyDB 序列表，原数仓训练表仅保留序列 ID 指针，通过训练预处理器在运行时执行分布式 Index Join 重建样本。 - [08:46](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=526s) 初版规范化架构面临严重性能滑铁卢，OLTP 设计的 ZippyDB 无法直接应对高吞吐 AI 训练，存在严重的读放大、高读取延迟以及存储吞吐损耗三大瓶颈。 - [10:52](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=652s) 传统的逐行写入方式在 RocksDB 中频繁触发 Compaction，不仅大量侵占 CPU 和闪存带宽，还造成多层 SSTable 遍历的严重读放大。 - [12:23](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=743s) 利用分布式大数据引擎在外部完成 SSTable 的构建、压缩与优化，并通过 Direct-Load API 绕过 WAL/MemTable 直存导入，将物理读取开销降至近乎单次 I/O 命中。 - [13:44](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=824s) 在 RocksDB 层面引入原生并发 Multi-Scan API 和相邻 I/O 合并机制，将批处理扫描从线性排队累加转为受限于单次最慢扫描的并发读取，显著缓解了 GPU 供给停顿。 - [15:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=922s) 为 ZippyDB 赋予模式感知能力，在存储服务端直接执行特征投影与时间窗口截断，避免将不透明二进制大对象全量回传，使跨网络传输体积减少约三分之二。 - [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 优化后的 Sequence Storage 将单台存储服务器读吞吐从 200 MB/s 提升至 1+ GB/s，以可控成本将支持的序列长度上限从 2,000 跃升至 80,000 个事件。 - [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 应对新型 AI/ML 工作负载不能局限于单点优化，必须在客户端、分布式计算引擎、底层存储与查询协议间实施端到端的全栈式创新。 - [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 系统未来的演进重点包括通过智能编码和分层存储将序列容量再扩 10 倍以捕获用户全生命周期历史，以及将规范化存储框架通用化推广至物品与内容特征等领域。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=5s) 推荐系统训练数据存储的扩展挑战与传统优化对于推荐系统而言，训练数据的存储正在演变成一个全行业面临的严峻挑战。推荐模型极度渴求 EB 级（Exabyte scale）的海量数据来做出更精准的预测，因此，扩展训练存储系统已经成为各大推荐系统团队的首要任务。过去数年间，Meta 在训练存储领域投入巨资，构建了一系列关键优化技术以跟上数据规模的爆炸式增长。这些传统优化工作主要可以归纳为四大类：列式压缩（columnar compression）、行重排（row reordering）、列重排（column reordering）以及感知模型的放置策略（model-aware placement）。工程团队充分挖掘数据间的相似性以及通用的访问模式，从而实现更高的数据压缩比，并最大限度地减少物理数据副本。在新的数据范式出现之前，这些优化方案在应对包含稠密特征（dense features）和稀疏特征（sparse features）的传统训练数据时表现得非常出色。 - 推荐模型依赖 EB 级训练数据提升预测精度，存储扩展成为大模型推荐系统的当务之急。 - 传统存储优化的四大核心支柱：列式压缩、行重排、列重排与模型感知放置。 - 核心思路是利用数据相似性与访问模式压缩数据、减少物理副本，完全适配传统的稠密与稀疏特征。 ### [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) 新数据范式：用户序列带来的百倍膨胀与基础设施危机然而，一种全新的数据范式彻底改变了局面——它就是“用户序列”（User Sequences）。所谓用户序列，记录的是用户在数月乃至数年间的原始交互历史，其中编码了极其丰富的用户行为、内容理解数据以及上下文环境信息。由于数据密度极高，用户序列的体积比传统特征大了 100 倍甚至更多。在模型演进方面，过去三年里 Meta 全面转向基于用户序列的建模，将其同时应用于广告推荐与自然内容推荐系统，并在业界发表了多篇论文，展示了该技术对底层 AI 模型的巨大提振效果。数据显示，随着用户序列长度的持续增加，模型质量呈现出稳定且显著的提升。但在数据基础设施端，体积暴涨 100 倍的用户序列自 2023 年起变成了推动训练数据存储增长的唯一主导因素。即便此前部署了所有的存储优化手段，数据增长曲线依然在急剧加速： - 训练数据总量已攀升至数十个 EB（tens of exabytes）。 - 数据分散存储在 10 多个数据中心内。 - 训练存储消耗了 Meta 整体数据中心电力预算的 10%。除了高昂的财务与能耗成本，EB 级的数据规模还引发了一系列连锁技术瓶颈： - **被迫降采样**：受限于数据中心容量硬指标，团队不得不对样本进行持续的降采样，从一开始就忍痛丢弃掉大量极具价值的训练样本。 - **历史留存受限**：由于容量告急，多数场景下训练样本仅能保留 1 个月，导致季节性规律（seasonality）等宝贵的长周期知识彻底丢失。 - **研发迭代缓慢且昂贵**：在探索新型数据格式时，工程师需要重新处理和生成极其庞大的数据量，导致实验过程耗时冗长且成本高企。传统的特征优化方案已经无法维系系统运转，团队必须从最底层重新审视训练数据的生成与组织机制。 ### [04:06](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=246s) 数据重叠与反规范化困局：自包含行带来的副本爆炸在深入具体解决方案之前，Sarang Masti 梳理了核心概念以及引出该设计的关键观察。在推荐系统工作流中： - **训练行（Training Row）**：在推理时记录的一条训练样本，完整截取了推理瞬间所使用的所有机器学习特征快照，并附带指示预测表现优劣的标签。 - **训练表（Training Table）**：在特定时间窗口内记录的所有训练行的集合。 - **训练预处理器（Training Preprocessor）**：负责读取这些训练行，重建训练样本，并在训练阶段源源不断喂给 GPU。 Meta 既有的训练数据模型完全依赖于“完全自包含的行”（fully self-contained rows）。也就是说，每一行训练数据都自带推理时用到的全套特征完整快照。这种设计的初衷非常直观：在训练时重建样本极其轻量，预处理器只需读取单条训练行即可，没有任何外部依赖。然而，这种设计隐藏着致命缺陷——极其严重的数据重复（data duplication）。跨所有训练表的分析表明，时间上相近的多个训练行之间，机器学习特征的变化微乎其微，甚至在绝大多数情况下完全一致。在过去特征体积很小、仅包含高度浓缩的用户行为摘要时，这种自包含的设计权衡是完全合理的。但当单行体积因用户序列暴增 100 倍之后，相同用户的长序列特征在每一个推荐请求中被反复记录，热门内容特征甚至被重复复制上百万次，导致旧的数据模型彻底崩溃。 ```mermaid flowchart LR accTitle: Legacy Self-Contained Row Duplication accDescr: Diagram showing how legacy training rows duplicated the entire massive user sequence across multiple request rows. subgraph LegacyTable["原有训练表 (自包含行)"] R1["训练行 1 (请求 A) [用户序列快照 (大)] + 标签"] R2["训练行 2 (请求 B) [用户序列快照 (完全相同)] + 标签"] R3["训练行 3 (请求 C) [用户序列快照 (完全相同)] + 标签"] end LegacyTable -->|"直接读取 (无 Join)"| GPU["GPU 训练"] ``` ### [06:20](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=380s) 核心方案：数据规范化与序列存储系统架构针对该问题，工程团队提出的核心方案是“训练数据规范化”（Normalizing Training Data）。这一灵感直接来源于关系型数据库领域沉淀成熟的规范化范式。高层设计的逻辑十分清晰：不再将所有训练特征扁平地内联塞进单张大表，而是将数据拆分为两张表： 1. **用户序列拆离**：原本在每个训练行中重复存储的用户序列被剥离出去，存入独立的“用户序列表”（User Sequence Table），每个序列由唯一的用户序列 ID（User Sequence ID）寻址。 2. **轻量指针引用**：原始的训练表不再内嵌庞大的序列特征，转而仅保存一个指向序列表中对应记录的轻量级指针（lightweight pointer）。这种解耦彻底消除了重复副本，但同时也带来了工程架构上的经典权衡——大幅增加了训练样本重建的复杂度。训练预处理器现在不能再做简单的单表顺序读取，而是必须在训练运行时，将训练表的行与用户序列表进行联结（Join），动态拼装出完整的训练样本。只有让这个 Join 操作变得足够低开销，整个架构在总体拥有成本（TCO）上才具备可行性。为此，团队打造了名为“序列存储”（Sequence Storage）的专用基础设施架构： ```mermaid flowchart TD accTitle: Sequence Storage Architecture Overview accDescr: Architectural diagram of the normalized Sequence Storage system separating warehouse training tables and indexable ZippyDB sequence tables. subgraph StorageLayer["存储层"] TT["训练表 (Training Table) - 仅含标签与序列 ID 指针 - 底层：数仓 HDD 温存储"] UST["用户序列表 (User Sequence Table) - 存放原始长用户序列 - 底层：ZippyDB 分布式 KV (索引点查)"] end subgraph ComputeLayer["计算与训练层"] TP["训练预处理器 (Training Preprocessor) (专用连接器执行运行时 Index Join)"] GPU["GPU 集群 (模型训练)"] end TT -->|"流式读取指针"| TP UST -->|"点查获取序列数据"| TP TP -->|"组装完整训练样本"| GPU ``` - **训练表存储**：包含训练行及轻量级外键引用的训练表继续保留在数据仓库（Data Warehouse）中，由成本较低的机械硬盘（HDD）温存储支撑。 - **序列存储选型**：由于序列访问模式转变成了极高频的点查（point lookups），用户序列表被迁移至高性能的可索引存储系统。团队选择了 Meta 内部广泛使用的高性能分布式键值存储系统 ZippyDB。 - **预处理连接器**：在训练预处理器侧开发专用连接器，在训练阶段实时跨数仓和 ZippyDB 执行分布式 Join。 ### [08:46](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=526s) 初版架构的性能滑铁卢及三大存储端瓶颈在完成序列存储的第一版构建并接入生产训练管线后，现实却泼了一盆冷水：系统的低效程度远远超出了团队的预期。其整体综合运行成本甚至与此前“单表全量内联存储”的旧方案持平甚至更高。事后复盘发现这一结果在情理之中：ZippyDB 原本是为在线事务处理（OLTP）类的工作负载设计的，此前从未承载过吞吐量极大的 AI 训练场景；与此同时，训练预处理器侧引入的运行时样本动态拼装机制也极为复杂。为了让序列存储真正具备成本效益并能够在生产落地，团队必须同时在预处理器侧和存储服务端实施大规模专项优化。在存储服务端，团队面临着严苛的吞吐量考核目标——必须大幅降低存储服务器提供单位吞吐量（每 GB/s 训练吞吐）的硬件成本。经过性能剖析，团队锁定了三大导致存储服务端性能崩溃的核心瓶颈： - **读放大严重（High Read Amplification）**：存储服务器在应对 AI 训练负载时表现出反常的巨大读放大，直接挤占了服务器的有效 I/O 吞吐上限。 - **索引联结引发的高读延迟（High Read Latency due to Index Join）**：跨系统执行 Index Join 带来了显著的访问延迟，导致整个数据供给管线发生频繁停顿（stalls），进而严重拉低昂贵 GPU 的算力利用率。 - **系统层面的吞吐损耗**：底层存储引擎的内部交互机制未针对大规模流式点查进行调优，阻碍了吞吐潜力的充分释放。 ### [10:52](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=652s) 初始方案的瓶颈：逐行写入引发的读写资源争抢与读放大在演讲的第一部分末尾，Sarang 提出了数据过度拉取（over-fetching）对客户端成本造成压力的挑战，并将话筒交接给 Weiran，由其深入剖析具体的底层存储优化。Weiran 回顾了系统最早的数据填充策略：团队最初采用逐行（row-by-row）的方式向底层存储灌入数据。然而，这种朴素的写入方式在 RocksDB 中引发了极其严重的连锁反应——频繁触发后台数据压缩（compaction）。Compaction 过程疯狂消耗数据库服务器的闪存带宽（flash bandwidth）以及 CPU 算力，直接与在线服务用户的“读操作”（read serving）展开激烈的资源争抢。 ```mermaid flowchart TD accTitle: Row-by-Row Ingestion vs Offline SSTable Generation accDescr: Comparison of the original row-by-row write path causing multi-level SSTables and high compaction against the new pipeline that generates single-level SSTables externally. subgraph OldPath["旧方案：逐行写入路径"] direction TB A1["行级数据写入"] --> B1["MemTable & WAL"] B1 --> C1["RocksDB 频繁 Compaction"] C1 --> D1["深层 LSM 树 (跨多层 SSTable)"] D1 -.->|"多次闪存 I/O 读放大"| E1["高延迟读取"] end subgraph NewPath["新方案：外部 SSTable 离线生成与直存"] direction TB A2["Spark / Presto 离线流水线"] --> B2["外部执行 Compaction、优化与索引"] B2 --> C2["生成最终紧凑 SSTable"] C2 -->|"Direct-Load API (绕过 WAL / MemTable)"| D2["ZippyDB 节点 (原子替换)"] D2 -->|"单文件命中：~1 I/O per Read"| E2["极速读取"] end ``` 除了争夺系统资源，持续的行级写入还导致生成的 LSM-tree（Log-Structured Merge-tree）树高过深。在查询用户的行为序列（user sequence）时，单次读操作必须从树的最顶层一直遍历到最底层，检索跨越多个不同层级的 SSTable 文件。这种跨多层级文件的扫描在底层物理存储介质上造成了显著的“读放大”（read amplification）。这两大负面因素叠加在一起，极度压制了存储服务器的整体读取吞吐量：一方面服务器被迫划拨大笔资源应对写操作与压缩，另一方面每次读取请求在闪存上消耗的 I/O 次数被成倍放大。 - 写入方式：早期采用逐行写入（row-by-row insertions），严重依赖 MemTable 和 WAL。 - 瓶颈根源：后台 Compaction 大量挤占 CPU 与闪存 I/O，且深层 LSM 树引发多层 SSTable 遍历。 - 核心后果：存储读取吞吐量受限，闪存 I/O 放大严重，难以支撑大规模 AI 训练的高并发需求。 --- ### [12:23](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=743s) 架构重构：外部化 SSTable 生成与单层极速读取为了彻底根除 Compaction 带来的开销与读放大，团队跳出原有模式，引入了一套“离线/卸载式 SSTable 压缩与优化流水线”（offloaded SSTable compaction and optimization pass）。在底层 KV 存储系统 ZippyDB 中，团队新增了一套 Direct-Load API，允许服务器直接从外部导入预先构建好的 SSTable 文件。这一操作彻底绕过了昂贵的行级写入链路，既不需要写入预写式日志（Write-Ahead Log, WAL），也完全无需经过 MemTable 的缓冲与落盘。与此同时，团队改造了大数据计算引擎（如 Spark 和 Presto），赋予它们原生识别与输出 SSTable 文件的能力。借助这些分布式计算引擎，Meta 构建了独立的数据流水线，在 ZippyDB 集群之外离线完成数据的定期压缩、布局优化以及辅助索引构建。所有优化流程结束后，流水线输出最终形态的 SSTable 文件，并将其直接载入 ZippyDB 节点以替换老旧文件。通过这种外部化构建，存储端最终呈现的是一个极致扁平、专为读取优化的单层 LSM 树结构。从此，针对用户序列的每一次点查或扫描仅需访问单一 SSTable 文件，近乎完美地达成了“单次读取仅消耗一次物理 I/O”（one I/O per read）的性能突破。 - 旁路加载机制：引入 Direct-Load API，跳过 MemTable 与 WAL，实现无锁、无 Compaction 负担的极速数据加载与原子替换。 - 外部批处理协同：利用 Spark 和 Presto 在外部执行重量级的排序、压缩与构建，解耦训练存储与数据清洗算力。 - 存储结构优化：将深层 LSM 树扁平化，实现单 SSTable 命中，读取操作的 I/O 次数直接压降至每次读取 ~1 次 I/O。 --- ### [13:44](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=824s) 突破吞吐瓶颈：RocksDB 原生并发多段扫描（Multi-Scan）解决了读放大之后，团队面临的第二个棘手挑战是读取延迟过高。经过深度排查，他们发现底层的范围扫描（scans）竟然全是以单线程串行方式执行的。受限于 RocksDB 原生并不支持批量的并发扫描，上层的 ZippyDB 只能无奈地对批请求实施串行扫描。当训练任务发起一批用户序列的查询时，端到端延迟实际上是所有单个扫描耗时的线性累加。这种延迟积聚很快在训练端演变为严重的 GPU QPS 下滑风险，因为 GPU 集群由于等待特征数据拉取而陷入空转等待。 ```mermaid flowchart LR accTitle: RocksDB Native Multi-Scan Mechanism accDescr: Parallel scan and I/O coalescing workflow inside RocksDB's new multi-scan API. A["批量扫描请求 (Batch Reads)"] --> B["Multi-Scan API (RocksDB)"] subgraph Engine["RocksDB 引擎内部优化"] direction TB B --> C{"分析相邻扫描范围"} C -->|"命中相邻数据块"| D["I/O 合并 (Coalescing) 2 次合并为 1 次连续 I/O"] C -->|"分散数据块"| E["并行下发 I/O"] D --> E end E --> F["并发读取闪存介质"] F --> G["整体延迟受限于单个最慢扫描 (非线性累加)"] ``` 为此，团队在 RocksDB 层面直接研发并实现了“原生多段扫描”（native multi-scan）能力。当存储节点通过全新的 Multi-Scan API 收到一批读取请求时，RocksDB 引擎首先会审查这些目标范围，探测相邻扫描区间是否存在“I/O 合并”（I/O coalescing）的机会——如果两次扫描请求命中相邻的磁盘数据块，系统会将其合并为一次连续的底层 I/O 操作，而不再发起两次独立的离散 I/O。在完成范围合并后，引擎会并发向存储介质下发所有 I/O 请求。这一改进大幅削减了闪存介质的实际 IOPS 消耗，并将串行累加的长尾延迟彻底打平：多段查询的总延迟不再由总和决定，而仅取决于“多个并发扫描中最慢的那单次扫描”（single slowest scan），有效遏制了 GPU 空转，显著提升了训练阶段的 GPU QPS。 - 根本症结：原有 RocksDB/ZippyDB 缺乏批量多段扫描支持，扫描请求只能串行累加，成为 GPU 供给瓶颈。 - I/O 合并机制：自动识别相邻访问区间，将碎片化的细粒度 I/O 合并为单次大块读取，压低硬件 IOPS 开销。 - 并发检索：以并发异步下发取代线性排队，总耗时从 $\sum \text{Latency}$ 骤降为 $\max(\text{Latency})$。 --- ### [15:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=922s) 计算下推与模式感知：裁剪无用属性，消除过度拉取第三个核心痛点在于数据过度拉取（data over-fetching）。在原生的 RocksDB 和 ZippyDB 设计中，Key 与 Value 均被视作透明且不透明的二进制大对象（opaque blobs），底层存储对其内部结构完全无感知。然而，推荐模型中的“用户序列”（user sequences）本质上是结构化的时间序列事件列表（list of structs），每一个独立事件都携带着丰富的属性字段。但在实际训练中，绝大多数推荐模型只关心特定时间窗口内的事件，且通常只需要该事件下的一小部分特征属性，而非整包数据。在缺乏模式感知能力的情况下，旧系统只能将完整的用户序列从存储端通过网络全部回传给训练端，再在 GPU 侧的宿主机上执行无用数据的过滤与清洗。这种粗暴的拉取不仅极大地浪费了跨机网络带宽，还对 GPU 宿主机的 CPU 造成了沉重的解码与过滤开销，推高了整体流水线延迟。为了解决该问题，团队为 ZippyDB 查询引擎注入了模式感知（schematized data recognition）能力，使其能够根据预定义的 Schema 在存储服务端按需反序列化 Key-Value 数据。在此基础上，存储端落地了两项核心下推能力： 1. **特征投影（Feature Projection）**：在服务端解码键值后，仅保留模型真正订阅的事件属性，提前滤除无关字段； 2. **长度截断（Length Truncation）**：严格根据当前训练样本所关心的特定时间区间，就地剔除窗口之外的过期事件。这两项下推优化双管齐下，使从 ZippyDB 节点传输到训练端的数据体积直接缩减为原始大小的约三分之一（1/3），极大解放了网络带宽与 GPU 端的算力负担。 - 结构化认知：打破 Key-Value 的不透明黑盒形态，支持存储端直接解析用户行为事件的结构体字段。 - 算子下推：通过就地执行“特征投影”（过滤列）与“时间窗口截断”（过滤行），避免无效数据穿透到训练网络。 - 压降收益：回传数据量直接骤降约 66%（仅保留原先的 1/3），有效降低训练网络流量，彻底免除了 GPU 侧繁重的过滤消耗。 --- ### [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 落地成效、核心认知与面向“全生命周期历史”的未来演进随着上述一系列存储端攻关的全面投产，Sarang 接过话筒进行成效总结。他指出，整体存储服务器的端到端效率提升了约 5 倍：单台服务器所能支撑的读取吞吐量从项目初期的 200 MB/s 一路飙升至超过 1 GB/s，极大地缩减了基础设施搭建与维护成本。不仅如此，系统支持的最大用户序列长度从最初的 2,000 个事件直接扩展到 80,000 个事件（实现了近 40 倍的跨越式跃升），而系统总成本几乎没有出现显著膨胀。目前，这套序列存储系统（Sequence Storage）已在 Meta 生产环境中全量运行，强力支撑着跨越多个核心业务部门的 30 多个主流推荐模型。 ```mermaid flowchart TD accTitle: Sequence Storage Future Roadmap accDescr: The two core investment areas for future scaling: 10x sequence length and generalization to other normalized features. A["Sequence Storage 演进蓝图"] --> B["方向一：长序列能力再扩 10 倍"] A --> C["方向二：泛化为全 Meta 标准归一化存储"] subgraph Scaling["10x 序列容量扩展"] direction TB B --> B1["目标：从 80K 扩展至 ~800K 事件 (捕获用户全生命周期行为)"] B --> B2["挑战：超长尾数据带来的成本、可用性与延迟压力"] B --> B3["探索技术：更精细的编码压缩 (Smart Encoding) 与数据分层 (Tiering)"] end subgraph Generalization["标准化与通用化"] direction TB C --> C1["跳出单一用户行为序列，覆盖物料特征 (Item Features)"] C --> C2["支持内容级特征 (Content Features) 与各类高维 Embeddings"] C --> C3["终极愿景：成为 Meta 内部标准化统一归一化训练存储系统"] end ``` 紧接着，Sarang 总结了本次架构重构带来的三大核心启示（Takeaways）： 1. **训练数据规模是不可回避的严峻考验**：存储容量在成本上已达极限，而用户序列等新兴负载正试图将数据量进一步推高 5 到 10 倍，低成本演进是唯一出路。 2. **数据归一化（Normalization）是必然方向**：早期机器学习特征是高度压缩的用户画像摘要，可以通过数据冗余来换取 I/O 效率；但面对超长序列特征，冗余存储在成本上完全不可行，数据归一化已成为唯一的解法。 3. **应对 AI/ML 新型工作负载需要全栈式创新**：单纯优化某个单点无法奏效，团队必须对涵盖客户端、存储引擎、查询协议在内的整个 AI 训练技术栈进行颠覆性改造。最后，Sarang 展望了序列存储未来的两大核心投资方向： - **突破序列长度极限（10x Scale）**：团队当前的“北极星指标”（North Star）是将现有的 80K 事件上限再扩大 10 倍，从而完整捕获用户的全生命周期历史行为（lifetime user history）。面对随之而来的成本、可用性和时延挑战，团队正积极探索更先进的智能编码算法与冷热存储分层（tiering）架构。 - **全场景通用化拓展（Generalization）**：打破“仅服务用户序列”的边界，将这套高吞吐高索引的技术底座推广到 Item 特征、内容特征（Content features）以及高维 Embeddings 等其他特征类型中，最终将其打造为“Meta 内部所有归一化训练数据的统一标准存储底座（canonical storage）”。 - 吞吐翻倍：单机读吞吐由 200 MB/s 提升至 1+ GB/s（5x 效率提升）。 - 容量跃升：支持的序列长度由 2K 暴增至 80K 事件（近 40x 长度），支持全公司 30+ 核心模型。 - 未来蓝图：探索智能编码与分层存储以支撑 800K 全生命周期序列，并将架构泛化至全局通用 Normalized 训练存储平台。 ## Notable quotes > [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) 训练存储消耗了 Meta 整体数据中心电力预算的 10%。 — Sarang Masti > [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) 训练数据总量已攀升至数十个 EB（tens of exabytes）。 — Sarang Masti > [01:21](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=81s) 由于容量告急，多数场景下训练样本仅能保留 1 个月，导致季节性规律（seasonality）等宝贵的长周期知识彻底丢失。 — Sarang Masti > [04:06](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=246s) 相同用户的长序列特征在每一个推荐请求中被反复记录，热门内容特征甚至被重复复制上百万次，导致旧的数据模型彻底崩溃。 — Sarang Masti > [08:46](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=526s) ZippyDB 原本是为在线事务处理（OLTP）类的工作负载设计的，此前从未承载过吞吐量极大的 AI 训练场景。 — Sarang Masti > [12:23](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=743s) 通过这种外部化构建，存储端最终呈现的是一个极致扁平、专为读取优化的单层 LSM 树结构。从此，针对用户序列的每一次点查或扫描仅需访问单一 SSTable 文件，近乎完美地达成了“单次读取仅消耗一次物理 I/O”（one I/O per read）的性能突破。 — Weiran Liu > [13:44](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=824s) 多段查询的总延迟不再由总和决定，而仅取决于“多个并发扫描中最慢的那单次扫描”（single slowest scan），有效遏制了 GPU 空转，显著提升了训练阶段的 GPU QPS。 — Weiran Liu > [15:22](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=922s) 这两项下推优化双管齐下，使从 ZippyDB 节点传输到训练端的数据体积直接缩减为原始大小的约三分之一（1/3），极大解放了网络带宽与 GPU 端的算力负担。 — Weiran Liu > [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 单台服务器所能支撑的读取吞吐量从项目初期的 200 MB/s 一路飙升至超过 1 GB/s，极大地缩减了基础设施搭建与维护成本。 — Sarang Masti > [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 系统支持的最大用户序列长度从最初的 2,000 个事件直接扩展到 80,000 个事件（实现了近 40 倍的跨越式跃升），而系统总成本几乎没有出现显著膨胀。 — Sarang Masti > [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 早期机器学习特征是高度压缩的用户画像摘要，可以通过数据冗余来换取 I/O 效率；但面对超长序列特征，冗余存储在成本上完全不可行，数据归一化已成为唯一的解法。 — Sarang Masti > [17:25](https://www.youtube.com/watch?v=ZyAoUSmbntI&t=1045s) 团队当前的“北极星指标”（North Star）是将现有的 80K 事件上限再扩大 10 倍，从而完整捕获用户的全生命周期历史行为（lifetime user history）。 — Sarang Masti ## Mentioned Meta, Sarang Masti, Weiran Liu, @Scale Conference, ZippyDB, RocksDB, Apache Spark, Presto, GPU, LSM-tree, SSTable, WAL, MemTable ## Video description Training data for Meta's recommendation systems was entirely stored in Data Warehouse, structured as relational tables where each row captures labels and snapshotted features at the point of recommendation. New modeling techniques, such as learning from user sequences and multi-modality, has led to a 10-100x increase in feature size, making the training data increasingly cost-prohibitive due to high duplication. The same user's features are stored repeatedly for every recommendation request, with highly popular content features being duplicated potentially over a million times. We present a co-designed data and infrastructure in order to address the scaling challenge. By moving features out of training samples into a high-performance indexing storage and implementing model access pattern-aware pushdown optimizations, we have achieved a 10x storage cost reduction for the largest feature: long user sequences. Want to learn more about the @Scale Conference? Visit our website here: atscaleconference.com [Source](https://www.youtube.com/watch?v=ZyAoUSmbntI)

Source quotes

[1:21] Sarang Masti: 训练存储消耗了 Meta 整体数据中心电力预算的 10%。 [1:21] Sarang Masti: 训练数据总量已攀升至数十个 EB（tens of exabytes）。 [1:21] Sarang Masti: 由于容量告急，多数场景下训练样本仅能保留 1 个月，导致季节性规律（seasonality）等宝贵的长周期知识彻底丢失。 [4:06] Sarang Masti: 相同用户的长序列特征在每一个推荐请求中被反复记录，热门内容特征甚至被重复复制上百万次，导致旧的数据模型彻底崩溃。 [8:46] Sarang Masti: ZippyDB 原本是为在线事务处理（OLTP）类的工作负载设计的，此前从未承载过吞吐量极大的 AI 训练场景。 [12:23] Weiran Liu: 通过这种外部化构建，存储端最终呈现的是一个极致扁平、专为读取优化的单层 LSM 树结构。从此，针对用户序列的每一次点查或扫描仅需访问单一 SSTable 文件，近乎完美地达成了“单次读取仅消耗一次物理 I/O”（one I/O per read）的性能突破。 [13:44] Weiran Liu: 多段查询的总延迟不再由总和决定，而仅取决于“多个并发扫描中最慢的那单次扫描”（single slowest scan），有效遏制了 GPU 空转，显著提升了训练阶段的 GPU QPS。 [15:22] Weiran Liu: 这两项下推优化双管齐下，使从 ZippyDB 节点传输到训练端的数据体积直接缩减为原始大小的约三分之一（1/3），极大解放了网络带宽与 GPU 端的算力负担。 [17:25] Sarang Masti: 单台服务器所能支撑的读取吞吐量从项目初期的 200 MB/s 一路飙升至超过 1 GB/s，极大地缩减了基础设施搭建与维护成本。 [17:25] Sarang Masti: 系统支持的最大用户序列长度从最初的 2,000 个事件直接扩展到 80,000 个事件（实现了近 40 倍的跨越式跃升），而系统总成本几乎没有出现显著膨胀。 [17:25] Sarang Masti: 早期机器学习特征是高度压缩的用户画像摘要，可以通过数据冗余来换取 I/O 效率；但面对超长序列特征，冗余存储在成本上完全不可行，数据归一化已成为唯一的解法。 [17:25] Sarang Masti: 团队当前的“北极星指标”（North Star）是将现有的 80K 事件上限再扩大 10 倍，从而完整捕获用户的全生命周期历史行为（lifetime user history）。

Source quotes

[0:05] Weiran Liu: Across the recommendation systems industry, storing training data has evolved into a massive, exabyte-scale bottleneck. [1:21] Weiran Liu: The new data paradigm that "changed everything" is the adoption of long user sequences. [1:21] Weiran Liu: Training data expanded to tens of exabytes across more than 10 global data centers, consuming an astonishing 10% of Meta's total power budget. [8:46] Sarang Masti: as costly or if not more [10:52] Sarang Masti: one IO per read [13:44] Sarang Masti: only decided by the single slowest scan [17:25] Sarang Masti: Training data scale is a critical, persistent bottleneck [17:25] Sarang Masti: Normalization is the only viable path forward [17:25] Sarang Masti: Modern AI/ML storage requires full-stack co-design [19:28] Sarang Masti: lifetime user history

Original transcript

[0:05] Hello everyone. [0:06] My name is Weidong Liu. [0:08] With my colleague [0:10] Saran, we will we will walk you through [0:12] the power sequence storage [0:14] which is scared our training data for [0:16] the next generation recommender system [0:19] models. [0:21] For recommender systems, training data [0:22] storage is an industry-wide challenge. [0:25] Models desire exabyte scale data to make [0:29] better predictions. [0:30] And scaling the training storage becomes [0:33] the top priority [0:35] for the big recommender systems as we [0:37] seen in these papers. [0:40] Over the past years, [0:42] Meta invested heavily into training [0:44] storage. We have built critical [0:46] optimizations to keep up with faster [0:49] scale growth. [0:50] Our work can be categorized into four [0:53] categories: [0:54] columnar compression, [0:56] row reordering, column reordering, and [0:59] the model aware placement. [1:01] We exploit data similarities and the [1:03] common access patterns to compress the [1:05] data better and the minimize physical [1:08] data copies. [1:10] This optimization work well for the [1:12] training data with traditional features [1:15] like dense and sparse features [1:17] before a new data paradigm appeared. [1:21] The new data paradigm is user sequence [1:25] and it changed everything. [1:27] So, what is a user sequence? It consists [1:29] of users' raw history of interactions [1:32] from months to years. [1:34] It encodes richer user behaviors, [1:37] content understanding data, and the [1:39] contextual information. [1:41] With high data density, user sequences [1:44] are 100 x or more larger than [1:47] traditional features. [1:49] On the modeling side, Meta pivoted to [1:52] learn from user sequences over the past [1:54] 3 years [1:55] for both advertisement and organic [1:58] content recommendations. [2:00] We published the market for papers that [2:02] reported the effectiveness on boosting [2:06] our 12 AI models. [2:08] We have seen steadily and the [2:10] significant model quality improvements [2:13] as user sequence lens grows to be [2:15] longer. [2:18] Back to the data side, user sequence [2:20] with 100 x larger size than traditional [2:23] features becomes a single dominant [2:25] driving factor of training data storage [2:28] from 2023. [2:30] Even with all the previous storage [2:32] optimization mentioned, the growth [2:35] actually further accelerated. [2:37] Now, training data is of tens [2:40] exabytes stored in more than 10 data [2:42] centers and cost the 10% of meta's power [2:46] budget. [2:48] Moreover, in addition to the high cost, [2:50] this exabyte scale data list many other [2:53] challenges. [2:54] First, due to the data center capacity [2:56] constraints, we are forced to down [2:57] sample all the time and up front throw [3:00] valuable training samples away. [3:02] For the same reason, in many cases, [3:04] we're only able to keep training samples [3:06] for 1 month and the much longer term [3:08] knowledge like seasonality is lost. [3:11] In the end, we're experimenting new data [3:14] formats. We need to process and generate [3:17] huge volume of data, which is both [3:19] time-consuming [3:21] and costly. [3:24] So far, we have discussed that we [3:26] engineered a strong solution for [3:28] traditional training data features. [3:30] However, user sequences change the [3:33] landscape. [3:34] The old solution was no longer [3:36] sufficient. What comes next? [3:39] We need to rethink about how training [3:41] data is generated from the ground up. [3:43] With all that in mind, we will talk [3:45] about four things. [3:47] First is the observation data [3:49] duplication. [3:51] Next is a solution built on top of that. [3:54] After that, we will iterate a few [3:56] critical infra optimizations [3:59] and the future development. [4:01] With that, I will hand over to Sarang [4:04] for the next section. [4:06] >> Thanks for that for the nice [4:07] introduction on the AI training stack [4:09] and setting up the context. [4:11] Hi, I'm Sarang Mashti. Now, I'm here to [4:14] share a bit more about the challenges [4:16] with handling user sequence with our [4:18] current stack and how we tackle these [4:20] challenges. [4:22] But before we go there, I wanted to talk [4:24] about some basic concepts and a key [4:27] observation that led to the solution. [4:30] Let's start with a training row. [4:32] A training row is a sample that logged [4:34] for training at the time of inference [4:36] and captures the complete snapshot of [4:39] the ML feature used during inference [4:41] along with the label that sort of [4:42] indicates how good the prediction was. [4:45] A training table is [4:47] a collection of training rows logged [4:49] over a certain time period. [4:51] A training preprocessor reads these [4:53] training rows, reconstructs the training [4:55] samples, feeds it to the GPU at the time [4:58] of training. [4:59] Our previous data model for training [5:01] relied on fully self-contained rows. [5:03] That is, each training row stored a [5:06] complete snapshot of the features that [5:08] were used at the time of inference. [5:11] This makes it very easy to reconstruct [5:13] the training sample at the time of [5:15] training because all we need to do is [5:17] read the training row and nothing else. [5:19] However, there's a downside with this [5:21] approach, which is data duplication. [5:24] When we analyze the training data across [5:26] all our training tables, we find a lot [5:28] of data duplication. [5:30] The reason this happens is ML features [5:33] between training rows that are fairly [5:35] close to each other in time do not [5:38] change much and in fact in many cases, [5:40] they're exactly the same. [5:42] As a result, there's a lot of data [5:43] duplication. And this design choice was [5:46] intentional in the past when our [5:47] training growth size was small. [5:50] Since the ML features were highly [5:51] condensed summaries capturing user [5:54] behavior and interactions. [5:56] With user sequences though, the nature [5:58] of training data completely changed. [6:00] There was suddenly a desire to store a [6:03] lot more information about the user [6:05] interactions and behavior over a much [6:07] longer time period. Now the training [6:09] growth was suddenly 100x larger. [6:13] As a result, our previous data model was [6:14] no longer scaling and we needed to [6:16] revisit some of these fundamental [6:18] assumptions. So what was our solution [6:20] over here? [6:22] Our solution to this problem is [6:23] normalizing training data. [6:25] And normalization is a very well [6:27] researched pattern in the relational [6:29] database world. [6:31] And it's very relevant here as well. [6:33] The idea on a very high level is pretty [6:36] simple. Rather than storing all of the [6:38] training data in a single table, [6:40] we split the data into two tables. [6:42] Specifically, the user sequences, which [6:45] were duplicated and stored in line in [6:47] each of the training growth previously, [6:49] are now moved to a separate table called [6:51] a user sequence table where each user [6:53] sequence is addressed by a unique user [6:56] sequence ID. The original training table [6:59] containing training growth now no longer [7:02] store these features in line within the [7:04] row, but store a lightweight pointer to [7:06] the rows in the sequence table. [7:08] With this, we can reduce data [7:10] duplication significantly, but however, [7:12] there is a trade-off over here. [7:15] And the trade-off is there is a more [7:17] complex training sample reconstruction. [7:20] For reconstructing training samples now, [7:22] the training preprocessor needs to do [7:24] join reading a training row from the [7:26] training table and the user sequence [7:29] from the user sequence table to [7:31] reconstruct these training samples on [7:32] the fly at the time of training. [7:35] The entire solution becomes cost [7:37] effective only if we are able to [7:38] optimize these joins and we'll look into [7:41] how we optimize these joints in a bit. [7:44] Now, with that background, let's take a [7:46] look at the solution for handling user [7:49] sequences. [7:51] Sequence storage. [7:52] Sequence storage is the system that we [7:54] built for handling user sequences. [7:56] And this is depicted pictorially as [7:58] shown on the slide. [8:00] Like I mentioned before, training data [8:02] is now split into two tables. [8:05] The training table containing the [8:06] training rows with lightweight [8:07] references to keys in the user sequence [8:10] table continues to remain on warehouse [8:12] backed by warm storage HDD. [8:15] The user sequence table is moved to an [8:18] efficient indexable storage [8:20] since the access pattern we expect on [8:22] this table to be point lookups. [8:25] We decided to leverage ZippyDB, which is [8:27] a widely used distributed key-value [8:29] store within Meta [8:31] for this indexable storage. [8:34] On the train preprocessor side, we now [8:36] needed a special connector that does a [8:38] join between the training table and the [8:40] user sequence table to reconstruct these [8:42] training samples at the time of [8:44] training. [8:46] The first version of sequence storage [8:48] that we built [8:50] and integrated into our training [8:52] pipeline turned out to be a lot more [8:53] inefficient than we expected it to be. [8:56] In fact, the overall system [8:57] turned out to be as costly [9:00] or if not more as the original approach [9:03] of storing all of the data in a single [9:04] table. [9:06] In hindsight, this was expected because [9:08] ZippyDB was never used [9:11] for training workloads, and most of the [9:13] optimizations over there were for mostly [9:16] OLTP-ish like workloads. [9:18] On the training preprocessor side as [9:20] well, we had now a more complex [9:23] operation to reconstruct training [9:24] samples. [9:25] Hence, we had to land a lot of [9:27] optimization on both the training [9:29] preprocessor side and the storage server [9:31] side [9:32] to make sequence storage cost-effective [9:34] and practical. [9:36] In this talk, we'll be mostly focusing [9:38] on the optimizations that we landed on [9:41] the storage server side. [9:43] Now, with that background, let's take a [9:44] look at some of the major inefficiencies [9:46] that we [9:48] identified on the storage server side [9:50] for AI training workload and how we [9:53] address these inefficiencies. [9:56] On the storage server side, we were [9:58] looking at a major read throughput [10:00] optimization challenge with a very [10:02] aggressive target. Overall, we wanted to [10:05] reduce the cost per gigabyte per second [10:07] of training throughput that we could get [10:09] from the storage server. And we [10:11] identified three major sources of [10:13] inefficiencies that we had to optimize [10:15] in order to hit our throughput goals. [10:18] On a very high level, these [10:19] inefficiencies were high read [10:21] amplification. We noticed that the [10:23] storage servers had higher read [10:25] amplification for AI training workloads. [10:28] As a result, this limited the effective [10:30] throughput [10:31] that we could get from the storage [10:33] servers and we had to land a lot of [10:35] optimizations to reduce this [10:36] amplification. [10:38] The second one was high read latency due [10:41] to the index join. [10:42] We had to optimize the index join to [10:44] reduce the latency [10:46] and remove any stalls in our data [10:48] pipeline and ensure the GPUs were well [10:51] utilized. [10:52] And finally, there was a problem of data [10:55] over fetching. [10:56] We had to land a few features to reduce [10:59] over fetching of data and improve the [11:01] client side cost. [11:03] Now, I will hand off to Verena to talk [11:05] about some of these optimizations in a [11:07] bit more detail. [11:09] >> Thanks, Sura. [11:10] When we populate the initial storage, we [11:13] used the row by row insertions. [11:16] So, we observed that this triggered the [11:19] frequent background compactions in [11:22] RocksDB. [11:23] And the compaction ate much of the DB DB [11:26] server resources like a flash bandwidth [11:29] and the CPU. [11:30] Such resources usage compete with the [11:34] resources that they used for serving [11:36] read. [11:37] In addition, such insertions resulted [11:40] with a log structured merge tree with [11:42] high depth as shown on the right hand. [11:45] Each read for user sequence needs to [11:47] touch multiple SSTable files from the [11:50] top level [11:52] to the bottom level. [11:53] So, this multi-level log structured tree [11:56] traversal end up with tags read [11:58] amplifications here in average [12:01] on the storage media. [12:03] Both factors combined severely limit the [12:06] storage server read throughput because [12:09] we need to budget a substantial amount [12:11] of resources on serving [12:14] writes [12:15] as well as we much amplify the flash IO [12:19] usage per read. [12:23] For this challenge, we tackle it by [12:25] introducing a new offloaded SSTable [12:29] compaction and optimization pass. [12:32] In the VDDV, we introduced a new API [12:35] that allows direct load pre-generated [12:38] SSTables [12:40] from external [12:41] completely bypassing the expensive [12:44] memtable and the write-ahead logs in the [12:46] server [12:48] from the row-by-row insertion pass. [12:50] Moreover, we make our query engines like [12:53] Spark and Presto [12:56] to recognize SSTable files. [12:59] On the top, we are able to build data [13:02] pipelines to [13:04] periodically compact, optimize, [13:07] and index data outside of the VDDV [13:10] servers. [13:11] Once everything is done, we have our [13:13] final SSTable files. [13:16] We are able to load the newly generated [13:20] files into VDDV servers and the replace [13:23] the older ones. [13:25] This resulted with a single look [13:27] structured the merge tree optimized for [13:30] read. [13:31] Every single read now [13:34] is only served from a single SST table [13:36] file. [13:37] We pretty much achieved one IO per read. [13:44] For the second challenge, high read [13:46] latency, [13:47] we looked closely to be that the scans [13:50] were executed in serial. This was [13:53] because RocksDB doesn't support native [13:56] multiple scans and ZippyDB had to do the [13:59] scans serially. With that [14:01] implementation, when querying for batch [14:04] of user sequences, the total latency was [14:08] the sum of all individual scans. [14:10] This high latency led to training GPU [14:14] QPS regression risks [14:16] since we need to wait longer for [14:19] fetching user sequences. [14:22] Our solution was to implement native [14:25] multi-scan implementation in RocksDB. [14:28] When a batch of reads are received via a [14:31] new multi-scan API, [14:34] RocksDB first taking sight into adjacent [14:38] scan ranges to identify IO coalescing [14:41] opportunities. Meaning like if two IOs [14:44] are hitting the adjacent disk blocks and [14:47] whether we can do two instead of doing [14:50] two scans, we do one scan instead. After [14:53] identifying the coalesced ranges, we [14:55] issue those IOs in parallel against [14:58] these ranges. [15:00] This reduced the storage media IO IOPS [15:04] and improved the query latency. [15:07] Now, the total latency is only decided [15:10] by the single slowest scan from multiple [15:14] scans. [15:15] And therefore, this reduced the latency, [15:18] increased the training GPU QPS. [15:22] Last challenge was data over fetching. [15:26] In traditional RocksDB and GBDV, keys [15:29] and values are treated as opaque blobs [15:32] and not schematized. [15:34] On the other side, user sequences is [15:37] basically a time series of events. Think [15:40] about a list of structs. Each event [15:44] consists of multiple attributes. [15:47] Most of our models only needs to access [15:50] interested events within certain time [15:53] ranges [15:54] and a subset of event attributes instead [15:56] of all the attributes. [15:58] If this knowledge was not leveraged, we [16:01] need to fetch the entire sequence back [16:04] to GPU and then filter the out [16:07] uninterested data on the GPU side. [16:09] This over fetching effect increased the [16:12] network throughput because we [16:14] basically passed more data over [16:17] and they also added the significant CPU [16:20] and the latency tax on the GPU side for [16:23] the filtering. [16:26] To optimize, we made [16:29] the GBDV query engine to recognize [16:33] schematized the data [16:35] so that it can be called serialized the [16:37] user sequence key values on demand. [16:40] On the top, we implement feature [16:43] projection to filter out uninterested [16:46] event attributes [16:47] after decoding the key values. [16:50] In addition, we implement the length [16:52] truncation to keep only the events [16:55] within the specific time ranges that [16:58] matters to specific training samples. [17:01] Both optimizations in combination [17:03] reduced the data to be fetched from GBDV [17:06] to about 1/3 compared to the original [17:09] situation. [17:12] Due to the time constraints, we have [17:14] been brief on the technical stuff. [17:17] We'll post an engineering blog in more [17:19] details. [17:21] Now, I will pass to Saran again to [17:24] conclude this talk. [17:25] >> Thanks, Wei-Ning, for talking about some [17:27] of the optimizations that we landed on [17:29] the storage server side. [17:31] With all of these optimizations landed, [17:33] where did we end up with? [17:35] We ended up with a storage server that [17:37] was roughly 5x more efficient than when [17:39] we started. [17:40] Now, each storage server was able to [17:43] support more than a gigabyte per second [17:45] read throughput up from 200 megabytes [17:46] per second when we started. And this [17:49] significantly reduced the cost of the [17:51] overall setup. We were also able to [17:53] scale the user sequence length from 2K [17:55] events to 80K events, which is almost a [17:57] 40x improvement without significant [18:00] increase in cost. [18:02] Sequence storage is currently in [18:04] production serving 30 plus models across [18:07] various product groups. [18:09] With that, let's take a look at some of [18:10] the key takeaways for the talk. [18:14] To summarize, we have some interesting [18:15] takeaways. [18:16] First one, training data scale is an [18:19] important problem and will continue to [18:21] be an important problem at Meta. [18:23] We are already at the limit in terms of [18:25] how much data we can afford to store, [18:28] and workloads such as user sequences are [18:31] trying to push this limit by almost 5 to [18:33] 10x. [18:34] Hence, there is an immediate need to [18:36] deal with this growth in a [18:38] cost-effective way. [18:39] Second, normalization is the right [18:42] direction forward. [18:43] Previously, when ML were highly [18:46] condensed summaries capturing user [18:48] behavior and interaction, we could [18:50] afford data duplication in order to [18:52] optimize IO. [18:53] However, with user sequences, where [18:56] there is a desire to store a lot more [18:58] data, data duplication becomes cost [19:00] prohibitive. [19:01] So, normalization is the only path [19:04] forward. [19:06] Finally, handling these emerging AI/ML [19:08] workload [19:10] often require full-stack innovation. [19:12] For dealing with user sequences, we had [19:15] to rework pretty much entire air [19:17] training stack and land optimization [19:20] both on the client and the storage [19:21] server side in order to make this [19:24] solution cost-effective and practical. [19:28] Now, let's take a look at what's in [19:30] store for sequence storage in the [19:32] future. [19:34] We have two major areas of investment. [19:37] The first one is scaling sequence [19:39] length. [19:40] Today, we only store about 80K events on [19:43] sequence storage. [19:44] But our North Star is to increase this [19:46] by almost 10x to capture lifetime user [19:49] history. [19:51] This scale will bring along with it more [19:54] challenges in cost, reliability, and [19:57] latency. [19:58] We need to reach out to techniques such [19:59] as smarter encodings, tearing, etc. to [20:03] make this work. This is still an ongoing [20:06] exploratory work where we are trying to [20:08] understand the limits of the current [20:10] system and how much more we can scale. [20:13] The second area of investment is to see [20:16] if we can generalize sequence storage so [20:18] that is [20:19] it is more broadly applicable beyond [20:21] just user sequences. [20:23] There's an interest to leverage whatever [20:25] we have built for sequence storage [20:28] to handle other kinds of features such [20:30] as item features, content features, and [20:32] embeddings. [20:34] Our North Star is a canonical storage [20:37] for all of the normalized training data [20:39] and the meta. [20:41] Thank you.