The Semantic ID Layer : Rebuilding RecSys With Meta

The Semantic ID Layer : Rebuilding RecSys With Meta — Ai4

Ai4

AI-assisted summary of The Semantic ID Layer : Rebuilding RecSys With Meta — original source. Check the original for full context.

Source published:

Detailed notes

## Background This talk was delivered by Aditya Gautam, Machine Learning Technical Lead at Meta, and published by the Ai4 channel on YouTube on August 16, 2026. According to the page description, it addresses the architectural limits of traditional item embeddings in recommendation systems and presents Semantic IDs—hierarchical sequences of discrete tokens learned via residual quantization across dual content and user trees—alongside downstream retrieval primitives, ranking paradigms, and production trade-offs.

## TL;DR Conventional recommendation systems struggle with massive embedding tables, severe popularity bias, and fragile cold-start generalization on long-tail items. Transitioning to Semantic IDs structures both multimodal content and user behaviors into discrete hierarchical prefix trees using Residual Vector Quantization (RVQ). This dual-tree architecture enables seven core tree-walk retrieval primitives—including collaborative filtering, tunable exploration dials, streaming trend detection, and cold-start substitution—while allowing discrete tokens to serve as native targets in generative models and embedding features in discriminative rankers. While offering significant compute and memory savings, deploying Semantic IDs in production demands careful mitigation of quantization loss, codebook collapse, and downstream retraining cascades caused by identifier drift.

## Atmosphere & style The presentation carries the pragmatic, authoritative tone of a deep systems architecture post-mortem and tech talk. Gautam methodically dismantles long-standing recommender paradigms with analytical rigor, replacing abstract machine learning hype with clear, mechanistic explanations of tree traversals, codebook dynamics, and vector quantization. Listening to the talk feels like an intense whiteboard engineering session that seamlessly bridges high-level generative AI trends with gritty, real-world production constraints like data drift, latency, and CI/CD versioning.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) Conventional arbitrary item ID embeddings dominate parameter budgets and suffer from severe popularity bias, forcing recommendation models into brute-force memorization rather than generalization. - [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) Semantic IDs replace opaque high-dimensional continuous vectors with compact, hierarchical sequences of discrete tokens that represent concepts from coarse to fine granularity. - [02:13](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=133s) Residual Vector Quantization (RVQ) converts continuous representations into discrete IDs by iteratively assigning embeddings to codebook centroids and passing residual vectors to subsequent levels. - [02:13](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=133s) Optimizing RVQ with collaborative filtering losses aligns discrete codes with behavioral co-engagement patterns rather than relying strictly on visual or textual surface features. - [04:37](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=277s) Multimodal fusion across text, audio, and video inputs generates richer dense vectors prior to quantization, significantly reducing discrete code collision rates across massive catalogs. - [06:34](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=394s) Applying Semantic ID tokenization symmetrically to users creates a dual-tree architecture over a shared vocabulary, cutting storage and inference compute while eliminating multi-gigabyte embedding tables. - [07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) Dual prefix trees consolidate candidate generation into unified tree-walk operations, replacing disparate bespoke retrieval systems with structured traversals like collaborative neighborhoods and history-prefix profiles. - [07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) Exploration can be programmatically tuned by stepping upward from saturated leaf nodes to broader parent categories and descending into adjacent semantic branches. - [10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) Monitoring engagement traffic volume through discrete tree branches transforms viral trend detection into a streaming anomaly problem capable of capturing surges in sub-15-minute windows. - [10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) Hierarchical prefix trees streamline policy guardrails by pruning entire subtrees with high violation densities and mitigate cold-start by substituting saturated viral items with unexposed sibling candidates. - [12:56](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=776s) Semantic IDs naturally function as native tokens in autoregressive generative recommenders, whereas discriminative models consume them as multi-codebook embedding lookups across multi-task heads. - [14:29](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=869s) Vector quantization inherently incurs loss of continuous geometric nuance, making discrete Semantic IDs and continuous dense embeddings symbiotic partners rather than mutual replacements. - [15:57](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=957s) Codebook collapse from power-law catalog distributions requires balanced centroid initialization techniques, such as balanced k-means, to prevent dead codes and uneven cluster allocations. - [15:57](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=957s) Upstream codebook or tokenizer updates trigger breaking, cascading retraining requirements across tree indexes, diversity rules, feature stores, and ranking models, demanding strict CI/CD data contracts.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) The Limits of Raw Item Embeddings and the Shift to Semantic IDs

In modern recommendation architectures, conventional embedding layers have run into fundamental structural bottlenecks. High-capacity recommender systems traditionally represent items from massive corpora using arbitrary categorical IDs that carry zero inherent semantic meaning. When these embeddings are co-trained alongside the primary model, severe disparities emerge: high-engagement, viral items with high click-through rates develop rich, well-separated representations through popularity bias, whereas sparse, tail items suffer from acute cold-start degradation because the model cannot generalize across non-descriptive identifiers. Rather than learning true underlying semantic relationships, legacy embedding tables force recommendation models into brute-force memorization.

To resolve these architectural limitations, the recommendation stack is transitioning toward Semantic IDs. Under this paradigm, content is no longer indexed as opaque, high-dimensional floating-point vectors, but as structured sequences of discrete tokens that explicitly encapsulate hierarchical semantic concepts from coarse to fine granularity. For instance, an item might be tokenized as a sequence of discrete integers such as `41` (Sports), `12` (Basketball), and `89` (Los Angeles Lakers). Representing items as intuitive, discrete tokens allows recommendation engines to bridge the gap with native sequence models and Large Language Models (LLMs), treating recommendation entities as structured language rather than uninterpretable float arrays.

- Arbitrary item ID embeddings suffer from severe popularity bias, leaving tail and newly ingested content poorly represented. - Traditional embedding tables demand massive parameter budgets to memorize individual items rather than generalizing across concepts. - Semantic IDs replace high-dimensional continuous float vectors with compact, interpretable, discrete hierarchical tokens (e.g., `41 -> 12 -> 89`).

---

[02:13](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=133s) Constructing Semantic IDs via Residual Quantization

Building on foundational research from Google, Semantic IDs are generated by transforming continuous entity representations into hierarchical discrete tokens using Residual Vector Quantization (RVQ). An upstream content or entity encoder first maps raw input signals into a continuous embedding space. This dense vector is then passed through a succession of hierarchical codebooks, where each codebook represents a level of conceptual clustering—progressing systematically from broad categories at the top level down to granular sub-topics at deeper levels.

```mermaid flowchart TD accTitle: Residual Vector Quantization Flow for Semantic IDs subgraph Quantization ["Residual Quantization (RVQ)"] A["Dense Content / User Embedding"] --> B["Codebook Level 1: Coarse Centroid"] B -->|"Extract Token 1 & Residual"| C["Codebook Level 2: Sub-category"] C -->|"Extract Token 2 & Residual"| D["Codebook Level 3: Fine-grained Entity"] end subgraph Output ["Discrete Semantic ID"] D --> E["Semantic Sequence: [41, 12, 89]"] end ```

The iterative quantization process operates by projecting the input embedding onto the nearest centroid within the first codebook, assigning that centroid's discrete index as the first token, and computing the residual difference vector between the original embedding and the selected centroid. This residual vector is subsequently fed into the next codebook layer to extract the subsequent token, repeating iteratively across all codebook levels. Importantly, the learned space is not restricted to visual or textual surface features; training with a collaborative filtering loss aligns the discrete codes with behavioral co-engagement patterns, ensuring items frequently co-clicked or liked within the same user sessions cluster together naturally.

``` +------------------+-------------------------------------------------------------+ | Architectural Axis| System Trade-offs & Behavioral Impact | +------------------+-------------------------------------------------------------+ | Codebook Depth | Dictates sequence length and fine-grained resolution; | | | deeper codebooks capture micro-level entity distinctions. | +------------------+-------------------------------------------------------------+ | Codebook Breadth | Governs vocabulary size per level (e.g., 64 vs. 1024); | | | wider codebooks provide broader top-level category splits. | +------------------+-------------------------------------------------------------+ ```

---

[04:37](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=277s) Multimodal Fusion and Hierarchical Prefix Trees

When generating Semantic IDs in production platforms, entities rarely consist of a single data type. Modern systems ingest video, raw audio, thumbnail images, and text descriptions simultaneously, routing each through specialized modality encoders. These individual representations are combined using early or late multimodal fusion strategies to synthesize a unified dense representation before quantization. Utilizing rich multimodal inputs significantly mitigates "code collision"—an edge case where two distinct items are assigned identical discrete code sequences due to constrained codebook capacity over vast item catalogs.

```mermaid flowchart LR accTitle: Multimodal Fusion to Prefix Tree Pipeline subgraph Ingestion ["Multimodal Encoding"] T["Text"] --> ET["Text Encoder"] V["Video"] --> EV["Video Encoder"] A["Audio"] --> EA["Audio Encoder"] end subgraph FusionLayer ["Fusion & Quantization"] ET & EV & EA --> F["Multimodal Fusion"] F --> RVQ["Residual Quantizer"] end subgraph Tree ["Hierarchical Prefix Tree"] RVQ --> PT["Root Node<br/>└─ [41] Sports<br/> └─ [12] Basketball<br/> └─ [89] Lakers"] end ```

Once discrete Semantic IDs are generated across the item catalog, the entire corpus is structured into a hierarchical prefix tree. In this tree, the root branches out into broad category prefixes (such as sports), which subsequently branch into sub-genres (such as basketball), followed by specific teams, athletes, and individual pieces of media. Structuring items as paths through an optimized, pruned prefix tree provides the core index necessary for high-throughput candidate retrieval, ranking pipelines, and downstream diversity enforcement.

- Multimodal fusion combines text, audio, image, and video embeddings, drastically lowering discrete token collision rates across large-scale catalogs. - Discrete codes naturally arrange into prefix trees that enable fast pruning and deterministic traversal across the entire recommendation stack.

---

[06:34](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=394s) Dual-Tree Architecture: Content and User Semantic Spaces

The power of Semantic IDs becomes fully realized when the exact same tokenization and prefix tree principles are applied symmetrically to users. A user's real-time engagements, historical clicks, long-term consumption profiles, and static demographic features are processed by a dedicated user encoder. This produces a unified user embedding that is quantized through a shared vocabulary, creating a corresponding User Prefix Tree.

```mermaid flowchart TD accTitle: Dual Tree Recommendation Framework subgraph ContentSpace ["Content Domain"] C_In["Multimodal Item Data"] --> C_Enc["Content Encoder"] C_Enc --> C_Tree["Content Prefix Tree"] end subgraph UserSpace ["User Domain"] U_In["History, Clicks, Demographics"] --> U_Enc["User Encoder"] U_Enc --> U_Tree["User Prefix Tree"] end subgraph Interaction ["Unified Discrete Engine"] C_Tree <-->|"Tree-Walk Candidate Retrieval"| U_Tree end ```

Recent research findings from Google demonstrate significant production benefits when unifying user and content modeling around discrete semantic codes. Replacing massive continuous float vectors with short sequences of integers slashes memory footprints and storage overhead while drastically reducing training and inference serving compute. Because discrete prefixes share structural semantics across items and users, platforms observe notable engagement gains, substantial reductions in byte-per-user data consumption, and an effective mitigation of the traditional cold-start problem.

- User histories, static attributes, and real-time interaction signals are mapped directly into a discrete User Semantic ID tree. - Shared discrete vocabularies eliminate multi-gigabyte embedding parameter tables, converting costly continuous matrix operations into lightweight integer index operations. - Production implementations demonstrate improved user engagement, lower byte-per-user infrastructure bandwidth, and dramatic cold-start reductions.

---

[07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) Tree-Walk Retrieval Primitives: Collaborative Neighborhoods, Exploration, and Trends

With dual prefix trees established as the foundational layer, candidate generation across the recommendation funnel simplifies into structured tree-walk operations. Rather than relying on separate bespoke retrieval systems, seven distinct retrieval primitives emerge naturally from navigating the prefix hierarchy:

```mermaid flowchart TD accTitle: Tree-Walk Retrieval Primitives Root["Prefix Tree Traversal"] Root --> CF["User-to-User Neighborhoods<br/>(Collaborative Filtering)"] Root --> Hist["History-Prefix Profiles<br/>(Direct Exploitation)"] Root --> Exp["Prefix Step-Up Traversal<br/>(Tunable Exploration Dial)"] Root --> Trend["Real-Time Window Aggregation<br/>(Streaming Trend Detection)"] ```

1. **User-to-User Collaborative Filtering:** Locating a user's current node in the user prefix tree and finding neighboring user branches allows the retrieval engine to source candidates liked by similar behavioral cohorts. 2. **History-Prefix Interest Profiles (Exploitation):** By parsing a user's recent interactions across the past session, day, or month, the system compiles an interest profile representing engagement percentages across specific prefix codes. Traversing down identical prefixes fetches tightly matched candidates for targeted exploitation. 3. **Tunable Exploration Dials:** Rather than descending exclusively into a user's heavily saturated leaf nodes (e.g., Lakers highlight clips), the retrieval algorithm steps one or two levels up the tree—retreating from Lakers to the broader Sports node, then descending an adjacent branch such as Hockey. This provides a continuously tunable dial to balance familiarity with serendipitous discovery. 4. **Deduplication and Feed Diversity:** Semantic prefixes act as deterministic categorical boundaries. Downstream diversity layers enforce rules requiring a configurable gap of $n$ candidate items between identical prefix codes, preventing repetitive, monochromatic feeds. 5. **Streaming Trend Detection:** By monitoring the real-time consumption volume passing through tree branches over rolling 5-, 10-, and 15-minute windows, the platform detects sudden traffic spikes at specific prefix nodes to surface viral trends instantaneously.

- User interest profiles are dynamically assembled by calculating consumption proportions across prefix lengths to match the desired granularity. - Exploration is programmatically controlled by navigating upward toward parent nodes and descending adjacent semantic branches. - Real-time aggregation over discrete tree nodes surfaces immediate velocity spikes, enabling sub-15-minute viral trend retrieval without retraining dense encoders.

[10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) Streaming Trend Detection, Policy Guardrails, and Cold-Start Balancing

Tracking engagement through hierarchical Semantic ID tree nodes transforms complex real-time event monitoring into a straightforward volume-anomaly problem. By comparing the baseline volume of items passing through specific tree branches against historical distributions, systems can instantly surface emerging viral phenomena. A clear example occurred during the World Cup final: engagement around baseline soccer content spiked significantly, Argentina-related subtrees surged substantially higher, and Lionel Messi-specific nodes hit all-time peak traffic. By identifying these "hot nodes" in a streaming fashion, the recommendation infrastructure detects macro and micro trends organically without relying on delayed batch indexing.

```mermaid flowchart TD accTitle: Real-Time Tree Operations on Semantic IDs accDescription: Flow showing how Semantic ID hierarchical trees enable trend detection, policy blocking, and cold-start exploration.

subgraph StreamOps ["Tree-Walk Stream Operations"] A["Hierarchical Prefix Tree"] --> B["Hot Node / Trend Detection<br/>(e.g., World Cup, Messi surge)"] A --> C["Policy & Sensitive Blocking<br/>(Prune nodes with >50% violations)"] A --> D["Cold-Start Substitution<br/>(Swap viral nodes with cold sibling items)"] end

subgraph Downstream ["Downstream Ecosystem"] B --> E["Real-Time Exploration"] C --> F["Safe Candidate Pool"] D --> G["Quality & Engagement Feedback"] end ```

The same hierarchical structure streamlines content safety, policy enforcement, and cold-start exploration. For trust and safety operations, when policy violations cluster tightly—such as a subtree where over 50% of the passing items violate community guidelines—moderation systems can proactively block traffic at that prefix node or prune one level up in the hierarchy to calibrate precision and recall against sensitive topics.

To combat systemic popularity bias, where legacy viral items (like evergreen Los Angeles Lakers clips) crowd out new uploads, platforms leverage prefix similarity for cold-start substitution. By identifying under-exposed items sharing a discrete semantic prefix with high-engagement nodes, the system can route traffic to promising new candidates, evaluate their quality using engagement probabilities such as $P(\text{like})$, and escalate successful content up the discovery funnel to maintain a balanced creator ecosystem.

- **Trend detection mechanism:** Real-time volume divergence against historical baselines at prefix-level granularities. - **Policy enforcement tuning:** Dynamic pruning of parent/child nodes based on aggregate violation density across subtrees. - **Cold-start mitigation:** Replacing saturated viral content with unexposed sibling items that share identical semantic prefixes.

---

[12:56](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=776s) Diversity Controls and Downstream Ranking: Generative vs. Discriminative Models

Tree-structured discrete IDs directly solve the feed repetition problem within the deduplication and diversity layers. If a candidate retrieval set contains heavy prefix overlap—for instance, 50% of retrieved items sharing an identical three- or four-level prefix—the ranking pipeline can penalize or diversify those candidates. Instead of serving four consecutive Lakers highlights at the top of a user's feed, the system enforces balanced distribution across broader interest prefixes, interspersing sports, politics, and gardening.

Downstream ranking harnesses Semantic IDs through two distinct paradigms: generative and discriminative architectures. Generative recommendation models achieve the most substantial performance gains because they treat Semantic IDs natively as language tokens, auto-regressively predicting next-item codes using standard token-generation techniques.

```mermaid flowchart LR accTitle: Ranking Architecture Dual Consumption accDescription: How Semantic IDs are consumed natively by Generative models versus embedded across codebooks for Discriminative models.

A["Discrete Semantic IDs"] --> B["Generative RecSys"] A --> C["Discriminative RecSys"]

subgraph Generative ["Autoregressive Path"] B --> D["Native Token Decoding<br/>(Next-Token Generation)"] end

subgraph Discriminative ["Multi-Task Path"] C --> E["Per-Codebook Embedding Lookups"] E --> F["Multi-Objective Heads<br/>P(like), P(save), P(engage)"] end ```

In contrast, traditional discriminative models—which optimize multi-task objectives such as $P(\text{like})$, $P(\text{engagement})$, and $P(\text{save})$—cannot natively decode discrete sequences. Instead, they look up and learn dense representations for each code across $N$ codebooks, pooling and combining codebook-level embeddings during feature representation learning.

- **Diversity enforcement:** Prefix-level caps prevent candidate over-saturation across identical subtopics. - **Generative recommendation advantage:** Discrete codes function directly as native vocabulary tokens in autoregressive architectures. - **Discriminative integration:** Discrete tokens from each codebook level are projected into learned embedding spaces across multi-task classification heads.

---

[14:29](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=869s) Fundamental Trade-Offs: Quantization Loss, Model Expressiveness, and Collisions

Transitioning from continuous vector spaces to discrete hierarchical codes introduces real architectural trade-offs. The vector quantization step is fundamentally lossy; converting rich 512-dimensional floating-point vectors into a compact sequence of discrete indices inevitably sacrifices continuous geometric nuance. However, this trade-off provides an "information gain" in structural interpretability and cross-component reuse. In modern production environments, discrete Semantic IDs and continuous dense embeddings do not replace one another; rather, they coexist symbiotically to balance broad retrieval coverage with fine-grained scoring.

``` +-----------------------------------+-----------------------------------+ | Continuous Dense Embeddings | Discrete Semantic IDs | +-----------------------------------+-----------------------------------+ | 512-dim floating-point precision | Short hierarchical token sequence | | Near-zero collision probability | Finite codebook collision rate | | Maximally expressive geometry | Structured, interpretable tree | | Memorization / Large table footprint | Shared vocabulary / Token native| +-----------------------------------+-----------------------------------+ ```

A major limitation of tree-based quantization is the reduction in model expressiveness. While continuous 512-dimensional float spaces have virtually zero chance of collision, discrete codebooks introduce a finite collision rate where distinct items map to the same identifier string. However, optimizing collision rates yields diminishing returns. Engineers must avoid the "collision rate trap," where excessive engineering effort is spent eliminating harmless low-level leaf collisions at the expense of system latency and simplicity.

---

[15:57](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=957s) Production Engineering: Codebook Collapse, Data Drift, and the Versioning Challenge

Productionizing Semantic IDs introduces operational bottlenecks, beginning with codebook collapse. Because item engagement and content distribution follow steep power-law curves, standard residual quantization frequently under-indexes long-tail topics while heavily overloading popular clusters. To build balanced and stable codebooks, teams rely on robust initialization strategies, including balanced k-means centroid seeding, to prevent dead codes and stabilize code assignment across unbalanced catalogs.

```mermaid flowchart TD accTitle: Production Retraining and Versioning Pipeline accDescription: Demonstrates how updates to tokenizers or codebooks cascade across the entire recommendation stack.

subgraph CodebookUpdate ["Upstream Architecture Update"] A["Data Distribution Drift / Content Influx"] --> B["Tokenizer & Codebook Re-indexing<br/>(Width / Depth Expansion)"] end

subgraph CascadeRetrain ["Synchronized Retraining Stack"] B --> C["Tree Reconstruction"] C --> D["Retrieval & Candidate Generators"] D --> E["Diversity & De-duplication Layer"] E --> F["Discriminative & Generative Rankers"] end

subgraph Governance ["Production Guardrails"] F --> G["CI/CD Data Contracts & Versioned Serving"] end ```

Data drift and identifier versioning represent the most complex engineering hurdles in production. When rapid catalog expansion requires widening codebooks, increasing tree depth, or altering underlying tokenizers, the downstream ripple effect is severe:

- **Cascading retrains:** Modifying upstream semantic token definitions immediately invalidates downstream tree walks, candidate generation indexes, diversity rules, discriminative feature stores, and generative ranking models. - **CI/CD guardrails:** Platforms must maintain strict data contracts and coordinated CI/CD pipelines to ensure continuous retraining across every layer before switching token versions in live traffic. - **Drift foresight:** Teams must proactively analyze catalog velocity and domain shifts "before building the semantic ID" to select codebook dimensions capable of absorbing high content churn without triggering frequent breaking migrations.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) Item embeddings are hitting real limits: embedding tables that dominate the parameter budget, brittle cold start, and representations that force models to memorize instead of generalize. — Aditya Gautam

> [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) Rather than learning true underlying semantic relationships, legacy embedding tables force recommendation models into brute-force memorization. — Aditya Gautam

> [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) Under this paradigm, content is no longer indexed as opaque, high-dimensional floating-point vectors, but as structured sequences of discrete tokens that explicitly encapsulate hierarchical semantic concepts from coarse to fine granularity. — Aditya Gautam

> [06:34](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=394s) The power of Semantic IDs becomes fully realized when the exact same tokenization and prefix tree principles are applied symmetrically to users. — Aditya Gautam

> [07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) With dual prefix trees established as the foundational layer, candidate generation across the recommendation funnel simplifies into structured tree-walk operations. — Aditya Gautam

> [10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) Tracking engagement through hierarchical Semantic ID tree nodes transforms complex real-time event monitoring into a straightforward volume-anomaly problem. — Aditya Gautam

> [12:56](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=776s) Generative recommendation models achieve the most substantial performance gains because they treat Semantic IDs natively as language tokens, auto-regressively predicting next-item codes using standard token-generation techniques. — Aditya Gautam

> [14:29](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=869s) The vector quantization step is fundamentally lossy; converting rich 512-dimensional floating-point vectors into a compact sequence of discrete indices inevitably sacrifices continuous geometric nuance. — Aditya Gautam

> [14:29](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=869s) In modern production environments, discrete Semantic IDs and continuous dense embeddings do not replace one another; rather, they coexist symbiotically to balance broad retrieval coverage with fine-grained scoring. — Aditya Gautam

> [14:29](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=869s) Engineers must avoid the 'collision rate trap,' where excessive engineering effort is spent eliminating harmless low-level leaf collisions at the expense of system latency and simplicity. — Aditya Gautam

> [15:57](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=957s) Modifying upstream semantic token definitions immediately invalidates downstream tree walks, candidate generation indexes, diversity rules, discriminative feature stores, and generative ranking models. — Aditya Gautam

> [14:29](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=869s) Semantic IDs do not eliminate dense embeddings; they turn embedding geometry into reusable structure. — Aditya Gautam

## Mentioned Aditya Gautam, Meta, Google, Ai4, Residual Vector Quantization (RVQ), Large Language Models (LLMs), Semantic IDs, Los Angeles Lakers, Lionel Messi, World Cup

## Video description Aditya Gautam - Machine Learning Technical Lead Meta

Item embeddings are hitting real limits: embedding tables that dominate the parameter budget, brittle cold start, and representations that force models to memorize instead of generalize. This talk traces the shift to Semantic IDs — short, hierarchical sequences of discrete codes derived from content and learned with residual quantization.

This talk will briefly talk about how they're built, then make the case that the interesting move isn't tokenizing items, it's building two trees: one over content, one over users, sharing a vocabulary. From that foundation, seven retrieval sources fall out as tree walks — user-to-user neighborhoods, history-prefix expansion, a tunable exploration dial, prefix-frequency interest profiles, streaming trend detection, cold-start substitution, and prefix-level policy blocklists. Downstream, the same codes serve as ranking features and as generation targets.

The talk concludes with a pragmatic production architecture and an honest assessment of quantization loss, code collapse, semantic–behavior misalignment, identifier churn, privacy, and constrained-decoding latency. Semantic IDs do not eliminate dense embeddings; they turn embedding geometry into reusable structure.

[Source](https://www.youtube.com/watch?v=hV0RNjnWLYA)

中文简报

语义ID层：重构推荐系统架构 传统推荐系统面临庞大的嵌入表、严重的热度偏置以及长尾物品冷启动泛化能力脆弱等挑战。转向语义ID后，可通过残差向量量化（RVQ）将多模态内容与用户行为结构化为离散的分层前缀树。这种双树架构支持七种核心树遍历检索原语——包括协同过滤、可调探索机制、流式趋势检测以及冷启动替换——同时允许离散Token既作为生成式模型中的原生目标，又作为判别式重排模型中的嵌入特征。尽管能显著节省计算与内存开销，但在生产环境中部署语义ID仍需谨慎应对量化损失、码本坍塌，以及标识符漂移引发的下游重训级联问题。 传统的任意物品ID嵌入占据了绝大部分参数预算，并伴随严重的热度偏置，迫使推荐模型陷入暴力记忆而非泛化学习。 (0:04) 语义ID用紧凑、分层的离散Token序列取代了不透明的高维连续向量，由粗到细地表达不同粒度的概念。 (0:04) 残差向量量化（RVQ）通过迭代地将嵌入分配给码本聚类中心并将残差向量传递至后续层级，从而将连续表征转换为离散ID。 (2:13) 利用协同过滤损失优化RVQ，能够使离散编码与用户行为的协同互动模式保持一致，而非仅依赖视觉或文本的表面特征。 (2:13) 跨文本、音频和视频输入的多模态融合在量化前生成了更丰富的稠密向量，显著降低了海量商品目录下的离散编码碰撞率。 (4:37)

Reading summary

Modern recommendation architectures are moving away from brute-force continuous embedding tables toward discrete, hierarchical Semantic IDs, unifying multimodal item catalogs and user representations into structured prefix trees. Resolving Embedding Bottlenecks via Residual Quantization (0:04) Traditional recommender systems assign arbitrary categorical IDs to items, forcing embedding tables to memorize individual entities rather than learn conceptual relationships. This architectural design creates severe popularity bias: viral items develop rich representations, while sparse long-tail items suffer from acute cold-start degradation. To solve this, continuous representations are replaced by Semantic IDs—compact sequences of discrete hierarchical integers (such as 41 for Sports, 12 for Basketball, and 89 for Los Angeles Lakers) that bridge recommendation architectures with sequence models and LLMs. Semantic IDs are constructed using Residual Vector Quantization (RVQ), based on research from Google. Multimodal encoders first combine video, audio, thumbnail, and text signals into a dense representation, which mitigates code collision across expansive catalogs. The dense embedding is iteratively projected onto hierarchical codebooks, where each level extracts a discrete centroid index and passes the residual difference vector to the next layer. Incorporating collaborative filtering losses further aligns these discrete codes with behavioral co-engagement patterns. Dual Prefix Trees and Structured Candidate Retrieval (6:34) The architecture reaches full efficiency when applied symmetrically to both content and users. A user encoder quantizes interaction histories, click streams, and demographic features into a discrete User Semantic ID tree using a shared vocabulary. According to Google research, this dual-tree model eliminates multi-gigabyte continuous embedding tables, substantially lowers byte-per-user infrastructure bandwidth, and reduces training and serving compute. With dual prefix trees established, candidate generation simplifies into deterministic tree-walk operations rather than bespoke retrieval models. Engines navigate tree branches using distinct primitives: user-to-user collaborative filtering finds candidate items across neighboring user branches, while history-prefix profiles facilitate direct exploitation. For exploration, the algorithm steps up from specific leaf nodes to broader parent nodes before descending adjacent branches. Furthermore, semantic prefixes act as clear categorical boundaries, allowing downstream diversity layers to mandate spacing between identical prefix codes to prevent repetitive feeds. Streaming Trends, Policy Guardrails, and Cold-Start Balancing (10:49) Because Semantic IDs arrange content hierarchically, real-time event monitoring becomes a volume-anomaly calculation. By tracking traffic passing through prefix nodes over rolling 5-, 10-, and 15-minute windows against baseline distributions, systems detect emerging viral trends organically—demonstrated during the World Cup final when soccer, Argentina, and Lionel Messi subtrees experienced consecutive volume surges without requiring dense encoder retraining. This hierarchical layout also streamlines platform safety and creator ecosystem balance. Content moderation teams can dynamically block or prune entire subtrees when aggregate policy violation density exceeds thresholds, such as 50%. To counteract popularity bias where legacy viral content crowds out new uploads, platforms implement cold-start substitution by routing traffic from saturated nodes to unexposed sibling items that share the exact same semantic prefix, evaluating their quality via engagement probabilities like P(like). Downstream Ranking Paradigms and Production Engineering Challenges (12:56) Downstream ranking models ingest Semantic IDs through two mechanisms. Generative recommenders achieve strong performance gains by treating Semantic IDs natively as language tokens, predicting next-item codes autoregressively. Conversely, traditional discriminative models project tokens from each codebook level into learned embedding spaces to score multi-task objectives such as P(like) and P(save). However, quantization is inherently lossy, trading continuous geometric precision for structural interpretability. In production, power-law catalog distributions risk codebook collapse, requiring balanced k-means centroid initialization to prevent dead codes. The most severe operational bottleneck is data drift: altering tokenizers, codebook breadth, or tree depth invalidates downstream tree indices, retrieval layers, diversity rules, and ranking models, demanding strict data contracts and coordinated CI/CD retraining cascades before new token versions can be deployed to live traffic. While Semantic IDs streamline candidate retrieval and generative ranking, their discrete nature introduces finite code collisions and quantization loss. The primary unresolved engineering challenge remains operational: catalog drift and codebook modifications trigger costly, cascading retrains across the entire recommendation stack, making upfront codebook sizing and robust data contracts critical.

精炼阅读摘要

现代推荐系统正在从缺乏语义且极易产生流行度偏差的传统离散 ID 向量表，转向基于残差向量量化生成的语义标识符（Semantic IDs）。这一架构通过将实体与用户统一编码为分层的前缀树，不仅实现了低成本的高效检索与全流程实时调控，还打通了生成式大语言模型处理推荐实体的统一表示范式。 传统嵌入局限与语义标识符量化生成 (0:04) 传统推荐架构采用任意分类 ID 配合高维浮点向量嵌入，在与主模型联合训练时，高度依赖曝光的流行物品会占据丰富特征空间，而长尾与冷启动物品则因缺乏可泛化的语义信息而表现匮乏。这种依赖多吉字节参数表进行机械记忆的模式，促使系统向语义标识符（Semantic IDs）演变。Semantic IDs 将连续浮点向量转化为自粗至细的离散整数序列（例如以 41、12、89 分别指代体育、篮球和洛杉矶湖人），使推荐实体可以直接接入序列模型与大语言模型。 残差量化机制与多模态前缀树构建 (2:13) 基于 Google 的基础研究，Semantic IDs 利用残差向量量化（RVQ）技术将上游多模态编码器输出的连续嵌入投影到层级码本中。系统首先寻找最顶层码本的最近中心点输出首个标记，再计算残差并迭代输入后续子码本，逐层提取更细粒度的实体标记。通过引入协同过滤损失联合训练，离散编码不仅包含图文视听特征，还能对齐用户共现交互行为。码本的深度决定序列长度与分辨率，宽度决定各层词表规模；融合文本、视频和音频特征能有效降低不同物品映射为相同序列的“代码碰撞”概率，最终将全量内容组织为可确定性遍历与剪枝的分层前缀树。 内容与用户双树架构及其树遍历检索原语 (6:34) 当内容与用户空间对称应用相同的离散标记与前缀树时，语义架构的效能得到充分体现。用户编码器将实时交互、历史点击与静态画像映射为用户前缀树，消除了传统大规模连续矩阵运算，转为轻量级整数索引。在此双树结构上，候选召回抽象为七类树遍历操作：通过用户前缀分支寻找行为相似邻居实现协同过滤；依据历史前缀交互比例下钻以进行精准利用；退回上一级父节点并探索邻近分支（如从湖人回退到体育再下探至曲棍球）充当可调节的探索滑块；利用前缀间隔约束候选多样性；以及通过统计滚动窗口内的流量突增实时捕获爆款趋势。 流式监控、安全风控与冷启动平衡机制 (10:49) 分层前缀树将复杂的实时事件监测转化为基于树节点流量偏离基线的异常检测。例如在世界杯决赛期间，足球、阿根廷及梅西对应的子树流量呈现出阶梯式爆发，系统无需等待延迟批处理即可即时定位微观与宏观热点。在内容安全层面，若某一子树超过 50% 的物品存在违规行为，风控系统可直接在对应前缀节点或向上一级进行整体修剪拦截。此外，面对高热度旧内容挤占曝光的问题，系统可利用相同语义前缀寻找曝光不足的冷启动候选进行同位替换，结合预估互动率评估内容质量，实现生态平衡。 下游排序集成与生产工程挑战 (12:56) 在下游排序中，生成式模型原生将 Semantic IDs 视作词表标记以自回归方式预测下一个物品标记；而传统判别式多任务模型则按码本层级分别查找并融合嵌入向量，以预估点赞、保存等目标概率。从连续 512 维向量量化为离散编码必然带来几何 nuance 的损失，但获得了结构可解释性；盲目消除底层叶节点的低频碰撞并不具备工程性价比。在实际生产中，数据幂律分布易导致码本崩溃，需采用平衡 k-means 聚类初始化。更为棘手的是数据漂移与版本升级，一旦调整码本维度或分词器，下游的检索索引、多样性策略和生成式排序模型均需协同重训，必须依赖前置数据契约与严格的 CI/CD 机制保障平滑迁移。 语义标识符架构通过离散分层树结构解决了传统推荐系统参数庞大、流行度偏差严重及冷启动困难等核心痛点，使推荐引擎能够无缝拥抱生成式建模。然而，其根本挑战在于量化过程中的表达力损失与码本崩溃风险，以及分词版本变动对整条级联系统造成的重训成本；在工程落地时，团队必须在连续几何精度与离散结构化可扩展性之间寻求协同平衡。

Detailed notes — zh

## Background 本内容由 Ai4 于 2026 年 8 月 16 日发布在 YouTube 平台，主讲人为 Meta 的机器学习技术负责人 Aditya Gautam。视频围绕推荐系统中传统 Item Embedding 的物理与算法瓶颈展开，系统阐述了利用残差量化构建分层离散 Semantic ID、搭建内容与用户双前缀树以及重构推荐检索与排序体系的核心实践。 ## TL;DR 演讲首先剖析了传统连续 Embedding 导致的参数膨胀、死记硬背与冷启动脆弱等痛点，提出了基于残差量化（RQ-VAE）将实体转化为分层离散 Token 的 Semantic ID 方案。通过共享词表构建“内容前缀树”与“用户前缀树”，系统将协同召回、兴趣画像分析、探索旋钮、实时热点捕获与前缀安全风控等多种召回源统一抽象为高效的树遍历操作。在下游排序中，Semantic ID 既能作为原生 Token 驱动生成式推荐，也可为判别式模型提供多层级语义先验。最后，演讲深入探讨了工业化实践中必须权衡的量化损失、码本坍缩、数据漂移及全链路版本迭代等工程挑战。 ## Atmosphere & style 整场演讲洋溢着硅谷顶级科技大厂硬核、严谨且务实的工业工程气质。主讲人以清晰利落的架构思维层层递进，不仅以优雅的拓扑视角重构了复杂繁琐的召回体系，更毫无保留地揭示了实际落地中的诸多工程暗礁。叙事节奏紧凑洗练，在理论前瞻性与生产落地权衡之间取得了极佳的平衡，给人以深刻的架构启发感与工业实操说服力。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) 传统 Item Embedding 机制面临参数表急剧膨胀、长尾冷启动脆弱以及迫使模型“死记硬背”等物理与算法瓶颈，Semantic ID 推动推荐系统转向分层离散语义表示与泛化。 - [02:13](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=133s) Semantic ID 底层基于残差量化（RQ-VAE）机制，通过多层码本逐级量化残差，并融合多模态特征与协同过滤损失，将连续高维向量转化为自带层次语义的离散 Token 序列。 - [05:52](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=352s) 推荐系统将内容端与用户端分别组织为共享词表的“内容前缀树（Content Tree）”与“用户前缀树（User Tree）”，用紧凑整数索引替代浮点向量计算，大幅削减传输、存储及算力开销。 - [07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) 双前缀树结构将复杂的召回策略统一为树遍历（Tree Walks）操作，能够直接支撑用户协同召回（User-to-User CF）与多时间跨度的历史兴趣前缀利用。 - [07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) 通过树层级回退机制（Exploration Dial），系统可主动从细粒度叶节点向上回退并向相邻分支横向探索，实现推荐新颖度与多样性的可调控平衡。 - [10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) 流式监控系统通过对比各前缀节点的实时流量激增与历史基线，能够秒级捕获全网突发热点，无需依赖复杂的文本理解模型。 - [10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) 前缀树结构可应用于内容风控与安全审核，当特定分支违规比例超标时，系统可在对应前缀层级直接批量阻断，并灵活调节查准率与查全率。 - [12:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=724s) 利用相同前缀下的语义同质性，系统可主动用潜力新内容替换高曝光热门爆款进行试探分发，结合互动预估扶持新内容，打破流行度偏差（Popularity Bias）。 - [13:35](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=815s) Semantic ID 天然适配生成式推荐范式，大语言模型与自回归架构可将其作为离散 Token 原生解码生成；在判别式多目标排序中则作为多码本 Embedding 特征输入。 - [14:31](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=871s) Semantic ID 与连续稠密向量（Dense Embeddings）是互补关系而非互斥替代；工业实践中残差量化存在固有精度损失，过度追求极低碰撞率会陷入边际效用递减陷阱。 - [15:57](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=957s) 工业化落地的核心挑战在于码本坍缩（Codebook Collapse）、数据漂移以及全链路版本对齐危机，底层分词器与树结构的调整会穿透召回、去重及排序全流程。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) 传统 Embedding 的痛点与 Semantic ID 的核心理念 在现代推荐系统架构中，传统的 Item Embedding 机制正逐渐触碰其物理与算法瓶颈。传统系统直接为海量候选库中的每个 Item 分配一个互不相关的整数 ID，并让模型在协同训练过程中学习对应的浮点数向量。这种模式存在致命缺陷：由于 ID 之间缺乏固有的语义关联，模型被迫去“死记硬背”（Memorization）海量 ID，导致 Embedding 表急剧膨胀并占据绝大部分模型参数预算；同时，长尾和冷启动物品（Cold Start Items）由于曝光和点击匮乏，根本无法学到高质量的表征，而高频热门内容则因流行度偏差占据了过拟合的优势表征空间。 为了打破这种“强行记忆”的困境，Semantic ID（语义 ID）方案应运而生。其核心思想不再使用连续的高维浮点数向量或扁平的离散 ID 来代表内容，而是将每个实体映射为一段简短、具有层次结构的离散 Token 序列。这些 Token 自带层级语义，例如序列中的第一位 `41` 代表大类“体育（Sports）”，第二位 `12` 代表子类“篮球（Basketball）”，第三位 `89` 则细化到“湖人队（Lakers）”。这种离散分层的编码方式不仅极其直观，能够原生对接大语言模型（LLM）的 Token 预测范式，还天然赋予了实体语义泛化能力。 - **传统 Embedding 缺陷**：ID 缺乏内在语义、Embedding 词表无限膨胀、高频与低频内容表征严重失衡。 - **冷启动与泛化难题**：低曝光物品无法有效训练 Embedding，导致冷启动推荐效果脆弱；模型缺乏结构化先验，无法实现跨物品的语义泛化。 - **Semantic ID 的本质**：将连续高维向量量化为层次化离散 Token 序列，实现从“机械记忆”到“语义理解”的跨越。 --- ### [02:13](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=133s) 基于残差量化（Residual Quantization）构建语义编码与多模态融合 Semantic ID 的底层量化算法源自残差量化（Residual Quantization, RQ-VAE）。整个构建流程始于内容的多模态编码器（Encoder）：内容或用户的原始特征首先被转化为高维密集向量，随后依次穿过预设的多层码本（Codebooks）。在第一层码本中，系统找到与输入向量欧氏距离最近的码字（Codeword），记录其索引（例如 `41`），并计算输入向量与该码字之间的残差；该残差向量紧接着被送入第二层码本进行进一步量化，以此类推逐层迭代。每一层码本就像一个特定粒度的聚类空间，层级越深，所描述的语义就越精细。 在训练码本时，量化目标不仅限于捕捉视觉或文本的表层相似度。通过在损失函数中引入协同过滤损失（Collaborative Loss），系统可以让经常被同批用户连续点击、浏览的内容在码本空间中聚拢，从而将行为协同信号无缝注入离散语义编码中。此外，码本的设计面临“深度（Depth）”与“宽度（Breadth）”的权衡：若第一层码本容量较小（如 64），其首个 Token 只能区分“体育”等宽泛大类；若扩大至 1024，则第一层就能细分出“水上运动”、“冰球”或“田径”等领域，从而改变模型记忆与泛化的粒度。 ```mermaid flowchart TD accTitle: Residual Quantization and Multimodal Fusion Architecture subgraph MultiModal ["多模态特征融合层"] T["文本特征 Text"] --> TE["文本编码器"] I["图像特征 Image"] --> IE["图像编码器"] V["视频特征 Video"] --> VE["视频编码器"] A["音频特征 Audio"] --> AE["音频编码器"] TE & IE & VE & AE --> Fusion["早期 / 晚期特征融合 (Early/Late Fusion)"] Fusion --> Dense["统一密集嵌入向量 (Dense Embedding)"] end subgraph RQ ["残差量化生成 Semantic ID (RQ-VAE)"] Dense --> CB1["Codebook Level 1 (粗粒度: 如体育 41)"] CB1 --> |计算残差 Residual 1| CB2["Codebook Level 2 (中粒度: 如篮球 12)"] CB2 --> |计算残差 Residual 2| CB3["Codebook Level 3 (细粒度: 如湖人 89)"] CB1 -.-> ID1["Token 1: 41"] CB2 -.-> ID2["Token 2: 12"] CB3 -.-> ID3["Token 3: 89"] end ID1 & ID2 & ID3 --> SemanticID["最终 Semantic ID: [41, 12, 89,...]"] ``` - **残差迭代机制**：通过逐级量化残差逼近原始向量，多层码本索引串联即构成最终离散 ID 序列。 - **多模态表征与碰撞消解**：综合文本、图像、视频和音频特征进行早期或晚期融合（Early/Late Fusion），能够极大丰富表征信息量，有效缓解不同实体落入同一组 Semantic ID 的“编码碰撞（Collision）”问题。 - **目标函数灵活性**：既支持基于内容相似度的重构目标，也可结合点击共现等协同过滤信号进行联合优化。 --- ### [05:52](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=352s) 统一构建双前缀树：Content Tree 与 User Tree 获得海量内容的离散 Semantic ID 序列后，基础层架构的核心任务便是将这些离散序列组织为前缀树（Prefix Tree）。在前缀树中，根节点代表全局起点，子节点根据第一位 Token（如大类“体育”）分叉，随后沿着第二位 Token（如“篮球”）、第三位 Token（如“湖人队”）层层下钻。整棵树从粗粒度类别逐步收敛至具体的单项内容，为后续的高效检索、剪枝与多样性打散提供了强有力的拓扑数据结构。 同样的设计范式被完全复用到用户端。系统提取用户的长期消费历史、静态画像、实时互动及点击流特征，经由编码器生成用户表征向量，并使用同一套量化词表构建对应的“用户前缀树（User Tree）”。至此，推荐系统的底层基座演进为由共享词表驱动的“内容树”与“用户树”双树架构。这种结构在性能与工程效率上带来了飞跃：根据 Google 最新发表的研究，引入离散语义 ID 后，用户端各项核心互动指标全线提升，单用户传输与存储的字节开销大幅削减，冷启动表现显著改善；同时，由于紧凑的整数完全取代了沉重的高维浮点向量计算，模型的训练与推理算力成本均呈断崖式下降。 ```mermaid flowchart LR accTitle: Dual Prefix Trees Architecture subgraph ContentSpace ["内容前缀树 (Content Tree)"] CRoot["Root"] --> C_Sports["41: 体育"] C_Sports --> C_BB["12: 篮球"] C_Sports --> C_HK["18: 冰球"] C_BB --> C_Lakers["89: 湖人视频 A/B"] end subgraph UserSpace ["用户前缀树 (User Tree)"] URoot["Root"] --> U_Sports["41: 体育兴趣偏好"] U_Sports --> U_BB["12: 高频互动-篮球"] U_BB --> U_LakersFan["89: 重度湖人球迷画像"] end ContentSpace <--> |共享词表 & 拓扑遍历召回| UserSpace ``` - **双树共享词表**：内容与用户在统一离散拓扑空间中表达，彻底统一了供给端与需求端的空间表征。 - **算力与存储双降**：整数索引替代浮点向量极大压缩了显存与带宽占用，解决了海量参数吞吐瓶颈。 - **全链路复用**：构建高鲁棒性、高性能的前缀树索引，能贯穿支撑从召回（Retrieval）到排序（Ranking）的全套推荐管线。 --- ### [07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) 基于语义树检索：协同过滤、兴趣画像与探索机制 依托双前缀树结构，推荐系统的召回策略（Retrieval Candidates Generation）被优雅地统一为树上的图遍历（Tree Walks）操作。传统的复杂召回链路被拆解为多种基于前缀树的极速寻径逻辑： 1. **用户协同召回（User-to-User CF）**：在用户树中定位与当前用户处于相邻叶节点或拥有高度相似前缀路径的邻居群体，直接提取他们高频消费的内容子树作为候选。 2. **历史前缀利用（History Prefix Exploitation）**：统计用户在过去会话、单日或单月内消费内容的 Semantic ID 前缀分布，建立用户多维兴趣画像（Interest Profile），并在内容树相同前缀分支下快速捞取匹配内容，实现深度消费意图的精准满足。 3. **可调探索旋钮（Exploration Dial）**：为了打破信息茧房，当检测到用户长时间停留在某个极细叶节点（例如反复观看“湖人队”视频）时，算法可主动向上回退 1 到 2 个层级（回退到“篮球”乃至“体育”大类），并在同级邻近分支（如“冰球”或“水上运动”）进行横向探索，从而根据回退步数动态调控推荐结果的新颖度与多样性。 4. **多样性与去重控制（Deduplication & Diversity）**：利用前缀一致性，在召回与重排阶段强制设定规则，例如规定带有相同前缀序列的内容之间必须间隔 $N$ 个候选位，避免信息流出现高度同质化的刷屏。 5. **实时流式热点捕捉（Streaming Trend Detection）**：系统实时监控内容树各前缀节点在过去 5 分钟、10 分钟或 15 分钟内的全局消费激增情况，一旦某个前缀节点的访问流量发生突变，便可毫秒级捕获新兴突发热点并直接注入召回流。 - **树遍历统一召回源**：将协同过滤、强相关利用、弱相关探索与去重多样性转化为标准化的树搜索算子。 - **兴趣画像动态化**：通过截取不同长度的 Semantic ID 前缀，随心定制从宽泛到极致精细的用户兴趣分布比例。 - **受控探索与实时热点**：通过树层级回退机制实现多样性平滑调节，结合流式前缀流量监控秒级捕捉全网突发热点。 ### [10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) 实时趋势检测与基于前缀树的内容安全阻断 通过监控流经树中各个节点的历史数据量与实时流量突增，系统能够以极高灵敏度捕捉正在爆发的实时趋势。以世界杯决赛为例，如果平台上涌现出海量与阿根廷队及梅西相关的互动内容，监控系统会发现特定前缀分支的流量基线发生剧烈偏移：阿根廷相关节点的流量显著飙升，而梅西对应的节点更是直接冲上历史峰值。通过将流式互动的突增与历史基准体积比对，系统无需依赖复杂的文本理解模型，仅凭识别树上的“热点节点”（hot nodes）就能快速锁定当前的全网爆发趋势。 同样的树形分层结构还可以高效应用于内容风控与违规拦截。当安全审核系统发现某类敏感违规内容时，可以通过回溯 Semantic ID 树定位高风险节点——例如当发现流经某个特定前缀节点的内容有超过 50% 均属违规内容时，系统便可以直接在该节点层级（或向上/向下浮动一级）实施批量熔断拦截。平台工程师能够通过调整阻断所在的树层级，灵活调节对违规内容的查准率（Precision）与查全率（Recall）。 - **流式趋势捕捉**：对比节点历史基线与流式互动吞吐量，快速识别突发热点（如世界杯、阿根廷、梅西）。 - **前缀级策略阻断**：当某一树分支的违规内容占比超过阈值（如 50%）时，直接在对应前缀层级实施拦截阻断，实现安全策略的批量快速生效。 ```mermaid flowchart TD accTitle: 基于 Semantic ID 树的趋势识别与安全管控流程 subgraph StreamMonitoring ["流式监控与热点识别"] A["实时用户互动流"] --> B{"比对历史流量基线"} B -->|"流量显著激增"| C["锁定热点节点 Hot Node<br/>（如：世界杯 / 梅西）"] C --> D["触发实时热点推荐流量加权"] end subgraph PolicyControl ["前缀级安全风控"] E["内容安全审核信号"] --> F{"评估节点违规比例<br/>（如违规内容 > 50%）"} F -->|"高危节点分支"| G["在特定前缀层级执行批量阻断"] G --> H["根据业务调整层级<br/>平衡 Precision / Recall"] end ``` --- ### [12:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=724s) 结合前缀共享的冷启动扶持与生态平衡 在传统的推荐系统中，热门内容往往因为马太效应和流行度偏差（Popularity Bias）占据绝大部分分发曝光，导致新内容极难获得冷启动机会。在 Semantic ID 树中，同一叶子或前缀节点下往往聚集着语义相似的内容——例如同属于洛杉矶湖人队（Lakers）的内容中，既有已经极度病毒式传播的爆款视频，也有刚发布的新内容。 为了缓解冷启动困境，推荐系统可以主动对高曝光的热门内容实施部分替换，转而分发具有相同前缀特征的潜力新内容。系统在分发新内容时会施加质量校验与互动率预估（如 $p(\text{Like})$ 预估值），一旦新内容展现出优异的互动转化表现，系统便顺势将其权重拉升，推动其成为新的热门内容。这种机制让平台能够精细化调控冷启动分发杠杆，实现创作者生态的健康平衡。 - **流行度偏差替代**：识别热点节点下的高曝光爆款，主动用具有相同 Semantic ID 前缀的新内容进行替换试探。 - **质量预估与梯度扶持**：结合互动概率预估（$p(\text{Like})$）进行质量把控，表现优异的冷启动内容可快速晋升为热门推荐。 --- ### [12:56](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=776s) 下游排序的多样性打散与去重机制 在召回与排序之间的去重（Deduplication）与多样性（Diversity）控制环节，Semantic ID 的分层前缀提供了极简且强大的结构化工具。系统可以实时扫描召回候选池中的前缀重合度，若发现召回结果中有 50% 的内容都共享同一个 3 级或 4 级前缀，便意味着召回内容存在严重的同质化倾向。 为了避免用户的最终信息流中"顶部连续刷出四条湖人队视频"，系统可以在重排层利用前缀分布进行硬性或软性约束打散，强制在信息流中穿插体育、政治、园艺等不同领域的内容，实现自然的主题多样性控制。 - **多层级前缀去重**：通过设定 3 级或 4 级前缀的重复上限，防止单一垂直话题在候选池中过度聚集。 - **信息流多样性保障**：利用前缀类别约束保障最终 Feed 流涵盖体育、政治、生活等多元主题，消除审美疲劳。 --- ### [13:35](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=815s) 排序模型中的 Semantic ID：生成式推荐与判别式建模 将 Semantic ID 引入下游排序模型时，主要存在两种不同的范式路径，其中收益最显著的是**生成式推荐模型（Generative Recommendation）**。由于 Semantic ID 本质上是一串离散的层次化 Token，大语言模型或自回归架构可以像理解自然语言 Token 一样原生（natively）消费并生成它们，直接将推荐任务转化为离散 Token 的自回归解码生成。 而在传统的**判别式多目标排序模型**（例如联合预估点赞率 $p(\text{Like})$、互动率 $p(\text{Engagement})$、收藏率 $p(\text{Save})$ 的模型）中，Semantic ID 则作为输入特征使用。模型需要为每个码本（Codebook）层级构建离散 Embedding 表，并在判别式训练过程中持续联合更新，再将多层码本的 Embedding 进行拼接或融合，以此为多目标打分提供结构化的语义先验。 - **生成式推荐的原生契合**：生成式模型直接将 Semantic ID 视为语言词表中的离散 Token，端到端完成自回归生成。 - **判别式模型的多码本表征**：在多目标判别模型中为多层码本分别维护 Embedding 并在训练中联合优化。 --- ### [14:31](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=871s) 生产权衡：量化损失、碰撞陷阱与表达力折损 尽管 Semantic ID 带来了巨大的结构化红利，但工业级落地必须客观评估其带来的系统性权衡（Trade-offs）。首先，残差量化（Residual Quantization）本质上是有损压缩，不可避免地会损失一部分连续向量的精细信息。不过在实际生产体系中，稠密向量（Dense Embeddings）与 Semantic ID 并非互斥关系，两者各自提供独特的价值维度：前者保留连续空间的几何精度，后者赋予离散空间的可解释树形结构。 其次，离散树形量化会削弱模型的细粒度表达能力。在 512 维浮点数构成的连续向量空间中，两个不同内容发生精确碰撞的概率基本为零；但在离散码本空间中，不同内容被映射到相同叶子节点（Collision）的概率显著上升。针对碰撞率的优化存在明显的边际效用递减——过度追求极低碰撞率会使得码本设计过于庞大臃肿，导致系统收益锐减，因此在工业实践中切忌陷入"过度优化碰撞率的陷阱"。 | 维度 | 传统连续稠密向量（Dense Embeddings） | 离散层次化语义标识（Semantic IDs） | |:--- |:--- |:--- | | **空间表示** | 连续浮点空间（如 512 维 Float） | 层次化离散码本序列（Discrete Tokens） | | **碰撞概率** | 碰撞概率几乎为 0 | 存在受控碰撞率（需平衡粒度与码本大小） | | **可解释与检索** | 依赖高维向量近似最近邻（ANN）检索 | 支持前缀树遍历、实时聚合与规则拦截 | | **生成式适配** | 需额外映射层转换为生成目标 | 天然作为语言 Token 原生消费与自回归生成 | --- ### [15:57](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=957s) 工业化落地挑战：码本坍缩、数据漂移与全链路版本迭代 在生产环境中实现 Semantic ID 架构的工程化落地面临数个深层挑战。首要问题是**码本坍缩（Codebook Collapse）**：由于内容分布不均及平台固有的流行度偏差，向量量化聚类时极易出现大量向量扎堆在少数聚类中心、而其余聚类中心失活的现象。工程上必须通过精细的 K-means 初始化与聚类平衡策略，确保码本在各层级分布均衡稳定。 更为严峻的挑战在于**数据漂移（Data Drift）与版本迭代（Versioning）**。平台每天都有海量新内容涌现，当内容生态发生膨胀或分发倾斜时，整个链路（从分词器 Tokenizer、召回树构建、重排去重到下游排序模型）都需要持续重新训练以捕捉最新分布。 一旦需要调整底层分词器、扩展码本数量或修改树的深度与宽度，就会引发灾难性的全链路重构危机：底层 ID 定义一变，上层所有依赖该 ID 的检索索引、多样性策略和排序模型都必须同步重新训练并完成版本对齐。团队必须建立严密的数据 CI/CD 自动化流水线与防护栏（Guardrails），并在初期架构设计时审慎评估平台的数据漂移速率与内容增长预期。 - **码本坍缩防范**：通过精心设计的聚类初始化（如优化的 K-means 初始化）避免码本向量分布不均和节点失活。 - **碰撞候选决断**：制定清晰的仲裁策略，确保在不同内容发生相同 ID 碰撞时能精准选出最佳候选。 - **全链路版本协同**：底层码本或树结构调整会穿透影响召回、去重及排序全流程，必须依赖严格的数据 CI/CD 自动化与模型联合回训机制。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) 由于 ID 之间缺乏固有的语义关联，模型被迫去“死记硬背”（Memorization）海量 ID，导致 Embedding 表急剧膨胀并占据绝大部分模型参数预算 — Aditya Gautam > [00:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=4s) 将连续高维向量量化为层次化离散 Token 序列，实现从“机械记忆”到“语义理解”的跨越。 — Aditya Gautam > [02:13](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=133s) 通过在损失函数中引入协同过滤损失（Collaborative Loss），系统可以让经常被同批用户连续点击、浏览的内容在码本空间中聚拢，从而将行为协同信号无缝注入离散语义编码中。 — Aditya Gautam > [05:52](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=352s) 推荐系统的底层基座演进为由共享词表驱动的“内容树”与“用户树”双树架构。 — Aditya Gautam > [07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) 依托双前缀树结构，推荐系统的召回策略（Retrieval Candidates Generation）被优雅地统一为树上的图遍历（Tree Walks）操作。 — Aditya Gautam > [07:41](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=461s) 当检测到用户长时间停留在某个极细叶节点（例如反复观看“湖人队”视频）时，算法可主动向上回退 1 到 2 个层级（回退到“篮球”乃至“体育”大类），并在同级邻近分支（如“冰球”或“水上运动”）进行横向探索 — Aditya Gautam > [10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) 系统无需依赖复杂的文本理解模型，仅凭识别树上的“热点节点”（hot nodes）就能快速锁定当前的全网爆发趋势。 — Aditya Gautam > [10:49](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=649s) 当发现流经某个特定前缀节点的内容有超过 50% 均属违规内容时，系统便可以直接在该节点层级（或向上/向下浮动一级）实施批量熔断拦截。 — Aditya Gautam > [12:04](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=724s) 推荐系统可以主动对高曝光的热门内容实施部分替换，转而分发具有相同前缀特征的潜力新内容。 — Aditya Gautam > [13:35](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=815s) 由于 Semantic ID 本质上是一串离散的层次化 Token，大语言模型或自回归架构可以像理解自然语言 Token 一样原生（natively）消费并生成它们，直接将推荐任务转化为离散 Token 的自回归解码生成。 — Aditya Gautam > [14:31](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=871s) 稠密向量（Dense Embeddings）与 Semantic ID 并非互斥关系，两者各自提供独特的价值维度：前者保留连续空间的几何精度，后者赋予离散空间的可解释树形结构。 — Aditya Gautam > [15:57](https://www.youtube.com/watch?v=hV0RNjnWLYA&t=957s) 底层 ID 定义一变，上层所有依赖该 ID 的检索索引、多样性策略和排序模型都必须同步重新训练并完成版本对齐。 — Aditya Gautam ## Mentioned Meta, Aditya Gautam, Google, Ai4, Semantic ID, RQ-VAE, LLM, K-means, 洛杉矶湖人队 (Lakers), 阿根廷队, 梅西 (Messi) ## Video description Aditya Gautam - Machine Learning Technical Lead Meta Item embeddings are hitting real limits: embedding tables that dominate the parameter budget, brittle cold start, and representations that force models to memorize instead of generalize. This talk traces the shift to Semantic IDs — short, hierarchical sequences of discrete codes derived from content and learned with residual quantization. This talk will briefly talk about how they're built, then make the case that the interesting move isn't tokenizing items, it's building two trees: one over content, one over users, sharing a vocabulary. From that foundation, seven retrieval sources fall out as tree walks — user-to-user neighborhoods, history-prefix expansion, a tunable exploration dial, prefix-frequency interest profiles, streaming trend detection, cold-start substitution, and prefix-level policy blocklists. Downstream, the same codes serve as ranking features and as generation targets. The talk concludes with a pragmatic production architecture and an honest assessment of quantization loss, code collapse, semantic–behavior misalignment, identifier churn, privacy, and constrained-decoding latency. Semantic IDs do not eliminate dense embeddings; they turn embedding geometry into reusable structure. [Source](https://www.youtube.com/watch?v=hV0RNjnWLYA)

Source quotes

[0:04] Aditya Gautam: 由于 ID 之间缺乏固有的语义关联，模型被迫去“死记硬背”（Memorization）海量 ID，导致 Embedding 表急剧膨胀并占据绝大部分模型参数预算 [0:04] Aditya Gautam: 将连续高维向量量化为层次化离散 Token 序列，实现从“机械记忆”到“语义理解”的跨越。 [2:13] Aditya Gautam: 通过在损失函数中引入协同过滤损失（Collaborative Loss），系统可以让经常被同批用户连续点击、浏览的内容在码本空间中聚拢，从而将行为协同信号无缝注入离散语义编码中。 [5:52] Aditya Gautam: 推荐系统的底层基座演进为由共享词表驱动的“内容树”与“用户树”双树架构。 [7:41] Aditya Gautam: 依托双前缀树结构，推荐系统的召回策略（Retrieval Candidates Generation）被优雅地统一为树上的图遍历（Tree Walks）操作。 [7:41] Aditya Gautam: 当检测到用户长时间停留在某个极细叶节点（例如反复观看“湖人队”视频）时，算法可主动向上回退 1 到 2 个层级（回退到“篮球”乃至“体育”大类），并在同级邻近分支（如“冰球”或“水上运动”）进行横向探索 [10:49] Aditya Gautam: 系统无需依赖复杂的文本理解模型，仅凭识别树上的“热点节点”（hot nodes）就能快速锁定当前的全网爆发趋势。 [10:49] Aditya Gautam: 当发现流经某个特定前缀节点的内容有超过 50% 均属违规内容时，系统便可以直接在该节点层级（或向上/向下浮动一级）实施批量熔断拦截。 [12:04] Aditya Gautam: 推荐系统可以主动对高曝光的热门内容实施部分替换，转而分发具有相同前缀特征的潜力新内容。 [13:35] Aditya Gautam: 由于 Semantic ID 本质上是一串离散的层次化 Token，大语言模型或自回归架构可以像理解自然语言 Token 一样原生（natively）消费并生成它们，直接将推荐任务转化为离散 Token 的自回归解码生成。 [14:31] Aditya Gautam: 稠密向量（Dense Embeddings）与 Semantic ID 并非互斥关系，两者各自提供独特的价值维度：前者保留连续空间的几何精度，后者赋予离散空间的可解释树形结构。 [15:57] Aditya Gautam: 底层 ID 定义一变，上层所有依赖该 ID 的检索索引、多样性策略和排序模型都必须同步重新训练并完成版本对齐。

Source quotes

[0:04] Aditya Gautam: Item embeddings are hitting real limits: embedding tables that dominate the parameter budget, brittle cold start, and representations that force models to memorize instead of generalize. [0:04] Aditya Gautam: Rather than learning true underlying semantic relationships, legacy embedding tables force recommendation models into brute-force memorization. [0:04] Aditya Gautam: Under this paradigm, content is no longer indexed as opaque, high-dimensional floating-point vectors, but as structured sequences of discrete tokens that explicitly encapsulate hierarchical semantic concepts from coarse to fine granularity. [6:34] Aditya Gautam: The power of Semantic IDs becomes fully realized when the exact same tokenization and prefix tree principles are applied symmetrically to users. [7:41] Aditya Gautam: With dual prefix trees established as the foundational layer, candidate generation across the recommendation funnel simplifies into structured tree-walk operations. [10:49] Aditya Gautam: Tracking engagement through hierarchical Semantic ID tree nodes transforms complex real-time event monitoring into a straightforward volume-anomaly problem. [12:56] Aditya Gautam: Generative recommendation models achieve the most substantial performance gains because they treat Semantic IDs natively as language tokens, auto-regressively predicting next-item codes using standard token-generation techniques. [14:29] Aditya Gautam: The vector quantization step is fundamentally lossy; converting rich 512-dimensional floating-point vectors into a compact sequence of discrete indices inevitably sacrifices continuous geometric nuance. [14:29] Aditya Gautam: In modern production environments, discrete Semantic IDs and continuous dense embeddings do not replace one another; rather, they coexist symbiotically to balance broad retrieval coverage with fine-grained scoring. [14:29] Aditya Gautam: Engineers must avoid the 'collision rate trap,' where excessive engineering effort is spent eliminating harmless low-level leaf collisions at the expense of system latency and simplicity. [15:57] Aditya Gautam: Modifying upstream semantic token definitions immediately invalidates downstream tree walks, candidate generation indexes, diversity rules, discriminative feature stores, and generative ranking models. [14:29] Aditya Gautam: Semantic IDs do not eliminate dense embeddings; they turn embedding geometry into reusable structure.

Original transcript

[0:04] Next 20 minutes we will be talking about semantic  ID and we will be getting into understanding of [0:09] what this is, how to build it and how we can use  the semantic ids uh as raw signals and as a tree [0:15] prefix to build the entire layer of recommendation  stack all the way from foundational user content [0:21] understanding build the retrieval system the uh  items and other thing that you like uh in the [0:27] past like how do we retrieve using semantic ids  and then how do we use this thing in the ranking [0:31] models and diversity layer down the line. So we  will be going through entire semantic ids and [0:37] how this can be leveraged into recommendation  system. So to begin with we have a problem in our [0:43] typical embedding systems where we have a uh items  uh are presented from a big corpus in terms of ids [0:52] which does not have any inherent meaning. So and  also the problem is that like the cold distort uh [1:01] problem where when you have the embedding learns  during co-raining along with the model you see [1:06] that like some of the items which are really  which which are used very less does not really [1:12] have a very good embedding representation and the  items which are used a lot where there's a lot of [1:17] flickrough popularity bias those items learn their  embedding really well. So there is a cold start [1:22] problem and then we have a problem with respect  to memorization. So there comes a semantic id [1:28] where fundamentally how you we are presenting  a content is not like in terms of embedding [1:34] but in terms of discrete token and these tokens  are the one which are basically representing some [1:40] semantic and some topic or some granular or a core  level understanding about what it is. For example, [1:46] 41 could be a sports and then 12 could be a  basketball and then 89 could be a Laker team [1:53] or something which is representing the content  in their understanding of their uh semantic [1:59] meaning as compared to the embedding which is  purely uh number of floats. So it's herical [2:06] it is can be consumed as a token in in a native  language by LLMs and this is much more intuitive. [2:13] So uh this is a seminal paper from Google which  actually introduced the semantic embedding. I [2:18] would not be getting into details on this.  We can you can read online. There are a lot [2:22] of good blogs and the paper and everything out  there. Uh but at a high level we have a embedding [2:27] uh which is out from encoder of a content user  or whatever the entity you are representing. [2:33] It goes through a different level of cookbooks.  Consider each cookbook as like a one uh high level [2:39] uh understanding of uh topic or something.  uh the clustering or some some uh [2:45] co- representation of a content and as  you go down it's get more fine grained. [2:50] Um and what it does it basically how we train  this cookbook is that like we take the embedding [2:56] we look into the closest near nearby embedding of  a cookbook we assign that index and then we take [3:02] a residual from there and go back to the next and  we keep on iteratively doing that and that's where [3:07] when we index map from each cookbook we get  the ID specific ID corresponding to something. [3:14] So the concept or the underlying algorithm is the  residual quantization where we encode the content [3:20] have a cookbook and just iteratively keep on doing  that and uh learn the representation of a content [3:27] and here the content does not always need to  be semantically or something which is visually [3:32] similar. It could be something where the people  who are clicking the same items are actually [3:37] uh having a collaborative filing and we can  change the loss function to tune it into [3:41] any way possible. If you just have a collar  collaborative loss, it will find the content [3:47] which are actually clicked or liked by the people  in the same uh sessions or same domains like that. [3:54] Uh once we are uh done with this building  tree, there are two main factors when we are [3:59] building this whole semantic ids. One is the  depth that means number of cookbooks you want [4:04] and how breadth of a cookbook you want. uh just  to give an example let's suppose if you are [4:08] having a cookbook at at a level of 52 or like  something 64 you would have one sports ID but [4:14] if you extend it to 1 124 or something then  you would probably have water sports hockey [4:20] or like the athletic sports or something  top level category at cookbook level one. [4:26] So that's how you can change it and this will  change how what is a uh generalization happening [4:31] what is a memorization happening how course and  find you would want your content to be represented [4:37] and once you are able to do this we can use  this for anything any entity like for in [4:43] platforms we can have a content a video or kind  of like a textbook anything which is represented [4:50] together through using multimodality all the text  image video audio related feature goes into their [4:56] individual encoder. We get a representation  of an uh individual modality encoder [5:02] and once we do that we do different  kind of fusions which are possible in u [5:07] like a typical multimodality learning which is  early fusion and late fusion how you want to [5:12] interact those embeddings and once we get a one  finalized embedding including all the multimodal [5:16] content that's where we can apply this same  technique get the semantic ids related to this [5:22] uh one of the big advantage of using one modality  versus multimodality is that it allows you to have [5:28] a much richer and a better representation that  reduces your collision. Collision is basically [5:33] when two items which are kind of like similar but  they have the sim same semantic code uh at the end [5:39] which shouldn't be the case because uh you would  want every item to be represented if you uniquely [5:46] uh but it happens because we have a constrained  uh cookbooks numbers and we have lot more items [5:52] and once you are able to build the semantic ids  the next item to build the foundational layer is [5:56] to build the prefix tree. You take all the corpus  you have, you build it based on the prefix and [6:01] from the prefix you go to the next level and the  next level. For example, it could be the sports [6:06] then basketball then the Lakers and then a  particular things about within a Laker team. So [6:13] we are able to gr uh go from the root to a  specific content going into more granularity [6:18] and this is something very fundamental for  building the recommendation systems and search [6:23] recommendation because once you are able to build  this tree and prune it and build it in a nice [6:28] way with a higher performance you can use it for  entirely uh through entire recommendation stack [6:34] all the way from retrieval to ranking systems. So  how we did with the uh content we we would do the [6:41] same thing with the users. Here we are looking  into the long-term history, the static features, [6:44] the current history, engagement, clickth through  all everything whatever the you think can be [6:50] represented by the users and then we give it to  the encoder and we generate the representation [6:56] and we build the same tree for the user and the  content. So now we have these two trees which are [7:02] actually built as our foundational model. uh  to give an little bit of latest understanding [7:07] from the Google research paper. This happened  like last uh month only like two months ago [7:12] where they uh when they built this uh semantic  user ID they were able to see the metrics up [7:19] in all aspect all the way from engagement  the bite per user consumption has gone down [7:24] and the cold start problem has reduced a lot with  that the training and the serving compute has also [7:29] gone down because you are representing some bunch  of integer rather than that entire big float which [7:34] is much more computationally expensive and it's  also expensive from a storage perspective. So [7:41] once we have this foundation thing from the  retrieval sources you can think of these are [7:47] some of them laid out but there are much more  ways to re to retrieve the content. If you like a [7:52] particular content let's suppose about a football  about like a gardening or anything how does the [7:58] retrieval system figure out like what should be  the right retrieval uh candidates you have. So for [8:03] example user user is basically the users which are  similar in nature what they have liked and their [8:09] other similar users have liked pick those  candidate this is collaborative filtering [8:13] then we have a history prefix which is more  about building the user interest profile and [8:17] understanding all the content the users has  liked in last session last day last month and [8:22] then finding all those content which are having  the same prefix that means more exploitation [8:28] then we have exploration where  you don't have to go to the [8:31] exact the bottom where the user or her hot tree  or like liking the nodes you go one to level up. [8:37] For example, from basketball you go to the sports  and then you go to the other node and find the [8:40] hockey and re recommend and fetch those candidates  for exploration. Similar to that we can figure out [8:46] the trends which we will be talking about and once  we are able to do this we can use the semantic ids [8:50] and other aspect like dduping what is a prefix  thing. All the content all the 10 20 100 content [8:55] which is retrieved by retrie layer. How many of  them are having the same prefix diversity? We [9:01] want to have some diversity rule that means after  every once the contents of a particular prefix is [9:06] shown we want to have a gap of n candidate so that  it is not very repetitive in your feed. So how do [9:11] we do that like the same thing uh semantic ids  is another way to kind of like build this thing. [9:17] So uh going to the user collaborative and  the interest profile, we look into the user, [9:22] we go down the tree, we look into what the  user is doing, where the tree lies and what [9:26] are the similar user, what what are the videos  and other thing that they have lived or what [9:31] are the interest that they have this current  user has actually missed and use that to fetch [9:35] new candidates. Similar to that interest profile  user has been what is a percent of consumption or [9:41] engagement of user on different categories all the  way from sports to basketball or politics or funny [9:48] engagement or any kind of videos which uh the user  is uh consuming. We can build a a profile just [9:55] based on looking into user history extracting the  semantic ideas of all the content and basically [9:59] having a particular prefix length to see what is  the right granularity we want. uh exploration as [10:05] we talked about like we know that okay user has  been consuming this node a lot like uh there has [10:10] been like all the node and anything below that  like is all basketball Lakers video that's user [10:15] has been consuming it let's go to two level up  go to the other side of the tree and see like why [10:22] what can we do to actually make this user feel  more diverse what is the higher diversity of [10:27] the feed that's where the exploration can be  tuned up and down based on how many nodes you [10:32] want to go and which path of the tree would want  to take based on the uh prefix tree that we have [10:37] built. Um trend how do we detect a trend? So  we look into the volume when from the tree we [10:44] look into the volume which is happening in less 5  minutes 10 15 minutes and we see that what is the [10:49] historical volume of the data passing through each  nodes and we would be seeing there is a bump which [10:54] happens for example during the world cup final  Argentina and Messi related like if there are a [11:00] lot of content which is basically representing uh  which are liked or engaged by the users we looked [11:06] into that and we'll say that okay the world cup  final has baseline has gone like much higher as [11:11] compared to what it was historically. Argentina  has gone considerably way higher and the Messi [11:16] has been like the highest ever possible. So that  means there is some trend which is happening with [11:20] the world cup Argentina and Messi. That is how we  can detect the trend based on the historic volume [11:26] and what is happening in the engagement which is  happening in a streaming fashion and like just [11:30] understanding the hot nodes out there. So yeah  similar to that we can do the cold start problem [11:37] and uh block the content. So if there is some  policies or something where the varation is [11:41] happening we know that like okay this is the part  of the tree where we are seeing a lot more 50% of [11:46] the uh content semantic ID passing through this  nodes are actually blocked content. So let me [11:52] try to actually block it at this level or one  level below up depending on how uh you want [11:57] to actually change the precision and recall of  those uh sensitive content and cold start. So [12:04] like we talked about the hot node here which  is out there and if let's suppose there in that [12:10] in a particular tree there are bunch of like  these content which are available which are [12:14] more or less same leakers or something there are  some new content and there is some really viral [12:19] content out there which is recommended because of  the popularity bias in the recommendation system. [12:24] Uh one of the way to kind of like um remove  this or mitigate this cold start problem is to [12:30] recommend the content which is similar in nature  which is having certain prefix uh commonality [12:36] and rather than having the popular content  recommend this do a quality check and make [12:39] sure if this content is very well engaged it has  a good P like or something personal type then just [12:45] up this content and bring it more popular. So  that's where the ecosystem balancing can happen [12:50] and we can figure out what is uh the right way  to actually tune this uh coldest start problem. [12:56] Okay, we talked a lot about retrieval. We know in  the DD we can use certain prefix and see that okay [13:01] there 50% of the retrieve content is belonging to  the same three-level prefix or fourle prefix. I [13:07] probably do not want it to be this uh repetitive.  I can change it in the ranking. We will be talking [13:13] about how we will be using this in the diversity.  The same thing we can look into the particular end [13:18] level of prefix. We say that in diversity final  feed we would want like a sports content. We would [13:23] want to have politics or some gardening or some  other kind of thing out there. We do not want like [13:28] a laker video coming four times in your feed on  the top. So that's one way to actually do this. [13:35] And in the ranking model there are two ways to  use this thing. Uh the biggest gain uh is usually [13:41] in the generative recommendation because they  understand these semantic ids as token natively [13:47] because of the native nature. They can consume  it as if like it is just another language token [13:52] and they can generate this token. They can  figure out what are the right ways to do it. [13:56] They understand it natively. So it's much  better. You we can use this also in a [14:01] discriminative models like which are having a  multi- uh objective problem like the P likes, [14:06] P engagement, P saves and something like that.  But here we have to take like the uh each co [14:15] cookbook and then represent and learn an embedding  throughout when we are actually learning that [14:20] discriminative model and then there could be  one cookbook there could be n and there could [14:24] be multiple ways to actually club these together  to find the embedding in a discriminative way. [14:29] Okay. So uh we talked about like the all  the way from uh ranking recommendation [14:36] um the codes how we can use this thing all the  retrieval sources we have but there are of course [14:42] uh trade-off associated with everything. Uh so  when we are doing the quantization in the initial [14:47] stage it's lossy there is some information  loss but there is also a certain information [14:52] gain which is happening. So in production system  what you usually would be seeing is that embedding [14:58] and semantic ids has its own uniqueness. There  is something which both of them are bringing [15:03] on the table which is useful for the content to  discover and try to figure out what is happening. [15:08] The second thing is that um this tree thing  reduces the model expressiveness and basically [15:14] it just kind of like created some uh not  very granular um understanding of the [15:22] content like for example embedding you have like  a decimal point and you have like 512 of those [15:27] float values out there the the prop probability of  collision or having the same thing is more or less [15:32] none in that that space. So yeah and the  third thing is the collision uh uh rate trap [15:39] which is basically we try to optimize that there  is a limited collision but after a certain point [15:44] we see that like the value on the collision  uh optimizing this collision goes down and um [15:50] that's where uh we don't want to overoptimize this  uh parameter also. Okay. Uh and then talking about [15:57] like uh some of the variable factor like the  code uh stability um is in and like how do we [16:03] productionalize this kind of like system is  that uh first we look into the codebook collapse [16:09] which is basically each code book has like bunch  of vectors what which we representing a particular [16:16] uh semantic uh topic or something like that.  But uh what we are see that like because of the [16:22] popularity bias because of the dis uh disbalanced  content distribution that clustering or that [16:28] uh understanding at the cookbook level is not  very balanced. So that's where we can do a [16:33] bunch of thing like a proper kin initializations  of those vectors and everything to figure out [16:37] what is the right way to build a right stable  and a little bit of balance cookbook and then [16:43] uh collision is the second thing that we talked  about like we have to figure out like what are [16:47] the distinct item if the collision happen what  would be the right way which would be the right [16:51] candidate to be picked up and then there's a drift  which is happening in production because we have [16:56] a lot of new content coming up and when we have a  new content coming up uh the whole recommendation [17:02] system needs to be like continuously trained all  the way from recommendation ranking, diversity [17:07] layer everything to make sure we are able to  actually do something very fresh and understand [17:12] the drift which is happening and the last point  which is the most complicated in production is the [17:17] versioning. So we have built the semantic ids we  are using it like we have built the trees we are [17:22] using it a retrieval diversity layer ranking layer  everything out there. So when you want to change [17:29] any of the one thing let's say tokenizer or you  want to have more cookbook because you know that [17:33] your content is growing a lot and you would want  to probably change the depth and width of this [17:38] that means the versioning needs to happen at all  the places it needs to happen at the semantic how [17:44] you are building the semantic ids how you are  building the trees what are the foundational [17:49] layers how it is happening and once it is  happening like what are the u mechanism to [17:55] make sure we are having the proper guard rails to  make sure this whole CI/CD CI uh data everything [18:01] is happening in a right manner so that whenever  there is some change at the underlying layer [18:06] all the models and everything are kind of like  are trained and retrained in the right manner. [18:11] This is a really really hard problem and that's  where you would want to think through before [18:16] building the semantic ID what is the drift in your  data. How how uh do you think that like you would [18:21] want to change the semantic ID because there is  explosion of content or something on your platform [18:26] and you want to versioning it better. So  uh just keeping this in mind these are [18:30] some of the ways to actually be careful. Um  u thank you very much really appreciate it.