Meta Halves Dubbing Latency to Deliver 500M Weekly Translated Reels

Content Accessibility at Scale: AI Voice Translations at Meta | Esra Cansizoglu from Meta — @Scale

@Scale

Deploying joint speech recognition and language-agnostic retrieval allows Meta to bypass historical user silos and serve foreign-language creators to global audiences under a single video identifier.

AI-assisted summary of Content Accessibility at Scale: AI Voice Translations at Meta | Esra Cansizoglu from Meta — original source. Check the original for full context.

Source published:

Key ideas

  • Meta now serves over 500 million views on voice-translated Reels each week across Facebook and Instagram, spanning more than 15 languages. (0:05)
  • The engineering pipeline replaces cascaded ASR-to-machine-translation architectures with Joint Speech Translation and Recognition (JST&R), slashing first-token latency by 50% through synchronized fast and slow context encoders. (8:27)
  • Meta maintains a single video ID architecture across all localized audio tracks, requiring recommendation retrieval to decouple semantic interest from spoken language to avoid cold-start silos. (13:47)
  • The retrieval layer balances exploration and exploitation across billions of candidate videos using three primary pillars: interest profiling, dense item embeddings, and cross-lingual user-to-user taste clustering. (16:35)

Detailed notes

## Background This talk was published on YouTube by the @Scale channel on October 30, 2026, featuring Meta machine learning engineer Esra Cansizoglu. No additional episode description or show notes were provided with the upload.

## TL;DR Esra Cansizoglu outlines Meta's end-to-end AI voice translation system for Reels, which serves over half a billion translated views weekly across more than 15 languages. She explains how Meta pairs computer vision lip-syncing with Joint Speech Translation and Recognition (JST&R), a dual-encoder architecture that cuts first-token latency in half compared to cascaded models. The presentation details the infrastructure dilemma of indexing multiple translated audio tracks under a single video ID, which obscures downstream engagement metrics. To solve the resulting cold-start recommendation problem, Meta balances exploration and exploitation in the retrieval phase through interest profiling, neural content embeddings, and cross-border user taste clustering.

## Atmosphere & style The presentation strikes an engaging balance between relatable domestic storytelling and rigorous machine learning engineering. Cansizoglu grounds the complex topic with warm personal anecdotes about cooking foreign recipes with her children and analogies to Andy Weir's sci-fi novel *Project Hail Mary*, before transitioning into dense architectural schematics. Concrete video demos—from Sylvester Stallone's inflection-preserved dubs to a Portuguese shrimp ramen tutorial—give the technical breakthroughs an immediate, tangible resonance, culminating in an optimistic outlook on scaling cross-cultural communication.

## Bulletin - [00:05](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=5s) Meta now serves over 500 million views on voice-translated Reels each week across Facebook and Instagram, spanning more than 15 languages. - [08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) The engineering pipeline replaces cascaded ASR-to-machine-translation architectures with Joint Speech Translation and Recognition (JST&R), slashing first-token latency by 50% through synchronized fast and slow context encoders. - [13:47](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=827s) Meta maintains a single video ID architecture across all localized audio tracks, requiring recommendation retrieval to decouple semantic interest from spoken language to avoid cold-start silos. - [16:35](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=995s) The retrieval layer balances exploration and exploitation across billions of candidate videos using three primary pillars: interest profiling, dense item embeddings, and cross-lingual user-to-user taste clustering.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=5s) Over half of Facebook users regularly consume video content in multiple languages, despite available global video supply remaining overwhelmingly English-dominated. - [00:05](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=5s) Voice-translated Reels generate over half a billion views weekly across Meta platforms, preserving the original speaker's vocal timbre and cadence. - [03:26](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=206s) Production dubbing requires a parallel pipeline combining audio signal processing, generative voice cloning, and computer-vision-based lip-sync synthesis. - [03:26](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=206s) When creators produce content in new languages, recommendation systems face a severe cold-start problem due to a lack of historical language-specific engagement data. - [08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) CNN front-end feature extraction isolates primary speaker audio from challenging real-world acoustic noise like stadiums, wind, and overlapping dialogue. - [08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) Joint Speech Translation and Recognition (JST&R) simultaneously transcribes and translates, cutting first-token latency in half compared to cascaded pipelines. - [08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) JST&R employs a dual-encoder design: a fast encoder emits swift local translations, while a slow encoder pauses to buffer context for complex phrases. - [13:47](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=827s) Meta represents translated videos under a single video ID to simplify infrastructure, but this blends engagement signals across different language audiences. - [15:20](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=920s) The recommendation engine focuses algorithmic exploration at the retrieval bottleneck, narrowing billions of video candidates down to hundreds of thousands before ranking. - [16:35](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=995s) Content understanding models construct multidimensional interest profiles, separating user topical affinity from the spoken language of the video. - [16:35](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=995s) Item-to-item retrieval utilizes neural network embeddings as content signatures to identify conceptually similar media across language barriers. - [16:35](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=995s) User-to-user retrieval matches cross-lingual audiences based on shared consumption taste profiles, boosting distribution for lower-resource languages. - [18:37](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=1117s) Voice translation is bundled with automated comment translation so international viewers can participate seamlessly in native-language discussions. - [18:37](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=1117s) Ongoing engineering hurdles include modeling code-switching multilingual users, enhancing language prediction heuristics, and optimizing GPU compute costs.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=5s) Breaking Language Barriers: From Kitchen Recipes to Global Scale

Esra Cansizoglu, a machine learning engineer at Meta, opens her talk with a relatable domestic scene: spending time with her children in the kitchen trying out international recipes, only to find themselves completely stuck whenever a promising dish is written or spoken in a language they cannot understand. That personal friction mirrors a massive global opportunity on Meta's social platforms. As Cansizoglu explains, more than half of Facebook's active users regularly consume video content in more than one language. While viewers represent over 160 different linguistic backgrounds, available video content remains heavily skewed toward English. Bridging this imbalance requires taking brand-new content and making it effortlessly accessible to anyone in a language they understand.

Drawing an analogy to science fiction, Cansizoglu compares Meta's mission to Andy Weir’s novel *Project Hail Mary*, where the protagonists Ryland Grace and Rocky the alien bridged a total communication void to save their worlds: “Well, our version has fewer spaceships, but the idea is somewhat similar.” To demonstrate the technology in practice, she plays a Reel featuring actor Sylvester Stallone recounting his early days earning $34 a week as a movie theater usher, which he viewed as his real college education. Through Meta's voice translation pipeline, Stallone's speech is automatically delivered across multiple target languages while retaining his distinct vocal tone and inflection.

The scale of this operation is already vast. Cansizoglu highlights the reach the team has attained while articulating Meta's core mission: “Today we deliver more than half a half a billion views on voice translated reals every week.” She underscores that the ultimate objective is not merely algorithmic translation, noting: “And for us this is about removing language barriers so people can connect authentically from anywhere in the world.”

- Over 50% of Facebook users watch video content in more than one language. - Meta supports an audience spanning over 160 distinct languages, despite global video supply being heavily dominated by English. - Voice-translated Reels account for over 500 million views each week across Meta’s platforms.

[03:26](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=206s) The End-to-End Pipeline: Production, Lip-Sync, and Cold Starts

Building high-quality voice translation requires navigating two major bottlenecks: generating authentic content and distributing it to the right people. On the production side, synthetic speech must avoid sounding synthetic or robotic, the translation must preserve nuance, and video frames must be modified through computer vision lip-syncing so that the speaker’s mouth movements naturally match the translated phonemes. Once produced, the platform faces a severe recommendation challenge known as the cold-start problem. When an English-speaking creator suddenly releases audio in Hindi or Portuguese, platform recommendation algorithms cannot rely on past historical engagement. Cansizoglu describes the dilemma plainly: “A creator who was producing in English until now now all of a sudden starts producing in Hindi in Portuguese. Then the question is how do we find that Portuguese speaking viewers who would be interested in their content.”

The system relies on parallel generation pipelines blended before distribution. On the audio track, signal processing algorithms evaluate whether there is sufficient intelligible speech and check whether the source language is currently supported; if cleared, generative AI dubbing models synthesize the translated voice. Concurrently, computer vision models analyze the video stream, verifying face presence and visibility before synthesizing adjusted lip movements. Once blended into a finalized Reel, Meta's recommendation engine evaluates user-video relevance using language-aware heuristics. The user interface allows viewers to override choices, designate preferred languages, or mark languages they do not wish to translate. Cansizoglu emphasizes: “Through their efforts, users are able to change the language of the real or provide their preferred or do not translate languages.” When explicit viewer preferences are absent, ML inference models predict comfortable consumption languages automatically.

```mermaid flowchart TD accTitle: End-to-End Voice Translation Pipeline accDescr: Diagram illustrating the parallel audio and video production flows merging into distribution and language-aware recommendation.

subgraph Production ["Production Flow"] Input["Source Video & Audio"] Input --> SP["Audio Signal Processing<br/>(Speech check, language ID)"] Input --> CV["Computer Vision<br/>(Face detection & visibility)"] SP --> GenAI["Generative AI Audio Dubbing<br/>(Natural voice clone)"] CV --> LipSync["Lip-Sync Synthesis<br/>(Video re-encoding)"] GenAI --> Blended["Blended Voice-Translated Reel"] LipSync --> Blended end

subgraph Distribution ["Distribution & Recommendation"] Blended --> RecSys["Language-Aware Recommendation<br/>(Cold-start matching)"] RecSys --> Delivery{"User Preferences<br/>Available?"} Delivery -->|"Yes"| Direct["Render in User's<br/>Preferred Language"] Delivery -->|"No"| Fallback["ML Language Detection<br/>Fallback Model"] end ```

- Audio pre-processing screens out background sound and validates that content contains sufficient speech in a supported language. - Computer vision modules detect face orientation and mouth visibility to trigger neural lip-sync re-encoding. - UI preference tools let viewers choose specific playback languages or establish explicit "do not translate" lists. - Recommendation engines apply cold-start heuristics to pair newly translated creators with target language audiences globally.

[08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) Audio Generation and Dubbing: Joint Speech Translation and Recognition (JST&R)

Shifting focus to core dubbing architecture, Cansizoglu connects the project to the live translation demo seen on Meta Ray-Ban smart glasses, where Mark Zuckerberg conversed live on stage with a Spanish-speaking creator. Built on research spearheaded by Niko Moritz, Christian Fuegen, and their team (published at ICASSP), the technology adapts live conversational AI to video dubbing. While smart glasses prioritize absolute minimal latency, Reels dubbing emphasizes high throughput, consistency, and scalability for billions of viewers. In real-world media, acoustic environments are notoriously messy, contaminated by stadium crowds, overlapping conversations, concert music, and wind noise. Using a directional multi-channel microphone array and a convolutional neural network (CNN) front-end for feature extraction, Meta's streaming automatic speech recognition (ASR) architecture isolates individual speaker tracks from acoustic distractors.

Rather than relying on traditional cascaded architectures—where an ASR module finishes transcribing before feeding raw text into a separate machine translation engine—Meta utilizes Joint Speech Translation and Recognition (JST&R). This joint model processes speech recognition and translation simultaneously using dual slow and fast encoders. The fast encoder provides swift translations for unambiguous words where local context suffices, whereas the slow encoder waits for richer token history when sentence structure or idiomatic context demands it. Cansizoglu articulates the division of labor: “Fast encoder can quickly translate some words when there is no need for longer context.” She adds: “While with slow encoder, the system knows when to pause and look at more history of tokens to be able to provide accurate translation.”

Benchmarking reveals that this synchronized dual-encoder design matches or exceeds cascaded pipeline latency while cutting first-token latency in half without degrading translation accuracy. Cansizoglu ties these technical achievements back to real creator discovery by presenting a Portuguese cooking Reel. A chef preparing creamy ramen topped with *cucaracha*-style shrimp is rendered into natural, fluent English, preserving his original enthusiasm and tone so that non-Portuguese speakers can follow along step by step.

- Conventional cascaded systems run ASR independently before passing text to machine translation, compounding latency. - Meta's JST&R (Joint Speech Translation and Recognition) method handles transcription and translation simultaneously. - A dual-encoder mechanism splits tasks: the fast encoder emits immediate translations, while the slow encoder buffers context tokens to ensure semantic accuracy. - JST&R reduces first-token latency by 50% compared to baseline cascading models while maintaining equivalent or improved overall latency. - Multi-channel acoustic feature extraction powered by CNNs strips out background crowd, wind, and music interference to isolate the primary speaker.

[13:47](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=827s) Infrastructure Architecture and the Single Video ID Challenge

Following a brief cooking demo clip illustrating a voice-translated recipe reel—mixing black pepper, heating oil, and frying shrimp—Esra Cansizoglu steps back to address the core systems engineering problems behind distributing translated media. On the infrastructure side, Meta represents a voice-translated Reel using a single underlying video ID. While this architectural choice makes implementation cleaner by fitting neatly into Meta’s pre-existing infrastructure data models without requiring multiple duplicate media assets, it introduces severe side effects for downstream ranking and recommendation pipelines.

Recommendation algorithms at Meta capture user engagement signals at the individual video-user interaction level, which means machine learning models historically learn language preferences implicitly from watch history and interaction patterns. When a translated Reel collapses all localized audio tracks under one video identifier, every viewer's interaction—regardless of whether they listened in English, Spanish, or Hindi—gets blended into that single entity's engagement metrics. This makes creator reach across language barriers exceptionally difficult to unlock. As Esra Cansizoglu frames the challenge: “If a creator produced in English until now, how can we find the Spanish speaking viewers who would be interested in seeing their content in their own language?”

- Storing translated reels under a single video ID leverages existing infrastructure models but pools disparate language engagement into one bucket. - Recommendation systems rely on user-video interaction granularity, causing implicit language-learning signals to become entangled when multiple audio tracks share one ID.

[15:20](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=920s) Balancing Exploration and Exploitation in Video Retrieval

To break out of historical language silos and deliver newly translated videos to appropriate audiences, Meta relies on maintaining equilibrium between exploration and exploitation. When a fresh reel arrives, the recommendation system operates with significant uncertainty: beyond basic extracted text metadata, it cannot know whether the video will resonate with different cultural demographics or language cohorts. Exploration provides unproven content the initial impression opportunities required to gather real viewer engagement signals. Once these human feedback loops are established, exploitation steps in to distribute the content with high algorithmic confidence to wider audiences.

Maintaining this equilibrium is the primary focus of Cansizoglu’s team, who concentrate their algorithmic efforts on the retrieval phase of the recommendation funnel. Retrieval is the high-throughput bottleneck where massive catalogs of billions of candidate videos must be filtered down into a targeted candidate pool of hundreds of thousands of candidate videos for an individual viewer before heavier ranking models execute. As Cansizoglu emphasizes: “For a healthy ecosystem, these two modules should be in a perfect balance.”

```mermaid flowchart TD accTitle: Exploration and Exploitation in Retrieval accDescr: Pipeline showing how billions of candidate videos pass through exploration retrieval strategies before exploitation ranking.

subgraph Ingestion ["Content Ingestion"] A["Billions of Candidate Reels<br/>(Single Video ID per Reel)"] end

subgraph Retrieval ["Exploration in Retrieval"] B["Retrieval Engine"] C1["Interest Exploration<br/>(Content Understanding)"] C2["Item-to-Item Retrieval<br/>(Neural Content Signatures)"] C3["User-to-User Relevance<br/>(Cross-Lingual Taste Matching)"] B --> C1 B --> C2 B --> C3 end

subgraph CandidatePool ["Filtered Candidate Pool"] D["Hundreds of Thousands<br/>of Candidate Videos"] end

subgraph Exploitation ["Exploitation & Delivery"] E["Confident Exploitation<br/>& Ranking Pipeline"] F["Personalized Delivery to Viewers"] end

A --> B C1 --> D C2 --> D C3 --> D D --> E E --> F ```

[16:35](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=995s) Three Retrieval Approaches: Interests, Embeddings, and User-to-User Relevance

To power exploration within the retrieval layer, Meta deploys three distinct algorithmic approaches: interest exploration, item-to-item retrieval, and user-to-user retrieval. Cansizoglu addresses the common user question of how Facebook instinctively knows someone is fascinated by niche topics like woodworking. Meta runs large-scale content understanding models that construct multidimensional interest profiles for both content and viewers. By decoupling the underlying topic interest from the spoken language, the system can perform language-aware retrieval, presenting a high-affinity reel to a user who would otherwise never have engaged with the original creator due to the language barrier.

The second mechanism is item-to-item (or video-to-video) retrieval, powered by deep neural networks that map video assets into dense vector representations termed embeddings. Cansizoglu describes these embeddings as distinct "content signatures": if two pieces of content share core conceptual or semantic features, their signatures align closely in vector space. Combining these embedding signatures with a viewer’s or their social circle’s historical engagement allows the system to surface structurally similar videos. The third strategy, user-to-user retrieval, clusters viewers who exhibit overlapping taste profiles regardless of geography. Cansizoglu highlights how critical this is for speakers of lower-resource or less globally dominant languages, noting that a viewer’s tastes might align perfectly with someone halfway across the globe who speaks Marathi in India. As Cansizoglu explains: “This part focuses on finding users who have similar tastes to you and recommending content based on their engagement history.”

- **Interest Exploration:** Deep content understanding models build semantic interest profiles for creators, content, and viewers to enable language-agnostic discovery. - **Item-to-Item Retrieval:** Neural network embeddings act as "content signatures" to recommend conceptually aligned videos based on individual and friend engagement data. - **User-to-User Retrieval:** Matches viewers across different geographic and linguistic boundaries based on shared engagement patterns, expanding reach for languages like Marathi.

[18:37](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=1117s) Current Scale, Comment Translation, and What Lies Ahead

AI voice translation at Meta has graduated from experimental research into a scaled production system operating across both Instagram and Facebook. The platform supports more than 15 languages, serving millions of voice-translated Reels to international users daily. To prevent jarring discrepancies between video audio and on-screen interactions, Meta bundles voice translation with automated comment translation, allowing users to watch a foreign creator’s reel with localized speech while reading translated viewer reactions in their native tongue.

Looking ahead, Cansizoglu outlines key engineering challenges that remain unsolved. Expanding linguistic support to additional regional dialects and elevating translation precision remain central objectives. A particularly tricky modeling problem is viewer language prediction for multilingual users who regularly switch languages or consume mixed-language feeds. Additionally, optimizing compute infrastructure to curb heavy GPU utilization remains a critical priority for scaling the feature cost-effectively. Reflecting on the company's ethos, Cansizoglu concludes: “As we always say at Meta, the journey is 1% done.”

- Meta’s voice translations currently support over 15 languages across both Instagram and Facebook, delivering millions of reels. - Integrated comment translation accompanies voice dubbing so viewers can engage with discussion threads in their own language. - Key roadmap items include supporting complex multilingual user behaviors, improving language prediction algorithms, and implementing infrastructure optimizations to reduce GPU utilization. - Closing encouragement from Cansizoglu invites users to discover culinary culture freely: “So, the next time you find a recipe in a language that you don't understand, I hope you can just hit play.”

## Mentioned Andy Weir, Christian Fuegen, Esra Cansizoglu, Facebook, ICASSP, Instagram, Mark Zuckerberg, Meta, Meta Ray-Ban, Niko Moritz, Project Hail Mary, Ryland Grace, Sylvester Stallone

[Source](https://www.youtube.com/watch?v=mUJgk2qm6QI)

中文简报

Meta将配音延迟减半，每周分发5亿次翻译版Reels播放量 通过部署联合语音识别与跨语言检索，Meta得以打破以往的用户孤岛，在单一视频标识符下将外语创作者内容推荐给全球受众。 Esra Cansizoglu概述了Meta面向Reels的端到端AI语音翻译系统，该系统涵盖15种以上语言，每周提供超过5亿次翻译播放量。她解释了Meta如何将计算机视觉唇形同步技术与联合语音翻译与识别（JST&R）相结合；与级联模型相比，这种双编码器架构将首个词元的生成延迟缩短了一半。演讲详细阐述了在单一视频ID下索引多条翻译音轨所带来的基础设施难题，这种做法会模糊下游的互动指标。为了解决由此产生的冷启动推荐问题，Meta在检索阶段通过兴趣画像、神经内容嵌入以及跨国用户兴趣聚类，实现了探索与利用之间的平衡。 Meta目前每周在Facebook和Instagram上为语音翻译版Reels提供超过5亿次播放量，覆盖15种以上语言。 (0:05) 该工程流水线用联合语音翻译与识别（JST&R）取代了级联式ASR至机器翻译架构，通过同步快慢上下文编码器，将首个词元的生成延迟降低了50%。 (8:27) Meta在所有本地化音轨中保持单一视频ID架构，这要求推荐检索将语义兴趣与口语语言解耦，以避免冷启动孤岛问题。 (13:47) 检索层依托三大核心支柱在数十亿候选视频中平衡探索与利用：兴趣画像、稠密物品嵌入以及跨语言用户间品味聚类。 (16:35)

Reading summary

Meta machine learning engineer Esra Cansizoglu outlines how the company deploys AI voice translation and neural lip-syncing to address global content imbalances, serving over 500 million translated Reel views weekly. The system combines synchronized speech-translation models with specialized retrieval algorithms to distribute localized media without relying on historical engagement data. Dual-Track Audio-Visual Dubbing Pipeline (3:26) According to Cansizoglu, more than 50% of Facebook users watch video content in multiple languages across a community spanning over 160 linguistic backgrounds, yet available video inventory remains heavily dominated by English. To bridge this divide, Meta built an end-to-end production architecture that translates media while preserving the speaker's vocal tone and inflection, generating more than half a billion voice-translated Reel views every week. The system runs audio and video generation pipelines in parallel. On the audio track, signal processing models verify intelligible speech in supported languages before generative AI dubbing clones the original voice. Simultaneously, computer vision models verify facial visibility and resynthesize mouth movements to align naturally with translated phonemes. For viewers, Meta's interface allows manual selection of preferred languages or explicit 'do not translate' exclusion lists, falling back on machine learning inference to predict consumption preferences when explicit settings are missing. Joint Speech Translation and Latency Reduction (8:27) Dubbing media for billions of users requires processing speech in noisy acoustic environments featuring background crowds, music, and wind. Meta adapts conversational AI research from Niko Moritz, Christian Fuegen, and their team by pairing directional multi-channel microphone arrays with convolutional neural network (CNN) front-ends to extract speech features and isolate primary speaker audio. Rather than using conventional cascaded pipelines that transcribe speech through automatic speech recognition (ASR) before passing text to machine translation, Meta utilizes Joint Speech Translation and Recognition (JST&R). This architecture processes transcription and translation simultaneously using dual encoders: a fast encoder emits immediate translations for unambiguous words, while a slow encoder buffers token context when idiomatic nuances require sentence history. Cansizoglu reports that this dual-encoder design reduces first-token latency by 50% compared to cascading baselines without degrading semantic translation accuracy. Single Video Identifiers and Cold-Start Hurdles (13:47) On the systems engineering side, Meta maps all voice-translated audio tracks for a Reel to a single underlying video ID. While this decision allows translations to fit within existing media infrastructure without creating duplicate assets, it complicates downstream distribution. Recommendation algorithms evaluate engagement at the individual video-user interaction level, meaning implicit language signals from diverse audiences become blended under one asset identifier. This structural pooling creates a cold-start problem when creators release translated media into new regions. If a creator historically produced content solely in English, algorithmic recommendation models have no historical engagement metrics to determine which Hindi- or Portuguese-speaking viewers would enjoy the translated video, preventing standard recommendation models from identifying receptive audiences. Multi-Strategy Retrieval for Cross-Lingual Discovery (16:35) To overcome cold-start barriers, Meta focuses algorithmic intervention on the high-throughput retrieval phase of recommendation, which narrows billions of catalog videos down to a candidate pool of hundreds of thousands before ranking models run. The system maintains an equilibrium between exploration—allocating initial impressions to unproven media to gather behavioral feedback—and confident exploitation for broader delivery. Cansizoglu's team relies on three retrieval strategies to drive language-aware exploration: interest exploration applies content-understanding models to decouple subject matter (such as woodworking) from language; item-to-item retrieval converts videos into vector embeddings, or 'content signatures,' to match structurally aligned media based on user and friend history; and user-to-user retrieval clusters viewers across geographic borders based on overlapping tastes, enabling discovery for speakers of lower-resource languages such as Marathi. Meta's voice translation pipeline operates across Facebook and Instagram in more than 15 languages and includes synchronized comment translations so viewers can read discussions in their native language. Cansizoglu emphasizes that key engineering hurdles remain unresolved, including accurately predicting language preferences for multilingual users with mixed-language habits, expanding coverage to regional dialects, and reducing intensive GPU utilization across the infrastructure.

精炼阅读摘要

Meta 机器学习工程师 Esra Cansizoglu 阐述了公司如何通过端到端生成式 AI 语音翻译、口型同步以及重构推荐检索架构，消除语言隔阂并将跨语言视频规模化分发至全球受众。 跨语言消费需求与端到端生成式翻译流程 (0:05) 在 Meta 的社交平台上，超过 50% 的 Facebook 用户经常观看多种语言的视频内容，受众覆盖超过 160 种语言背景，但全球视频供给严重偏向英语。目前 Meta 平台每周语音翻译 Reels 的播放量已超过 5 亿次。为了将优质内容转化为无语言障碍的体验，系统构建了并行的音视频处理流水线：音频预处理模块首先过滤背景噪声并检验语音有效性及语言支持，再由生成式 AI 配音模型克隆原作者的音色与语调；计算机视觉模块则同步检测面部朝向与嘴部可见度，触发基于神经网络的口型同步重新编码，使说话者的唇形动作与翻译后的音素自然匹配。 联合语音翻译与识别架构降低延迟并提升保真度 (8:27) 传统级联架构通常先完成自动语音识别（ASR）转录，再将文本传入机器翻译引擎，容易造成延迟叠加。Meta 采用了基于 Niko Moritz 和 Christian Fuegen 等人研究的联合语音翻译与识别（JST&R）架构，利用卷积神经网络（CNN）前端结合多通道麦克风阵列特征提取，在复杂的背景杂音、风噪和人群干扰中分离主说话人声音。该架构通过双编码器协同工作：快速编码器负责在无需长上下文时迅速输出常见词汇的翻译，慢速编码器则在遇到复杂句式或成语时缓冲上下文标记以确保语义准确。基准测试表明，JST&R 在保持甚至改善整体延迟的同时，将首字生成延迟降低了 50%，并在葡萄牙语烹饪教学等实际场景中完整保留了创作者原有的语气与表达热情。 单一视频 ID 架构下的推荐冷启动与探索平衡 (13:47) 在底层系统工程方面，Meta 将包含多种语言配音的单个 Reel 绑定在统一的“单一视频 ID”下，以兼容现有数据模型并避免生成冗余的媒体资产。然而，推荐算法以往是在单视频与用户交互的粒度上学习偏好，单一 ID 导致英语、西班牙语或印地语等不同语言受众的互动信号混杂在一起，从而引发严重的冷启动难题。为解决创作者跨语种破圈的阻碍，推荐流水线在检索阶段着重维持“探索”与“利用”的动态平衡。系统首先给予新翻译视频初始曝光以收集真实受众反馈，待建立置信度后，再由利用模块将内容广泛推送给目标受众。 基于兴趣画像与向量嵌入的三维检索机制 (16:35) 为了在从数十亿候选视频筛选至数十万候选池的高吞吐检索瓶颈中实现冷启动匹配，Meta 部署了三种并行的算法探索机制。首先是兴趣探索，通过深度内容理解模型构建创作者、内容与用户的多维兴趣画像，将主题兴趣与原始语言解耦，实现跨语言召回；其次是基于深度神经网络的“物品到物品（Item-to-Item）”检索，将视频资产映射为高维密集向量作为内容签名，结合用户及其好友的互动历史推荐概念相似的视频；最后是“用户到用户（User-to-User）”检索，跨越地理与语言界限聚类品味重叠的用户，使得印度马拉地语等小语种受众也能根据全球相似品味人群的历史互动发掘适配内容。 目前 Meta 的语音翻译系统已在 Instagram 和 Facebook 支持超过 15 种语言，并配套推出了评论自动翻译功能以保证互动一致性。Cansizoglu 指出，未来的工程挑战包括为频繁切换语言的多语言用户构建更精准的语言预测模型、扩大对更多小语种与方言的支持，以及优化底层基础设施以降低大规模 GPU 算力消耗。

Detailed notes — zh

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 10 月 30 日发布。视频为 Meta 机器学习工程师 Esra Cansizoglu 的技术演讲，页面未提供额外的简介说明，核心围绕 Meta 如何在大规模平台上利用生成式 AI 语音翻译打破语言障碍展开。 ## TL;DR Meta 机器学习工程师 Esra Cansizoglu 深入解析了平台每周交付超 5 亿次播放的 AI 语音翻译 Reels 系统的全套架构。该体系主要由两大支柱构成：一是融合了声学降噪、快慢双编码器 JSTR 统一架构及 CV 口型同步的多模态配音流水线；二是针对单视频 ID 架构下语言标签混杂困境，在召回阶段通过兴趣探索、物品到物品及用户到用户三大策略实现的“探索与利用”跨语言推荐机制。目前该功能已在 Instagram 和 Facebook 支持逾 15 种语言，未来团队将持续攻克复杂多语言用户建模与算力优化等挑战。 ## Atmosphere & style 整场演讲呈现出典型的大厂硬核工程美学与亲切生活质感的交融。Esra 从与孩子在厨房照着外语食谱做饭的生活痛点娓娓道来，巧妙穿插《挽救计划》的科幻隐喻和史泰龙的声音克隆实例，将复杂的 ASR、音视频同步与推荐算法讲述得生动易懂。讲者的语调从容、严谨且富有感染力，让听众在领略尖端生成式 AI 与海量分布式推荐工程魅力的同时，也能深切体会到技术连接全球多元文化、消弭隔阂的温度。 ## Bulletin - [02:10](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=130s) Meta 语音翻译 Reels 每周播放量已突破 5 亿次，功能全面落地 Instagram 与 Facebook，目前支持超过 15 种语言跨语言分发。 - [08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) 团队摒弃传统“语音识别+机器翻译”的级联架构，采用快慢双编码器的 JSTR 统一模型，将首字生成延迟直接削减 50%。 - [14:13](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=853s) 在底层沿用单视频 ID（Single Video ID）的前提下，系统通过在召回阶段部署兴趣探索、物品到物品及用户到用户三种策略，解决了多语言冷启动分发瓶颈。 - [00:05](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=5s) Facebook 超过半数用户习惯观看一种以上语言的内容，受众覆盖 160 多种语言，推动 Meta 将技术研发重点聚焦于跨语言多模态合成与动态偏好预测。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=5s) Facebook 平台上超 50% 用户观看多语言内容，受众覆盖 160 余种语言，但优质内容供给严重向英语倾斜，催生了大规模生成式 AI 翻译需求。 - [02:10](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=130s) 语音翻译 Reels 每周在 Meta 旗下平台交付超过 5 亿次观看，以西尔维斯特·史泰龙等创作者为例展示了高度还原音色和语调的跨语言克隆效果。 - [03:44](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=224s) 端到端系统主要由音视频“配音流水线”与“相关性推荐引擎”两大支柱构成，需同时兼顾音色自然度、口型同步精度与用户动态语言偏好匹配。 - [05:47](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=347s) 生产流水线引入了严格的前置检测门禁：音频端校验有效语音长度与语种支持，视频端则评估人脸角度、分辨率与五官可见度以支持高质量口型重塑。 - [08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) 该技术直接反哺自 Ray-Ban Meta 智能眼镜的声学团队成果，针对复杂噪声环境采用多麦克风阵列与 CNN 特征提取实现精准的说话人分离。 - [08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) JSTR 联合模型通过快编码器处理常规词汇、慢编码器捕捉长上下文语境，在确保全局翻译质量的同时大幅降低了首个 Token 的响应延迟。 - [14:13](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=853s) 系统在底层架构上沿用单视频 ID（Single Video ID），导致不同语言版本的互动数据混合，给传统基于单一行为反馈的推荐系统带来了严重干扰。 - [15:20](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=920s) 为了帮助非原生语言创作者跨圈层触达新语种观众，推荐引擎依靠探索机制在受控小流量中收集互动数据，再通过利用机制实现规模化扩散。 - [15:20](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=920s) 在计算量最大的召回阶段，Meta 依托深度内容理解画像的“兴趣探索”实现跨语言感知召回，精准匹配小众兴趣受众。 - [15:20](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=920s) “物品到物品”召回利用大型神经网络提取视频高维嵌入签名，根据内容特征与主题相似度跨越语言边界匹配相关内容。 - [15:20](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=920s) “用户到用户”召回通过发掘跨语言品味高度一致的人群圈层，有效打破了小众语种（如马拉地语）用户的冷启动孤岛效应。 - [18:37](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=1117s) 语音翻译功能目前已打通评论区自动翻译，使全球用户在观看跨语言短视频时能够获得沉浸式的跨文化社交互动体验。 - [18:37](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=1117s) 未来核心演进方向聚焦于扩展非主流语言支持、优化复杂多语言混合用户的偏好预测模型，以及持续降低 GPU 推理与音视频合成的算力成本。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=5s) 厨房里的灵感与 Meta 的跨语言愿景 演讲者 Esra Cansizoglu 是 Meta 的一名机器学习工程师。她首先分享了一个极具生活气息的个人故事：她非常喜欢和孩子们一起待在厨房里，尝试制作来自世界各地的各种菜系美食。然而，每当遇到心仪的食谱是用外语写成或录制时，他们就会被语言障碍彻底卡住。这正是她和团队在 Meta 致力于解决的核心问题——通过生成式 AI 语音翻译（AI Voice Translations），为全球数十亿用户提供无障碍的内容消费体验。Esra 希望通过这场分享，让观众理解“当一滴全新的内容被系统发现并分发，从而在平台上促成真实人际连接的那一奇妙时刻”。 在 Meta 平台上，语言使用情况展现出了惊人的多样性。Esra 分享了一个令她自己都感到意外的数据：Facebook 上超过半数的用户观看内容时会涉及一种以上的语言，这表明平台用户具备天然的“多语言适应能力（language-savvy）”。平台上的受众说着超过 160 种语言，但内容生产的分布却极为不均，绝大部分优质内容仍然被英语主导。这就引出了团队必须直面的核心问题：如何让世界上任何一个人，都能用自己听得懂的语言去享受每一条内容？Esra 幽默地借用科幻小说《挽救计划》（Project Hail Mary）来打比方，指出书中主角 Grace 和外星人 Rocky 即使没有共同词汇，也能靠自制翻译器拯救恒星；而 Meta 的版本虽然“少了宇宙飞船”，背后的工程理念却十分相似。 - Facebook 上超过 50% 的用户观看一种以上语言的内容。 - 平台上的受众群体覆盖 160 多种语言，但内容供给严重向英语倾斜。 - 演讲者将该技术比作《挽救计划》中的翻译装置：“Well, our version has fewer spaceships, but the idea is somewhat similar.” ### [02:10](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=130s) 震撼展示与规模化挑战：从西尔维斯特·史泰龙到半十亿周播放量 为了直观呈现产品效果，Esra 展示了一段好莱坞巨星西尔维斯特·史泰龙（Sylvester Stallone）在 Facebook 上发布的 Reels 短视频。史泰龙是该功能的高频使用者。在视频片段中，史泰龙回忆起自己早年在电影院当引座员、周薪仅有 34 美元的艰难岁月，坦言大众只把引座员当成穿西装的工具人，但他却将电影院视为自己的大学。随后现场演示了 AI 将其声音精准克隆并翻译成其他语言的版本，音色、语调乃至情感起伏完全保留了他本人的声线特质。 这种极具真实感的跨语言呈现正在催生巨大的平台影响力。目前，Meta 每周通过语音翻译 Reels 交付的播放量已突破 5 亿次（half a billion views）。但 Esra 强调，要在如此体量下稳定运行绝非易事。整个系统既要攻克前向生成（音色克隆、口型同步）的保真度，还要在生产完成后解决极为复杂的推荐分发与用户偏好理解。正如 Esra 所言：“Today we deliver more than half a half a billion views on voice translated Reels every week. And for us this is about removing language barriers so people can connect authentically from anywhere in the world.” - 规模指标：语音翻译后的 Reels 目前每周产生超过 5 亿次观看。 - 核心愿景：打破语言鸿沟，让身处世界各地的创作者与观众建立真实的情感共鸣。 ### [03:44](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=224s) 端到端系统架构：配音、口型与“冷启动”难题 Esra 概述了本次演讲的核心框架，指出要实现这一产品体验，系统主要由两大核心支柱构成：一是负责音频与视频生成的“配音（dubbing）”流水线，二是负责精准匹配的“相关性与推荐（relevance and recommendation）”机制。随后她深入剖析了生产与分发各环节交织的复杂技术挑战。 在生产层面，音频合成必须高度自然，绝不能带有机械冰冷的机器人质感，同时翻译必须准确传达原意；为了追求极致的沉浸感，系统还提供了口型同步（lip sync）能力，必须借助计算机视觉算法重塑讲话者的面部与嘴部运动，使其与新生成的译文发音严丝合缝。而在内容生成后，分发端立刻面临严重的“冷启动问题（cold start problem）”。Esra 举例解释道：一名以往完全用英语创作的创作者，突然被系统生成了印地语或葡萄牙语版本，系统必须在没有历史互动数据的前提下，精准找到可能会对该内容感兴趣的讲葡语受众。此外，由于用户通常掌握多种语言，系统还必须在毫秒级时间内判断并为其渲染最符合当下意图的首选语言。 ```mermaid flowchart TD accTitle: 音视频生产与多语言分发架构图 accDescr: 展示从原始视频输入，经过音频处理与视觉口型同步，合并生成多语言视频，再经由推荐引擎分发给终端用户的完整流程。 subgraph AudioStream["音频处理流 (Audio Stream)"] A1["预处理检测 (语音充足性/语言识别)"] --> A2["生成式 AI 翻译模型 (GenAI)"] A2 --> A3["目标语言克隆音频"] end subgraph VideoStream["视觉处理流 (Video Stream)"] V1["人脸检测与可见度分析 (CV)"] --> V2["口型同步与面部重编码"] V2 --> V3["匹配新语言的视频画面"] end subgraph Fusion["多模态融合与分发"] M1["音视频对齐融合 (Voice Translated Reel)"] M2["多语言感知推荐系统 (Language-Aware RecSys)"] M3["界面与偏好控制 (UI/ML 语言匹配)"] end A3 --> M1 V3 --> M1 M1 --> M2 M2 --> M3 ``` - 音频要求：发音自然逼真、消除机械感、语义传达精准。 - 视频要求：CV 算法精准追踪面部轮廓，根据目标语言音素实现自然的口型重塑。 - 分发挑战：解决非原生语言创作者在小语种受众圈层中的冷启动难题。 - 语言选择机制：优先尊重用户在 UI 界面显式设置的“偏好语言”或“不翻译语言”列表；若无明确输入，则依赖 ML 模型推断用户当前最适应的消费语言。 ### [05:47](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=347s) 生产流水线剖析：从信号预处理到多模态合成 深入到生产流水线内部，该流程被精细地划分为音频与视频两条并行管线。在音频端，原始素材并非直接送入大模型，而是先由多个信号处理算法执行严格的预处理门禁检查：系统需要判定视频中是否包含足够时长的有效语音段落，以及源语言是否处于当前产品支持的语种清单中。只有通过这些前置校验，生成式 AI（Gen AI）模型才会介入，完成跨语种语音重合与音色克隆。 在视频端，若该条内容启用了口型同步功能，计算机视觉（CV）算法组件便会逐帧扫描视频，计算关键几何问题：画面中是否存在清晰可见的人脸？人脸的角度与分辨率是否足以支持高质量口型形变合成？计算完成后，CV 模型会输出经过重编码的嘴部面部动作。最终，重新合成的音频轨与调整后的视频轨在流水线末端汇合封装，生成一条全新的多语言 Reels 短视频。随后，这件新内容进入推荐候选池，推荐系统基于跨语言认知引擎，将原本受制于语言壁垒而无法触达的用户与优质内容连接起来。 - 音频预检指标：语音信号活跃度分析、源语种分类验证。 - 视频预检指标：面部检测定位、五官可见度与遮挡评估。 - 降级保障：若用户未主动指定播放语言，系统自动回退至用户多语言理解模型进行动态预测。 ### [08:27](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=507s) 核心技术突破：从 Meta 智能眼镜到 JSTR 统一翻译架构 随后，Esra 将话题转向底层配音算法的演进背景。她特别提到了 Meta Connect 大会上引起轰动的 Ray-Ban Meta 智能眼镜实时翻译演示——当时马克·扎克伯格（Mark Zuckerberg）戴着眼镜，与台上讲西班牙语的创作者流畅对话，眼镜即时完成了双向语音互译。这套技术由 Esra 的同事 Niko Moritz 和 Christian Fuegen 团队主导研发，其声学突破直接反哺了 Reels 的离线语音翻译系统。不过两者的工程侧重点有所不同：智能眼镜追求极致的低延迟，而视频平台的配音技术则更聚焦于高吞吐量（throughput）、跨长音频的声学一致性（consistency）以及支撑全网规模的扩展性（scalability）。 在现实声学场景中，用户上传的音频往往充斥着背景杂音、演唱会喧闹、赛场轰鸣、周围人群的交头接耳以及风噪等定向干扰。智能眼镜系统采用了定向多通道阵列方案（Directional Multi-Channel Setup），左侧采集多麦克风阵列信号，右侧通过卷积神经网络（CNN）特征提取模块支持流式自动语音识别（ASR），并在底层实现关键的“说话人分离（speaker separation）”，从而精准抽离出主角声音。 在翻译架构上，Meta 摒弃了传统的“ASR 识别文本 -> MT 机器翻译”级联流水线（cascaded system），转而采用了自研的联合语音翻译与识别模型——JSTR（Joint Speech Translation and Recognition）。该模型的核心在于同时训练快、慢双编码器（Fast and Slow Encoders）： - **快编码器（Fast Encoder）**：针对无需复杂语境的常规词汇实现极低延迟的即时转换。 - **慢编码器（Slow Encoder）**：当遇到需要结合长上下文才能准确理解语义的句式时，系统懂得适时停顿并调取更多历史 Token，确保翻译的精准度。 Esra 展示的实验对比数据显示，与传统级联架构相比，JSTR 模型不仅翻译延迟全面降低或基本持平，更将首个 Token 延迟（First Token Latency）直接缩短了一半，同时尾部延迟保持优良。正如 Esra 强调的那样：“So being able to separate speaker speech and then having accurate translation with low latency or high throughput and consistency is key for us to succeed in this product.” 在讲完生产底层的这一重大技术飞跃后，Esra 再次切回推荐视角。她放出了自己极喜爱的一位葡萄牙语烹饪博主的短视频：画面中博主正熟练地煎制着刚捕捞的新鲜大虾、撒入红椒粉与现磨黑胡椒制作浓郁拉面，而耳边流淌出的却已是流畅自如的英语旁白。正是 JSTR 与多模态合成的深度融合，让优质生活内容跨越了语言的鸿沟。 - 技术同源：继承自 Meta Ray-Ban 智能眼镜声学团队与 ICASSP 论文成果。 - 噪声抑制：结合 CNN 特征提取与多麦克风阵列，有效过滤赛场、风噪等环境音并剥离干扰人声。 - 架构革新：JSTR 统一架构取代级联串联，快慢双编码器协同平衡局部速度与全局语境。 - 性能实测：相较于基线系统，JSTR 将首字生成延迟（First Token Latency）降低了 50%。 ### [14:13](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=853s) 单视频 ID 架构与推荐系统的分发挑战 在深入探讨推荐算法的具体挑战之前，Meta 面临着来自底层基础设施的根本性制约。为了兼容现有的基础设施数据模型，经 AI 语音翻译的 Reels 视频在底层依然只由一个单一的视频 ID（Single Video ID）来表示。这种设计极大地简化了系统存储与流转，但对推荐引擎却带来了巨大的隐患。推荐系统通常是在“视频-用户”这一细粒度层面上捕获互动数据的，这意味着算法原本是在隐式地学习和适应语言偏好；而一旦将所有多语言版本的互动数据全盘混入同一个视频 ID 中，原本清晰的语言画像就会被稀释打乱。 因此，团队面临的核心痛点变成了如何解决创作者的触达边界（Creator Reach）。Esra Cansizoglu 尖锐地指出了这一分发困境：“If a creator produced in English until now, how can we find the Spanish speaking viewers who would be interested in seeing their content in their own language?”（如果一位创作者以往只制作英语内容，我们该如何精准找到那些本来会对该内容感兴趣、并希望能用母语观看的西班牙语观众呢？）解决跨语言推荐的关键，在于如何在缺少初始互动数据的情况下突破圈层，建立全新的冷启动触达路径。 ```mermaid flowchart TD accTitle: 语音翻译在单视频ID下的推荐挑战与召回探索机制 accDescr: 展示了多语言语音生成后混入单视频ID，并通过探索与利用机制在召回阶段完成分发的过程。 subgraph DataModel["底层数据模型"] A["单视频 ID (Single Video ID)"] --> B["混杂的多语言互动数据\n(英文/西语/其他语种互动混合)"] end subgraph Balancing["推荐系统核心平衡"] B --> C["探索机制 (Exploration)\n冷启动并收集初步互动"] C --> D["利用机制 (Exploitation)\n高置信度精准规模化分发"] end subgraph RetrievalStage["召回阶段探索三路线 (亿级到十万级)"] C --> E["兴趣探索 (Interest Exploration)\n内容理解与用户兴趣画像"] C --> F["物品到物品召回 (Item-to-Item)\n高维向量签名与相似度匹配"] C --> G["用户到用户召回 (User-to-User)\n跨语种品味相似人群协同"] end ``` ### [15:20](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=920s) 探索与利用：在召回阶段打破语言壁垒 为了攻克冷启动分发障碍，Meta 的推荐机制高度依赖“探索与利用”（Exploration and Exploitation）这两大核心模块的协同运作。当一条新内容进入系统时，除了一些基础文本信息外，算法既不知道它属于什么细分话题，也无法断定不同受众是否会对它产生兴趣。探索机制的核心使命就是给新内容一个被用户发现的机会，并在小范围测试中收集真实的人类互动数据；随后，利用机制便能够依据这些积累的数据，建立起足够的置信度，将内容高效且精准地分发给更广泛的受众。正如 Esra Cansizoglu 所强调的：“For a healthy ecosystem, these two modules should be in a perfect balance.”（对于一个健康的生态系统来说，这两个模块必须处于完美的平衡之中。） Esra Cansizoglu 的团队专注于在召回阶段（Retrieval Stage）实施针对语音翻译内容的探索机制。在 Meta 的推荐全链路中，召回是计算过滤最为剧烈的一环，它需要将数十亿级别的海量视频池压缩为针对特定用户的数十万候选视频。为了在该阶段精准探索潜在受众，团队采用了三种互补的技术策略： - **兴趣探索（Interest Exploration）：** 常有用户好奇系统如何知道他们喜欢像“木工制作（Woodworking）”这样的小众视频。Meta 训练了超大规模的内容理解模型，专门为视频内容和用户建立深度兴趣画像。通过该画像，系统可以执行跨语言感知召回（Language-aware Retrieval），把用户原本因语言障碍而无法理解、但深度契合其兴趣的高质量内容精准推送到面前。 - **物品到物品召回（Item-to-Item Retrieval）：** 该方法利用大型神经网络模型提取每个视频的特征，生成高维向量嵌入（Embeddings）。这些嵌入就如同视频的内容数字签名（Content Signatures），具有相似特征和主题的内容会产生相近的签名。结合用户自身或其好友的历史互动数据，系统即可基于内容签名相似度召回相关视频。 - **用户到用户召回（User-to-User Retrieval）：** 该策略致力于寻找与当前用户品味和审美高度重合的其他用户群，并基于这群相似用户的历史互动记录进行跨界推荐。Esra Cansizoglu 解释道，这一方法对于小语种观众尤为关键，例如一位普通用户与印度某位操持马拉地语（Marathi）的用户在内容喜好上可能高度一致，协同过滤能够彻底消除小众语种的冷启动孤岛效应。 ### [18:37](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=1117s) 落地成效与未来演进路线 AI 语音翻译与跨语言推荐的深度整合，使 Meta 能够在大规模场景下消弭人与人之间的语言隔阂。目前，该语音翻译功能已经在 Instagram 和 Facebook 两大平台上全面落地，支持超过 15 种语言，每天向全球用户分发数以百万计的 Reels 短视频。除了核心的音视频翻译外，该功能还与评论区自动翻译打通，使用户在观看异国创作者视频的同时，能顺畅阅读被翻译成英文或其他母语的评论，实现了完整的跨文化沉浸式互动体验。 展望未来发展，Esra Cansizoglu 引用了 Meta 内部广为人知的一句信条：“As we always say at Meta, the journey is 1% done.”（正如我们在 Meta 常说的那样，这段旅程才刚刚完成了 1%。）团队目前锁定了多个关键演进方向： - **拓展语种支持：** 进一步扩大语音翻译支持的语言库，覆盖更多非主流和小众语言。 - **提升翻译准确度：** 持续优化多模态大模型的翻译质感，确保语气、情绪与原视频口吻的高度拟合。 - **复杂多语言用户建模：** 优化观众语言预测（Viewer Language Prediction）模型。在现实中，许多用户日常在多种语言之间自由切换，对这类多语言混合使用者的精准偏好预测是一个极具挑战性的技术难点。 - **基础设施优化：** 针对模型推断和视频合成优化 GPU 算力资源消耗，降低推理延迟与大规模部署的硬件成本。 分享的最后，Esra Cansizoglu 幽默地回应了开场展示的烹饪短片：“So, the next time you find a recipe in a language that you don't understand, I hope you can just hit play.”（下一次当你刷到一个用你听不懂的语言讲解的食谱时，我希望你只需要放心地按下播放键就好。） ## Notable quotes > [02:10](https://www.youtube.com/watch?v=mUJgk2qm6QI&t=130s) Today we deliver more than half a half a billion views on voice translated Reels every week. And for us this is about removing language barriers so people can connect authentically from anywhere in the world. — Esra Cansizoglu ## Mentioned Esra Cansizoglu, Meta, Facebook, Instagram, Reels, 西尔维斯特·史泰龙, 马克·扎克伯格, Niko Moritz, Christian Fuegen, Ray-Ban Meta 智能眼镜, JSTR, Project Hail Mary, ICASSP [Source](https://www.youtube.com/watch?v=mUJgk2qm6QI)

Source quotes

[2:10] Esra Cansizoglu: Today we deliver more than half a half a billion views on voice translated Reels every week. And for us this is about removing language barriers so people can connect authentically from anywhere in the world.

Original transcript

[0:05] I love spending time with my kids in the [0:07] kitchen. We like trying out different [0:11] recipes from various cuisines all over [0:13] the world. But if the recipe we are [0:17] interested in is in a different [0:20] language, that's where we get stuck. My [0:24] name is Estrajan Solo. I am a machine [0:27] learning engineer at Meta. And this is [0:29] what I and my team work on. We provide [0:34] content accessibility for billions of [0:37] users through AI voice translations at [0:40] Meta. [0:41] By the end of this talk, hopefully you [0:44] will have a better understanding about [0:47] that magical moment. How one drop of [0:51] brand new content gets discovered and [0:54] distributed to form authentic human [0:57] connection on our platforms. [1:00] Here is something that surprised me. [1:03] Over half of our users on Facebook watch [1:07] content in more than one language. So [1:11] our users are language saving. So we [1:15] have viewers from inund [1:18] more than 160 languages [1:21] but as you can imagine the content isn't [1:24] even evenly spread. So most of this is [1:27] dominated by English although our users [1:31] are from all over the world. So then the [1:34] real question is how can we make each [1:38] and every content available for anyone [1:42] in a language they understand? [1:46] Well, if you have seen project Hail [1:48] Mary, this wasn't a problem for Raki and [1:52] Grace, right? They didn't speak a single [1:55] common word, but they were able to build [1:57] their translator and save the stars. [2:02] Well, our version has fewer spaceships, [2:05] but the idea is somewhat similar. [2:10] Let's first see how the product looks. [2:13] That's Sylvester Stallone. He is one of [2:16] our frequent users. Let's listen what he [2:19] says in different languages [2:22] in his own voice. getting paid $34 a [2:25] week to work in a movie theater as an [2:28] usher, which most people don't even [2:30] think as human beings. They're just a [2:31] guy in a suit. But I saw that place as [2:35] being my college. I had been to college. [2:38] I hadn't learned anything. [3:05] It's exciting, isn't it? Yes. Today we [3:09] deliver more than half a half a billion [3:12] views on voice translated reals every [3:14] week. And for us this is about removing [3:18] language barriers so people can connect [3:21] authentically from anywhere in the [3:23] world. [3:26] But this is not an easy problem to [3:28] solve. Both producing this type of [3:31] content. Even after production, solving [3:34] recommendation problem or language [3:36] understanding are not easy problems and [3:39] solving them at scale would require a [3:42] lot of effort. [3:44] Here is what to expect from today's [3:46] talk. I will first start with system [3:49] overview at a high level and then from [3:53] this big picture we will dive into two [3:55] important pieces. One of them is audio [3:58] generation which we call dubbing. The [4:01] other one is relevance and [4:02] recommendation problem and then I will [4:05] conclude with next steps. [4:09] So what translations are complex because [4:14] there are tons of things to work well to [4:17] be able to have a successful product. On [4:21] the production side in audio generation [4:24] we should make sure the resulting audio [4:27] is natural. It's not sounding robotic. [4:31] We would like to make sure translation [4:34] conveys the meaning. It's accurate. [4:37] In addition to audio, we also provide [4:40] lip sync capability. So for this to [4:43] work, we need to make face movements [4:46] according to the language being spoken [4:48] and it should be accurate as well. Even [4:51] after production, the second challenge [4:54] we have is now we have this content. How [4:57] do we understand who to show this [5:00] interesting content especially in the [5:02] languages that we doubt? This is called [5:05] cold start problem. The problem is this. [5:08] A creator who was producing in English [5:12] until now now all of a sudden starts [5:15] producing in Hindi in Portuguese. Then [5:18] the question is how do we find that [5:20] Portuguese speaking viewers who would be [5:23] interested in their content. [5:26] Even after solving the relevance problem [5:28] another challenge awaits us because we [5:31] have multiple languages available and [5:33] remember our viewers are language s. We [5:37] would like to render the video in the [5:39] right language in the preferred language [5:41] of the user which is also a hard problem [5:44] to detect. [5:47] Let's move into production flow a little [5:49] bit. So in production flow, we provide [5:53] both audio and video generation [5:55] capabilities. [5:56] For audio, we have several signal [5:59] processing algorithms that are doing [6:02] some pre-processing checks. There are [6:04] questions we need to answer. Is there [6:06] enough speech in this content to [6:08] translate? [6:10] What is the language of this content? Is [6:12] it among supported languages in our [6:14] product? Once all these checks succeed, [6:18] then our Gen AI model which I will speak [6:21] in a bit handles the rest and creates [6:24] the audio in the translated language. [6:27] The second stream is on video [6:29] generation. If lip sync is enabled in [6:32] this video, then we have several [6:35] computer vision algorithms which would [6:38] take the video frames and answer a lot [6:41] of questions. Does this video have face [6:44] in it? Is it visible enough to create [6:46] the lip sync? And from there, computer [6:50] vision algorithms are creating the new [6:52] video encoding as well. Finally, these [6:55] two streams are blended together. Now, [6:58] we have a voice translated video. [7:02] Let's talk about what happens next. Now, [7:04] this real is a brand new content in our [7:08] pool. We have a big machinery which we [7:12] call recommendation algorithms that [7:14] would solve the relevance between users [7:17] and content. We use these methods in a [7:21] language aware way because we would like [7:24] to especially recommend content in a [7:27] language that you understand. Originally [7:30] you wouldn't have been consuming without [7:32] voice translations and that's where we [7:34] come in. So after solving user video [7:38] relevance, we would like to also make [7:40] sure we are rendering it in the correct [7:41] language. Here I would like to give a [7:44] huge shout out to our user interface [7:46] team because they are the ones who are [7:49] working on making this consumption [7:52] experience frictionless. Through their [7:54] efforts, users are able to change the [7:58] language of the real or provide their [8:01] preferred or do not translate languages. [8:04] And our distribution flow on the [8:07] delivery side respects these choices [8:10] from the viewers. If there is no input [8:13] from the user, then we fall back into [8:16] our machine learning models which [8:19] detects what kind of languages you are [8:21] comfortable consuming in. And that's not [8:24] an easy problem either. [8:27] Now let's dive deeper into audio [8:29] generation or dubbing problem. [8:32] You may remember the live translation [8:34] demo on metaran glasses where Mark [8:38] Zuckerberg was speaking to a creator who [8:42] speaks in Spanish and there was live [8:44] translation going on on the stage. My [8:47] colleagues Nikico Moritz and Christian [8:49] Fuan and their team are working exactly [8:52] on this problem. [8:54] So their research on live translation [8:58] enabled voice translations audio [9:00] generation as well. If you are [9:02] interested in knowing more about the [9:04] details, please feel free to take a look [9:07] at the IAS paper linked. [9:09] In the case of live translation, the [9:12] major constraint is latency. While for [9:16] voice translations, we care about [9:19] throughput, consistency, and scalability [9:23] because we would like to make this [9:26] possible for billions of users at scale. [9:31] So in an acoustic scenario, the real [9:35] audio is not clean. It's messy. It will [9:39] contain other things than the speaker [9:42] itself. It might have some background [9:44] noise. Maybe there's a concert or game [9:47] ongoing in the background. There might [9:49] be some background conversations or some [9:51] directional noise, wind and other [9:53] things. The system should be able to [9:57] separate out destructors from the speech [10:00] so that it finds the main speaker to [10:03] translate. It's an essential feature of [10:06] our method. [10:08] What you are looking at here is [10:11] directional multi- channelannel setup [10:14] for meta rayband glasses. On the left, [10:17] what you are seeing is the stream of [10:21] microphones and the signals, audio [10:23] signals we are getting from there. On [10:26] the right, you can see our feature [10:28] extraction module which is backed by a [10:31] convolutional neural network and [10:33] providing us an streaming ASR model. As [10:37] you can see on the bottom, the output is [10:41] different streams per speaker and having [10:44] this separation, speaker separation is [10:47] key for a good voice translations [10:49] product. Let's talk more how we achieve [10:53] this audio generation. [10:55] So in the classical systems usually we [10:58] see a cascaded streaming service as you [11:01] see on the left. So there is an [11:03] automated speech recognition module [11:06] whose output gets fed into machine [11:08] translations module. Our method is [11:12] called joint speech translation and [11:14] recognition in short jar. And the main [11:18] difference is we solve both of these [11:21] recognition and translation problems [11:23] simultaneously [11:25] using slow and fast encoder. When it [11:30] comes to translations, [11:32] context matters a lot. So with slow [11:36] encoder and fast encoder, what we [11:39] achieve is understanding the context. [11:43] Fast encoder can quickly translate some [11:46] words when there is no need for [11:49] longer context. While with slow encoder, [11:53] the system knows when to pause and look [11:57] at more history of tokens [12:01] to be able to provide accurate [12:03] translation. So once these two encoders [12:08] trained simultaneously, [12:10] we can achieve both accuracy and better [12:13] latency. [12:15] As you can see here on the top table, [12:19] when we compare using slow and fast [12:22] encoders with a classical cascaded [12:25] system, we are improving latency or at [12:29] worst comparable and we don't sacrifice [12:32] latency for that. As you see on the [12:34] bottom, we are improving first token [12:38] latency by half by using JAR compared to [12:41] the baseline system. and with less token [12:44] latency it is comparable. So being able [12:48] to separate speaker speech and then [12:51] having accurate translation with low [12:54] latency or high throughput and [12:57] consistency is key for us to succeed in [13:02] this product. [13:05] Now we have the video ready produced. [13:10] Let's dive into recommendation. [13:12] Let me first remind you I have a [13:15] personal reason why I work on this [13:17] project. Yes, I like cooking. So, I [13:20] would like to show you a cooking video [13:23] from one of my favorite creators. [13:26] He produces it in Portuguese, but thanks [13:29] to his translations, I can listen to him [13:32] in English and try out his recipes. [13:34] Let's listen from him. If you're a fan [13:37] of ramen, but you like it really creamy, [13:39] you're going to love this recipe. And [13:41] with kukaracha style shrimp, it's [13:42] another level. Freshly caught shrimp. We [13:44] add paprika, salt, and freshly ground [13:47] black pepper. We mix very well. A little [13:49] bit of hot oil. And in go the shrimp. [13:51] And don't forget to fry. [14:13] Before we dive into recommendation [14:15] challenges, I would like to list out [14:18] some of the challenges we are facing on [14:20] the infrastructure side. [14:23] So a voice translated real is [14:26] represented with a single video ID which [14:30] is easier because this way we are [14:32] utilizing existing data model in in the [14:36] infrastructure. [14:37] However, this brings another challenge [14:40] because recommendation system captures [14:43] engagement data at video user grade. [14:47] This means as a result it's learning [14:50] language implicitly. [14:52] When we are using a single video ID, all [14:55] of the engagement data gets blended into [14:58] the single video ID. So for us, the [15:02] challenge is solving creator reach. [15:06] If a creator produced in English until [15:08] now, how can we find the Spanish [15:12] speaking viewers who would be interested [15:15] in seeing their content in their own [15:18] language? [15:20] We solve this problem by using [15:23] exploration and exploitation. [15:27] These two terms are critical for a [15:30] healthy recommendation system. [15:33] Let's assume there is a brand new [15:35] content coming in. We don't know whether [15:38] this content will be engaging to [15:40] different audiences. We don't know what [15:43] kind of topic this content is except [15:45] some of the text information. [15:48] What exploration does is it gives this [15:51] content a chance to be discovered by [15:53] viewers. Through exploration, we collect [15:57] some human viewer engagement data which [15:59] then helps with exploitation. [16:02] Through this data, in exploitation, the [16:05] system is more confident how to [16:08] distribute this content effectively. For [16:10] a healthy ecosystem, these two modules [16:13] should be in a perfect balance. And [16:16] that's where my team comes in. We [16:19] especially work on exploration of this [16:21] content in the retrieval stage because [16:25] retrieval is where billions of videos [16:29] gets filtered into hundreds of thousands [16:32] of videos for a given user. [16:35] We solve exploration using three [16:38] different approaches. [16:40] We follow item to item or video to video [16:43] retrieval. [16:45] We pursue user to user relevance as well [16:48] or we do interest exploration. [16:53] Let me speak more about each of them in [16:55] depth. [16:57] I usually get this question. How does [17:00] Facebook understand I am into [17:01] woodworking videos? Well, at Meta, we [17:06] have huge content understanding models [17:09] that builds interest profile for content [17:13] and our users. This way, we will be able [17:16] to find your interests and perform [17:19] language away retrieval by showing you a [17:22] content that you wouldn't otherwise [17:24] understand without voice translations. [17:27] The second technique we follow is [17:29] itemtom retrieval. We have large neural [17:33] network models that generates features [17:36] per content which we call embeddings. [17:39] You can think of these embeddings as [17:42] content signatures. [17:44] If two content is similar then you would [17:47] expect their signatures to be similar as [17:49] well. By using embeddings and by looking [17:53] at use your user engagement data or your [17:57] friends user engagement data, we are [18:00] able to find similar content that would [18:02] be interesting for you. [18:05] The third method we follow is [18:07] userto-user retrieval. This this part [18:12] focuses on finding users who have [18:15] similar tastes to you and recommending [18:17] content based on their engagement [18:19] history. [18:20] This type of methods are critical [18:23] especially for viewers who speak a [18:26] language that are less popular in the [18:28] world. I bet you might be in the same [18:31] taste with a user who speaks Marathi in [18:34] India for example. [18:37] To sum up with voice translations at [18:40] Meta we are blurring the boundaries to [18:44] achieve human connection at scale. And [18:48] this has been possible with the [18:50] innovation in AI and solving this highly [18:54] complex infrastructure and production [18:57] problems and distribution problems. So [19:00] currently voice translation support more [19:02] than 15 languages on both IG, Instagram [19:06] and Facebook delivering millions of [19:08] reals to users. Along with command [19:11] translation, it provides a seamless [19:14] experience. the creator that I told you [19:16] about. I can see the comments on their [19:19] reels in English as well. So, that's [19:21] amazing. [19:23] Here is what's next for us. We would [19:25] like to further improve this product by [19:28] supporting more languages. As we always [19:30] say at Meta, the journey is 1% done. So, [19:34] we would like to also improve [19:35] translation accuracy. And we want to [19:38] support multilingual users better [19:41] especially viewer language prediction is [19:43] a hard problem and having language s [19:47] users makes it even more challenging. [19:50] Lastly we would like to do also some [19:52] infra optimizations so that we can [19:56] reduce GPU utilization for the product. [20:01] So, the next time you find a recipe in a [20:04] language that you don't understand, I [20:06] hope you can just hit play. Thank you.