Meta Deploys AV1 Video to Most Mobile Devices via Custom Encoder

@Scale

Meta overcame mobile thermal limits to roll out AV1 across its calling fleet by pairing an in-house low-complexity encoder with machine-learning device scoring and dynamic H.264 fallbacks.

AI-assisted summary of Adopting AV1 for Real-Time Communication (RTC) at Scale | Yu Chen Sun Eric from Meta — original source. Check the original for full context.

Source published: 2026-10-30

Key ideas

Meta has expanded AV1 deployment across the majority of mobile devices in its fleet by pairing an in-house, low-complexity software encoder with the dav1d decoder. (3:30)
To solve Android hardware fragmentation and thermal throttling, Meta implemented an ML-based telemetry model that outputs continuous device 'RTC Scores' alongside dynamic runtime fallback to H.264. (7:09)
Real-time rate control uses a Video Buffer Verifier (VBV) leaky bucket model to prevent momentary bit overshoots from causing buffer drain delays that exceed latency budgets. (13:25)
Packet loss resilience is handled without expensive IDR keyframe storms by combining dynamically toggled two-layer temporal scalability with Long-Term Reference (LTR) recovery frames. (15:58)

Detailed notes

## Background Published on YouTube by the @Scale channel on October 30, 2026, this technical presentation features Yu Chen Sun Eric from Meta's Media Foundation team discussing video call quality across Meta's family of apps. No show notes or video description were provided by the publisher.

## TL;DR Yu Chen Sun Eric details Meta's engineering journey to scale AV1 video compression across the majority of mobile devices for real-time communication (RTC). To navigate severe compute, battery, and thermal limits, Meta engineered an in-house low-complexity software encoder, transitioned decoding to dav1d, and deployed an ML-driven device eligibility framework coupled with dynamic runtime fallbacks to H.264. The architecture also integrates Video Buffer Verifier (VBV) rate control to prevent buffer bloat alongside hierarchical temporal layers and Long-Term Reference (LTR) frames to maintain stream continuity under packet loss without expensive keyframes.

## Atmosphere & style The talk delivers a methodical, deeply technical, and pragmatic systems engineering narrative. Eric speaks with measured authority and clarity, illustrating tough production challenges—like thermal throttling and Android fragmentation—with clear, problem-solution architectural breakdowns. It feels like an authentic insider debrief from the engineering trenches, striking a balance between rigorous performance optimization and collaborative industry vision.

## Bulletin - [03:30](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=210s) Meta has expanded AV1 deployment across the majority of mobile devices in its fleet by pairing an in-house, low-complexity software encoder with the dav1d decoder. - [07:09](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=429s) To solve Android hardware fragmentation and thermal throttling, Meta implemented an ML-based telemetry model that outputs continuous device 'RTC Scores' alongside dynamic runtime fallback to H.264. - [13:25](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=805s) Real-time rate control uses a Video Buffer Verifier (VBV) leaky bucket model to prevent momentary bit overshoots from causing buffer drain delays that exceed latency budgets. - [15:58](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=958s) Packet loss resilience is handled without expensive IDR keyframe storms by combining dynamically toggled two-layer temporal scalability with Long-Term Reference (LTR) recovery frames.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=5s) Under severe bandwidth constraints such as 100 kbps, AV1 preserves sharp visual fidelity where H.264 degrades into severe blurriness and blockiness. - [00:05](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=5s) Real-time communication imposes strict operating boundaries compared to VOD, requiring sub-300 ms end-to-end latency and simultaneous real-time encoding and decoding on client hardware. - [03:30](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=210s) Off-the-shelf open-source AV1 encoders consumed 14% more battery on mobile devices, leading Meta to develop a custom in-house encoder with an ultra-low-complexity preset matching H.264's footprint. - [03:30](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=210s) Switching client-side AV1 decoding from libaom to dav1d yielded better power efficiency, higher reliability, and measurable gains in user engagement metrics. - [07:09](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=429s) Static Android whitelisting based on RAM or OS version proved unreliable, prompting Meta to adopt an ML model that predicts device RTC capabilities using production telemetry. - [10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) Even recent flagship devices can suffer severe thermal throttling mid-call, requiring adaptive encoder presets that scale complexity up or down based on real-time encoding latency. - [10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) H.264 is pre-negotiated alongside AV1 during initial SDP setup, allowing seamless, mid-call codec fallback without tearing down the active media pipeline. - [10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) A peer-aware RTCP feedback loop reports receiver decoding latency back to the sender, triggering sender fallback to H.264 if the remote device CPU becomes overloaded. - [13:25](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=805s) Monitoring the Video Buffer Verifier (VBV) state is vital in RTC, as unsent buffer accumulation can easily push drain times past 200 ms and trigger acute network congestion. - [15:58](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=958s) A two-layer temporal hierarchy isolates an autonomous base stream from an enhancement layer, allowing decoders to drop enhancement frames without stalling playback. - [15:58](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=958s) Because hierarchical prediction inherently lowers raw coding efficiency, Meta dynamically enables temporal layering during packet loss and disables it once network conditions recover. - [18:53](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1133s) Long-Term Reference (LTR) frames allow encoders to emit lightweight LTR-predicted (LTP) frames after packet loss, preventing massive bitrate spikes from full IDR keyframes. - [20:08](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1208s) Meta is urging SoC vendors to deploy hardware-accelerated AV1 encoding and decoding across all market tiers to support high-efficiency conversational AI video agents.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=5s) The Evolution and Promise of AV1 in Real-Time Communication

Yu Chen Sun Eric, an engineer on the Media Foundation team at Meta working on video call quality across the Meta family of apps, opens the presentation by reflecting on the progress made over the past three years. Three years earlier, when Meta first shared their early work on AV1 for real-time communication (RTC), the codec could only be deployed on select high-end and mid-range mobile devices due to extreme computational demands. Today, Meta has achieved a major milestone: shipping AV1 to the majority of mobile devices across their applications.

A video codec serves two fundamental purposes in real-time communication: compressing video efficiently without visual degradation, and maintaining playback robustness during adverse network conditions. To illustrate the tangible benefits of AV1 over the ubiquitous H.264 (AVC), Eric presents a side-by-side video call comparison performed on Messenger using two Android phones. Under a severely constrained bitrate capped at 100 kbps, the H.264 stream appears distinctly blurry and blocky, whereas the AV1 stream remains visually crisp and detailed.

```mermaid flowchart LR accTitle: Real-Time Communication Constraints vs Benefits accDescr: RTC performance limits compared to native AV1 compression and adaptation features.

subgraph RTC["RTC Constraints"] C1["Latency < 300 ms"] C2["No Multi-Pass / Low Buffer"] C3["Simultaneous Encode + Decode"] end

subgraph AV1["AV1 Capabilities"] A1["Advanced Compression Efficiency"] A2["Reference Picture Resampling"] A3["Native Screen Sharing Tools"] end

RTC -.->|"Requires low complexity"| AV1 ```

Eric explains that RTC introduces strict operational boundaries compared to video-on-demand (VOD): - **Latency Thresholds**: End-to-end video latency must remain strictly under 300 milliseconds; exceeding this threshold introduces noticeable delays that frustrate users. - **Encoding Limitations**: RTC systems cannot rely on multi-pass encoding or deep decoder buffers; any sudden bitrate overshoot risks immediate frame freezing. - **Device Load**: The mobile client must execute both encoding and decoding simultaneously in real time, placing heavy demands on battery life and processing budgets. - **AV1 Advantages**: AV1 provides superior compression efficiency alongside native RTC tools such as Reference Picture Resampling (which adjusts resolution dynamically without forcing an intra/keyframe) and dedicated screen sharing coding tools built directly into the Main profile.

---

[03:30](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=210s) The Challenge: Hardware Capacity, Encoder Choice, and Dav1d

While AV1 delivers substantial compression improvements, the central hurdle is computational complexity. An AV1 software encoder can easily require twice the processing power of an H.264 encoder. In group calls, a mobile client may be forced to handle three concurrent encoding and decoding streams. Enabling AV1 on an incapable device results in severe frame drops, UI freezes, or complete app crashes. Android presents an especially daunting challenge due to the fragmentation of thousands of distinct device models.

Eric notes that selecting the encoder was Meta’s most critical decision. Three years prior, the team tested an open-source AV1 encoder on a Google Pixel 4a and observed a 14% increase in power consumption relative to H.264, making mobile deployment unviable. However, the team realized that higher efficiency does not mandate high complexity:

> “However, we think an AV1 encoder should not necessarily require high complexity encoding because a modern codec actually offer a larger coding tools coding a larger set of coding tool. So a well-defined encoder actually has more opportunity to find a better tradeoff between quality and complexity.” — Yu Chen Sun Eric

To solve this, Meta built an in-house AV1 encoder tailored specifically for RTC. It introduced an ultra-low-complexity preset capable of matching H.264's complexity footprint while retaining AV1's coding benefits. On the decoding side, Meta initially deployed `libaom`, but after extensive A/B testing against various open-source decoders, migrated to `dav1d`. The switch yielded superior power efficiency, higher reliability, and measurable gains in user engagement metrics.

---

[07:09](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=429s) ML-Based Device Eligibility System

Classifying capable devices on iOS is straightforward because Apple's hardware lineup is concise and easily validated via A/B testing. Android, by contrast, comprises thousands of hardware variations. Meta initially attempted heuristic-based whitelisting using parameters such as RAM size, release year, or Android OS version, but found all of them unreliable. While they later relied on Meta’s internal synthetic benchmarking systems, static rules could not handle real-world diversity.

```mermaid flowchart TD accTitle: ML-Based Device Eligibility Architecture accDescr: Telemetry collection pipeline feeding an ML model that outputs an RTC score to determine codec settings.

subgraph Logging["Client Telemetry"] P1["Real Call Product Data"] P2["Low-Level Performance Signals"] end

subgraph Intelligence["Model Inference"] M1["ML Eligibility Model"] M2["Device RTC Score"] end

subgraph Policy["Runtime Decision"] D1["Select AV1 Preset"] D2["Enable Decode Only"] D3["Fallback to H.264"] end

Logging --> M1 M1 --> M2 M2 --> Policy ```

To resolve this at scale, Meta designed a machine-learning-based eligibility system that replaces static rules with production telemetry: - **Telemetry Collection**: The logging pipeline extracts low-level performance signals directly from real user calls across older hardware and new annual phone releases. - **RTC Score Generation**: The machine learning model processes these signals and computes a continuous "RTC Score" characterizing how well AV1 operates on any specific device model. - **Policy Enforcement**: The RTC Score dictates whether a device can encode and decode, decode only, which encoder preset to configure, or whether the device must remain on H.264. - **Continuous Retraining**: As global AV1 call volume expands, fresh telemetry feeds continuously into the model to refine score accuracy and expand coverage over time.

---

[10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) Dynamic Runtime Adaptations: Presets, Fallback, and Peer Feedback

Even with ML device profiling, real-world deployment revealed that static pre-call qualification was insufficient. During live A/B tests, several high-end smartphones released as recently as 2023 suffered encoding and decoding failures. Meta discovered that under sustained RTC call loads, these devices experienced severe thermal throttling, causing CPU frequencies to plummet far below expected baselines.

To address fluctuating runtime conditions, Meta engineered three dynamic fallback mechanisms:

1. **Adaptive Encoding Presets**: The encoder exposes a spectrum of presets ranging from high to low complexity. The client continuously tracks real-time encoding latency. If latency surges, the encoder steps down to a lighter preset; once latency stabilizes, it steps back up to maximize visual fidelity. 2. **Dynamic Codec Switching (H.264 Fallback)**: If reducing the AV1 preset to its absolute lowest complexity level still fails to alleviate encoding latency, the client dynamically switches the live call to H.264. Both codecs are pre-negotiated during initial call setup (SDP exchange), enabling seamless fallback without tearing down the media pipeline. 3. **Peer-Aware Decoder Feedback & Health Signals**: Because high-end callers frequently connect with peers on low-end devices, a sender may encode AV1 successfully while the receiver's CPU lacks the horsepower to decode it in real time. Meta introduced a RTCP feedback loop where the receiver reports decoding latency back to the sender; if decoding delays breach thresholds, the sender falls back to H.264. Furthermore, internal device health signals like low battery trigger proactive fallback to H.264 to preserve device longevity.

Having established these dynamic safeguards to scale AV1 to the majority of devices, Eric transitions to discussing call quality enhancements, focusing on rate control and error resilience, and pointing out that even momentary bitrate overshoots trigger acute network congestion and severe packet loss.

[13:25](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=805s) Rate Control and the Video Buffer Verifier (VBV) Model

In Real-Time Communication (RTC), monitoring the average bitrate alone is inadequate; instant overshoots can severely disrupt call quality. To evaluate this behavior, Meta relies on the Video Buffer Verifier (VBV) leaky bucket model. Eric illustrates this with a scenario where the network video bandwidth is set to 100 kbps, and the encoder is instructed to output at that target. If frame $n$ produces 20k bits while 15k bits from previous frames remain unsent in the buffer, transmitting frame $n$ requires pushing out 35k bits total. Under a 100 kbps connection, draining that volume requires 350 milliseconds. In real-time video, end-to-end latency must generally remain under 200 ms; a 350 ms VBV delay introduces unacceptable latency, triggers network congestion, and causes the receiving video to freeze.

```mermaid flowchart TD accTitle: Leaky Bucket VBV Delay Model accDescr: Visual representation of unsent bits accumulating in the network buffer causing high VBV latency. subgraph S1["Buffer State"] A["Existing buffer backlog: 15k bits"] B["New encoded frame n: 20k bits"] end subgraph S2["Transmission Constraint"] C["Total payload: 35k bits"] D["Network Bandwidth: 100 kbps"] end subgraph S3["RTC Impact"] E["Drain Time: 350 ms"] F["Target Latency: < 200 ms\nResult: Congestion & Video Freeze"] end A --> C B --> C C --> E D --> E E --> F ```

To counter these spikes, the engineering team introduced rate control enhancements that instruct the encoder to continually monitor the VBV buffer level during runtime. This signal directs bit allocation across encoding threads: when an overshoot is detected, the target bit budget for subsequent frames is immediately reduced, pulling the VBV delay back into safe operating bounds. Eric stresses the fundamental challenge: “This is why accurate rate control is important in RTC.”

Rate control in RTC systems faces three main operational hurdles: - **Frequent Target Bitrate Swings:** Clients constantly adjust target bitrates in response to changing network conditions during an active call, requiring the encoder to preserve stable VBV delay through abrupt transitions. - **Dynamic Resolution Adjustments:** Call participants frequently trigger resolution shifts, forcing the rate controller to adapt bit allocations without destabilizing buffer levels. - **Undershooting Vulnerabilities:** While overshooting causes latency spikes and packet loss, undershooting misleads bandwidth estimation (BWE) algorithms, dampens the bitrate ramp-up process, and unnecessarily degrades visual quality.

---

[15:58](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=958s) Error Resilience: Mitigating Packet Loss with Temporal Layers

Standard video encoding relies on inter-frame prediction, where current frames depend on preceding frames. When network packet loss occurs, this dependency chain breaks; the decoder cannot reconstruct subsequent frames, prompting sender retransmission requests. If retransmission also fails, the decoding pipeline stalls, forcing the encoder to generate an intra keyframe (IDR). Keyframes are significantly larger in bit size and exhibit degraded visual quality relative to the bandwidth consumed. To prevent keyframe storms under packet loss, Meta implemented two error resilience techniques in AV1: temporal scalability and long-term reference (LTR) frames.

Temporal scalability replaces linear frame chains with a time-based hierarchy. In Meta's AV1 deployment, a two-layer structure is used. Layer 0 serves as an autonomous base layer providing a lower frame rate video stream that decodes independently of higher layers. Layer 1 operates as an enhancement layer, delivering supplemental frames to elevate the frame rate under favorable network conditions. If packets in the enhancement layer drop, the decoder discards them and continues rendering the base stream without freezing.

```mermaid flowchart LR accTitle: Two-Layer Temporal Hierarchy accDescr: Showing base layer 0 decoding independently while enhancement layer 1 adds frame rate. subgraph Base["Layer 0: Base Layer (Protected)"] L0_1["Frame 0"] --> L0_2["Frame 2"] L0_2 --> L0_3["Frame 4"] end subgraph Enh["Layer 1: Enhancement Layer"] L1_1["Frame 1"] L1_2["Frame 3"] end L0_1 -.-> L1_1 L0_2 -.-> L1_1 L0_2 -.-> L1_2 L0_3 -.-> L1_2 ```

Meta combined this structure with end-to-end system optimizations: - **Heavier Forward Error Correction (FEC):** Base-layer packets receive higher FEC redundancy because their survival is critical to maintaining stream continuity. - **Adaptive Retransmission Filtering:** If real-time round-trip time (RTT) is excessively high, retransmitting dropped enhancement frames arrives too late for playback; the sender proactively suppresses enhancement-layer retransmissions. - **Dynamic Temporal Layer Toggling:** Because hierarchical prediction exhibits lower coding efficiency than unconstrained prediction, temporal layering is toggled adaptively: enabled during high packet loss and turned off once network conditions recover to preserve peak video quality.

Eric underscores the importance of this multi-tiered tuning: “This kind of end to end system level optimization is critical to provide our user the best experience.”

---

[18:53](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1133s) Fine-Grained Recovery via Long-Term Reference (LTR) Frames

While temporal layers enforce a rigid hierarchy, Long-Term Reference (LTR) frames offer fine-grained control over frame dependencies. Under the LTR design, an encoder preserves designated frames in its reference buffer for extended periods. When packet loss occurs, the encoder avoids transmitting an expensive keyframe; instead, it generates a new frame that references the earlier, acknowledged LTR frame—a mechanism termed an LTR-predicted (LTP) frame.

```mermaid flowchart LR accTitle: Long-Term Reference (LTR) Frame Recovery accDescr: An LTR-predicted frame recovers the video stream by referencing an older verified frame instead of issuing a full keyframe. A["LTR Frame (Cached in Buffer)"] --> B["Normal Inter Frames (Dropped/Lost)"] A ==>|"Skip Corrupted Chain"| C["LTP Frame (Predicted from LTR)"] C --> D["Restored Stream Continuity"] ```

To maximize recovery speed over poor network links, Meta applied end-to-end intelligence: when the transport layer detects high packet loss, the system proactively issues LTP frames before decoding pipelines collapse entirely. This bypasses broken dependencies, minimizes recovery time, and prevents keyframe-induced bitrate spikes.

- LTR frames reside inside encoder and decoder frame buffers far longer than standard short-term references. - LTP frames prevent video freezes without the massive bit cost of full IDR keyframes. - Proactive LTP dispatch significantly elevates user experience over lossy networks.

---

[20:08](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1208s) Adoption Milestones, Hardware Demands, and Future Outlook

Eric reflects on the engineering trajectory that made widespread AV1 deployment possible. Three years earlier, running AV1 on low-end mobile devices was unfeasible due to computational constraints. Today, Meta ships AV1 across the majority of mobile devices in its fleet, delivering verified visual quality gains alongside measurable increases in user engagement. Eric assesses the ecosystem status: “We believe this is a good industry moment right now to adopt AV1 in RTC.”

Looking to the future, Eric issues an explicit call to the hardware industry, encouraging system-on-chip (SoC) vendors to implement AV1 hardware acceleration across all market tiers, covering both hardware decoders and hardware encoders. Beyond traditional person-to-person video calls, emerging real-time domains stand to gain substantial efficiency from AV1: - **Conversational AI Applications:** Real-time interactive AI video agents require high-efficiency, low-latency video compression to operate smoothly at scale. - **Industry Collaboration:** Eric envisions community members presenting advanced AV1 deployments for AI streaming at future events, inviting colleagues to advance next-generation real-time video architectures together.

## Notable quotes > [20:08](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1208s) We believe this is a good industry moment right now to adopt AV1 in RTC. — Yu Chen Sun Eric

## Mentioned Yu Chen Sun Eric, Meta, Messenger, AV1, H.264, Android, iOS, Google Pixel 4a, libaom, dav1d, Apple, Google, Video Buffer Verifier

[Source](https://www.youtube.com/watch?v=QO1LLoMJBM8)

中文简报

Meta通过自研编码器将AV1视频推向绝大多数移动设备 Meta通过将自研低复杂度编码器与机器学习设备评分以及动态H.264回退相结合，克服了移动端发热限制，在其通话设备网络中全面推广AV1。 Yu Chen Sun Eric详细介绍了Meta在实时通信（RTC）中将AV1视频压缩扩展到绝大多数移动设备的工程历程。为了应对严苛的计算、电池和发热限制，Meta设计了一套自研低复杂度软件编码器，将解码迁移至dav1d，并部署了基于机器学习的设备准入评估框架以及向H.264的动态运行时回退机制。该架构还集成了视频缓冲区检验器（VBV）码率控制以防止缓冲区膨胀，同时采用分层时间层和长期参考（LTR）帧技术，在发生丢包时无需昂贵关键帧即可维持码流连续性。 Meta通过将自研低复杂度软件编码器与dav1d解码器相结合，已在其网络中的绝大多数移动设备上推广部署AV1。 (3:30) 为解决Android硬件碎片化和过热降频问题，Meta实施了一种基于机器学习的遥测模型以输出持续的设备“RTC评分”，并配备了向H.264的动态运行时回退机制。 (7:09) 实时码率控制采用视频缓冲区检验器（VBV）漏桶模型，以防止瞬时码率超标造成超出延迟预算的缓冲区排空延迟。 (13:25) 丢包抗性通过将动态切换的双层时间可扩展性与长期参考（LTR）恢复帧相结合来实现，避免了高代价的IDR关键帧风暴。 (15:58)

Reading summary

Meta has expanded AV1 deployment across the majority of its mobile fleet for real-time communication by developing a custom low-complexity software encoder, machine-learning-driven device qualification, and dynamic runtime fallbacks. Engineer Yu Chen Sun Eric argues that while the codec delivers significant visual and efficiency gains under constrained bitrates, production success depends on system-level adaptations for rate control, device thermals, and packet loss. Balancing Compression Efficiency Against Real-Time Latency (0:05) Real-time communication (RTC) imposes strict constraints compared to video-on-demand: end-to-end latency must remain below 300 milliseconds, multi-pass encoding is impossible, and mobile clients must encode and decode simultaneously. In a Messenger test at a 100 kbps cap, AV1 produced a crisp, detailed stream where H.264 appeared blurry and blocky. AV1 also integrates native RTC tools like Reference Picture Resampling, which dynamically changes resolution without generating an intra keyframe, and dedicated screen-sharing tools in its Main profile. Overcoming Complexity with Custom Encoders and Dav1d (3:30) Because standard AV1 software encoders can demand twice the computational footprint of H.264—causing a 14% power increase in early Google Pixel 4a tests—running AV1 unconstrained risks frame drops, UI freezes, or app crashes. Meta addressed this by building an in-house RTC-focused encoder featuring an ultra-low-complexity preset that matches H.264's resource profile while preserving AV1's coding tools. On the receiving end, extensive A/B testing led Meta to replace libaom with dav1d, achieving improved power efficiency, reliability, and user engagement. ML-Based Device Qualification and Dynamic Fallbacks (7:09) Static heuristic filters such as RAM or OS version failed to reliably classify fragmented Android devices. Meta implemented a machine-learning model that processes production telemetry to assign each device model an continuous RTC score, which dictates whether the client can encode, decode only, select specific presets, or fall back to H.264. Because live thermal throttling can still cause CPU frequencies to collapse during sustained calls—even on devices released in 2023—Meta introduced dynamic safeguards. The client tracks encoding latency to step down presets, monitors RTCP receiver decoding latency to prevent peer stalls, and pre-negotiates H.264 during call setup to execute seamless fallback without dropping media sessions. VBV Buffer Control and Error Resilience (13:25) Instant bitrate spikes cause critical latency spikes under the Video Buffer Verifier (VBV) leaky bucket model; for example, a 35k bit accumulation at 100 kbps takes 350 ms to drain, triggering network congestion and visual freezing. Meta's rate controller continuously tracks VBV buffer occupancy across threads and lowers downstream bit budgets when overshoots occur, while also managing dynamic resolution shifts and avoiding undershoots that degrade bandwidth estimation. To prevent packet-loss-induced keyframe storms, Meta uses a two-layer temporal scalability hierarchy alongside Long-Term Reference (LTR) frames. Layer 0 provides an autonomous base frame rate protected by heavier Forward Error Correction (FEC), while Layer 1 enhancement frames can be dropped or have retransmissions suppressed if round-trip times are high. If dependencies fail, the encoder issues an LTR-predicted (LTP) frame referencing an acknowledged cached frame, bypassing broken chains without the massive bit penalty of an intra keyframe. Eric emphasizes that AV1 is now viable for widespread RTC deployment across mobile devices, but calls on system-on-chip vendors to accelerate AV1 hardware decoding and encoding across all pricing tiers. Meta anticipates that low-latency AV1 pipelines will become critical for emerging conversational AI video agents and interactive streaming applications.

精炼阅读摘要

Meta媒体基础架构团队工程师Yu Chen Sun Eric指出，通过自主研发低复杂度编码器、引入基于机器学习的设备准入系统、建立动态运行时回退机制，以及优化码率控制与抗丢包策略，Meta已成功将AV1视频编解码器规模化部署至旗下应用的大多数移动设备，打破了AV1在实时通信（RTC）中计算开销过大的瓶颈。克服算力瓶颈与编解码器重构 (3:30) 实时通信与点播视频存在本质区别：端到端延迟必须严格控制在300毫秒以内，无法依赖多遍编码或深度缓冲，且移动端设备需同时执行编码与解码任务，在多人通话场景下甚至要并发处理三个流。初始评估显示，在Google Pixel 4a上运行开源AV1编码器相较于H.264会导致功耗上升14%，直接阻碍了移动端落地。Meta认为现代编解码器具备更丰富的工具集，因而有空间在画质与复杂度之间寻找更优平衡。为此，Meta自研了针对RTC场景的AV1编码器，开发出计算消耗与H.264相当的超低复杂度预设，使得在100 kbps的极低码率限制下，AV1仍能保持画面清晰细致，而H.264则出现明显模糊与块效应。在解码侧，Meta通过大规模A/B测试从libaom迁移至dav1d，显著优化了移动端的电量消耗与运行稳定性，并带动了用户参与度指标的提升。 ML设备分级与动态运行时自适应 (7:09) 面对Android生态数千种机型的高度碎片化，Meta发现基于内存大小、系统版本或出厂年份等静态规则的白名单机制均无法准确预测设备性能。团队因而设计了基于机器学习的设备准入系统，从实际线上通话中提取低层性能遥测指标，由模型计算出连续的“RTC评分”（RTC Score），动态判定设备是支持编解码、仅支持解码、匹配何种预设，抑或退回H.264。然而，现场测试发现即便2023年发布的部分高端旗舰机，在持续通话负荷下也会因机身过热降频导致CPU性能断崖式下跌。为此Meta构建了三重动态保障：首先依据实时编码延迟自适应调节预设档位；若降至最低预设仍无法满足延迟要求，则利用呼叫建立阶段预协商的SDP平滑切换至H.264；同时引入RTCP接收端反馈回路监控对端解码延迟，防止低端接收机算力不足导致画面卡死，并结合低电量等设备健康信号主动触发回退保护。码率控制与VBV漏桶模型优化 (13:25) 在实时音视频传输中，仅仅监测平均码率不足以保障体验，瞬时码率过冲会瞬间填满网络队列并造成严重拥塞与丢包。Eric基于视频缓冲区验证器（VBV）漏桶模型指出，在100 kbps带宽限制下，若单帧产生20k比特且缓冲队列中尚有15k比特未发积压，排空总计35k比特的负荷需要耗时350毫秒，这将严重突破端到端延迟低于200毫秒的RTC体验红线，直接导致画面卡顿。为此，Meta工程团队在编码器内引入了对VBV缓冲水位的实时监控算法，一旦侦测到过冲风险便立刻削减后续帧的比特预算，将传输排队延迟拉回安全区间。该码率控制体系还协同处理了通话中目标带宽剧烈波动、动态分辨率调整（例如利用参考图像重采样技术避免发送关键帧），并有效防止了码率欠冲误导带宽估计（BWE）算法导致码率无法正常爬升的问题。时域分层与长期参考帧抗丢包 (15:58) 传统线性帧间预测链条在遭遇网络丢包时极易中断，迫使接收端请求重传乃至迫使发送端生成高码率、画质受损的IDR关键帧，进而引发网络拥塞恶性循环。为了在丢包环境下维持流畅播放，Meta在AV1中融合了两项抗丢包技术。第一项是双层时域可扩展性架构：基础层（Layer 0）以较低帧率独立解码，并获得高比重前向纠错（FEC）保护；增强层（Layer 1）提供高帧率补充，若网络往返时延（RTT）过高导致重传无法赶上渲染，发送端会自适应抑制增强层的重传请求。由于层次化预测相较于无限制预测存在一定的编码效率损耗，该分层机制仅在网络丢包加剧时动态激活，网络恢复后即刻切回常规模式。第二项是长期参考帧（LTR）机制：编码器在缓冲区内长期保留经确认的参考帧，发生丢包时直接生成参考该帧的LTP帧绕过断裂链条，系统甚至会在传输层检测到高丢包时主动下发LTP帧，实现亚秒级的极速恢复并避免关键帧风暴。 Meta团队认为当前已进入在RTC领域规模化采用AV1的关键窗口期，并向硬件行业发出呼吁，期盼系统级芯片（SoC）厂商在各产品层级普及AV1硬件编解码加速。除传统点对点视频通话外，Meta指出具备高压缩效率与超低延迟特性的AV1架构，将是未来支撑大规模实时交互式对话AI视频智能体（Conversational AI）等新兴应用落地的核心技术基座。

Detailed notes — zh

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 10 月 30 日发布，主讲人为来自 Meta 媒体基础设施（Media Foundation）团队的工程师 Yu Chen。页面未提供具体的详细简介，视频完整记录了他在技术大会上分享 Meta 如何历时三年将 AV1 编解码器大规模落地至移动端实时音视频通信（RTC）的技术演讲。 ## TL;DR 演讲系统复盘了 Meta 将新一代视频编解码标准 AV1 推向旗下 Messenger 等全系移动应用实时通话场景的三年演进历程。针对 AV1 算力开销巨大与 Android 生态碎片化严重的痛点，Meta 通过自研超低复杂度编码器内核、引入基于机器学习的设备资质预测模型，以及建立温控降频下的动态平滑降级与对端协同调度机制，攻克了移动端算力瓶颈。在端到端系统层面上，团队结合 VBV 延迟控制、两层时间分层（Temporal Layer）及长期参考帧（LTR）技术，彻底化解了码率过冲与网络弱网丢包难题，最终在全球绝大多数设备上实现了显著的画质与用户互动指标提升，并呼吁芯片厂商加速硬件 AV1 普及以迎接实时对话式 AI 时代。 ## Atmosphere & style 整场演讲呈现出典型的大厂硬核工程技术分享风格，语调沉稳、严谨且充满工程师的务实与自豪。Yu Chen 以“三年前后的能力对比”为叙事主线，由浅入深地层层剖析从算法内核到全链路系统联调的技术抉择，逻辑严密而富有现场说服力。倾听时既能真切感受到移动端算力与发热约束下的步步维艰，又能体会到当技术壁垒被逐一攻破、系统最终赋能海量长尾设备时的宏大工程成就感。 ## Bulletin - [00:05](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=5s) Meta 历时三年成功将 AV1 编解码器推广至绝大多数主流移动设备，覆盖旗下全系应用；在 100 kbps 的极端低码率实测中，AV1 的画面清晰度压倒性超越传统 H.264。 - [07:09](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=429s) 针对移动设备算力瓶颈与 Android 生态碎片化，Meta 摒弃静态硬件规则，构建了基于真实生产环境通话日志的机器学习（ML）评估系统，实时量化机型 RTC 分数并指导编解码策略。 - [10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) 为防止设备发热导致的温控降频（Thermal Throttling）造成卡顿，系统引入了自适应预设调度、编解码器向 H.264 无缝降级、以及基于接收端解码延迟反馈的端到端协同机制。 - [13:25](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=805s) 在弱网抗性方面，Meta 针对 AV1 落地了基于漏桶模型的 VBV 延迟控制、两层时间层（Temporal Layer）以及长期参考帧（LTR）技术，大幅降低了弱网下对笨重关键帧的依赖。 ## Key takeaways - [01:43](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=103s) 实时通信（RTC）要求端到端延迟严格压在 300 毫秒以内，无法使用点播视频的多遍编码和长缓冲区，对瞬间码率过冲容忍度极低。 - [01:43](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=103s) AV1 具备专为现代 RTC 打造的原生特性，包括主配置文件内建的屏幕共享工具以及支持无关键帧动态切换分辨率的参考图像重采样（Reference Picture Resampling）。 - [03:30](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=210s) 由于开源 libaom 编码器在移动端功耗较 H.264 激增 14%，Meta 转向自研 RTC 专用 AV1 编码器，并引入运算开销与 H.264 持平的超低复杂度预设。 - [03:30](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=210s) 在解码端，Meta 经过横向对比与线上大规模 A/B 测试，全量切换为能效与稳定性更优的 dav1d 解码器，并直接带来了用户留存与通话时长的增长。 - [07:09](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=429s) Android 设备的内存或芯片出厂年份无法真实反映计算性能，Meta 采用机器学习模型分析海量通话性能信号，动态预测每台设备的综合 RTC 分数。 - [10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) 即使是新款高端旗舰机，在持续通话中也会因温控降频而损失可用算力，迫使系统必须在通话生命周期内依据编码延迟实时动态调节预设。 - [10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) 通话建立之初信令协商层便同时就绪 AV1 与 H.264 双通道，确保在算力耗尽或对端解码超载时能够毫秒级平滑降级，杜绝双向来回震荡。 - [10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) 低端机作为接收端时往往面临解码瓶颈，发送端通过监听对端回传的解码延迟（Decoding Latency）与本地电量，前瞻性决定是否降级回退至 H.264。 - [13:25](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=805s) Meta 引入基于漏桶模型的 VBV 延迟指标监控瞬间码率过冲，避免因为网络排队耗时超标而挤爆传输信道或造成画面冻结。 - [13:25](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=805s) 码率欠冲（Undershooting）同样具有破坏性，会导致网络带宽估计（BWE）模块误判网络容量不足，从而延缓提速并牺牲画质。 - [15:58](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=958s) AV1 两层时间层架构将基础层与增强层解耦，在丢包时允许接收端直接丢弃增强层以维持流畅播放，系统优先为基础层配置前向纠错码（FEC）倾斜保护。 - [18:53](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1133s) 结合长期参考帧（LTR）技术，编码器在丢包破损时直接生成轻量的 LTP 预测帧快速恢复画面，替代了传统消耗海量带宽且易造成拥塞的关键帧方案。 - [20:08](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1208s) Meta 呼吁芯片设计厂商（SoC Vendors）在各层级移动芯片中全面普及硬件 AV1 编解码器，以释放端侧算力并支撑未来对话式 AI 等实时交互场景。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=5s) 开启新篇章：AV1 落地 Meta 移动端 RTC 的三年蜕变来自 Meta 媒体基础设施（Media Foundation）团队的 Yu Chen 登台开场。回溯三年前，他曾在这里分享团队将 AV1 引入实时通信（RTC）领域的早期探索。彼时受限于算力，Meta 只能在高配或中端机型上小范围试验性支持 AV1。而时至今日，AV1 已经成功铺设到了绝大多数移动设备上。Yu Chen 难掩兴奋地感慨，将这一全新编解码器带入 RTC 是一场漫长而艰辛的技术长征，而过去三年里团队取得的重大技术突破，终于让大规模普及成为了现实。在实时音视频通信系统中，视频编解码器扮演着最核心的角色，其核心任务主要有两项：一是尽可能无损地压缩视频体积，不让画面变差；二是在恶劣网络环境下维持视频播放的稳健性。为了直观展现 AV1 的威力，Yu Chen 展示了一组在 Messenger 应用上的端到端对比评测：两台 Android 手机并排进行视频通话，左侧采用传统的 H.264 编码，右侧采用 AV1 编码。在码率均被严苛限制在 100 kbps 的极端恶劣网络条件下，H.264 的输出画面已经模糊不堪，而 AV1 画面依旧清晰细腻，展现出了压倒性的压缩画质优势。 - **三年前的局限**：AV1 仅能在高端和部分中端移动设备上启用。 - **现阶段成果**：AV1 已全面支持绝大多数主流移动设备，覆盖 Meta 旗下全系应用（Family of Apps）。 - **极限低码率评测**：在 100 kbps 码率约束下，Messenger 上的 AV1 画质显著优于 H.264。 ### [01:43](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=103s) RTC 系统的极端严苛环境与 AV1 的天然优势对于不熟悉实时通信（RTC）的受众，Yu Chen 深入拆解了 RTC 与传统点播视频（VOD）在底层逻辑上的根本差异。点播视频可以采用多遍编码（Multi-pass Encoding），且播放端有充足的时间进行数据缓冲；而 RTC（典型如双向视频通话）必须对端到端延迟保持极致苛刻的控制。通常系统的端到端延迟必须压在 300 毫秒以内，一旦超出这个阈值，用户便会立刻感知到明显的对话停顿与卡顿。因此，RTC 架构师始终在“超低延迟”与“高画质”之间走钢丝——编码器不能做复杂缓冲与多遍重算，任何短暂的码率突增（Bitrate Overshoot）都会直接导致接收端画面冻结，甚至造成网络拥塞。更棘手的是，RTC 客户端设备往往必须同时承担实时的编码与解码双重任务，这会消耗巨大的电池电量与 CPU 算力。然而，AV1 之所以值得团队历时数年倾力攻坚，是因为它具备多项专为现代 RTC 量身定制的杀手级特性。除了在底层工具集上提供比 H.264 高得多的压缩效率外，AV1 的主配置文件（Main Profile）原生内建了屏幕共享（Screen Sharing）编码工具；此外，它还支持诸如参考图像重采样（Reference Picture Resampling）等特性，允许编码器在不强制发送庞大关键帧（Keyframe）的前提下，随网络抖动快速、平滑地动态切换视频分辨率。正如 Yu Chen 所强调的那样：“AV1 usually has much higher computing cost than H.264 AVC.” - **端到端延迟红线**：RTC 严格要求全链路延迟低于 300 毫秒，否则通话体验将出现严重迟滞。 - **系统限制**：严禁多遍编码，解码端缓冲区极小，对码率过冲（Bitrate Overshoot）容忍度极低。 - **参考图像重采样（Reference Picture Resampling）**：支持在无关键帧介入的情况下动态调整分辨率，极大节省带宽并避免卡顿。 - **原生屏幕共享工具**：AV1 主配置文件深度集成屏幕共享算法，契合多样化远程协作场景。 ### [03:30](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=210s) 算力黑洞与碎片化困局：为何 AV1 的规模化耗时数年？既然 AV1 优势如此明显，为何将其全面推向生产环境耗费了 Meta 团队数年心血？Yu Chen 一针见血地指出，核心阻碍就在于设备算力瓶颈（Capacity）。AV1 的编码复杂度往往数倍于 H.264，单一编码器实例的计算开销很容易飙升到 H.264 的两倍以上。在多人或群组视频通话场景下，一台移动设备甚至需要同时应付多路并发的编解码任务。如果贸然在算力不足的机型上强行启用 AV1，轻则导致视频剧烈冻结，重则引发客户端 App 直接崩溃。这种挑战在 Android 生态中被无限放大。Meta 的产品需要兼容市面上数以千计的 Android 机型，而随着团队尝试将 AV1 推向更多长尾设备，层出不穷的碎片化问题接踵而至。针对这一难题，Yu Chen 给出的论断掷地有声：“Picking the right encoder has been our most critical decision.” 三年前，Meta 采用开源的 libaom 编码器在 Pixel 设备上测试视频通话，结果发现功耗较 H.264 激增了 14%，这种电池消耗在移动端根本无法上线交付。但团队深信，现代编码器丰富的工具集并不等同于必然的超高计算负荷，只要合理平衡质量与复杂度，AV1 完全可以在低端手机上运行。为此，Meta 自研了专为 RTC 打造的内部 AV1 编码器，开创性地加入了“超低复杂度预设（Ultra-low Complexity Preset）”，在运算开销上直接对齐 H.264，并辅以自适应预设调度机制。在解码器侧，尽管解码复杂度低于编码，但在低端设备上依然至关重要。Meta 在早期使用 libaom 解码器后，经历了一系列开源对比测试与大规模 A/B 实验，最终全量切换为 dav1d 解码器。dav1d 卓越的功耗效率与运行稳定性，最终在生产环境中带来了显著的用户时长与互动指标（Engagement）正向收益。 ```mermaid flowchart TD accTitle: Meta RTC AV1 编解码器选型与优化演进 accDescr: 展示 Meta 在编码器与解码器演进路线上的技术选型和复杂度压降策略 subgraph Encoder["编码器演进路线 (Encoder)"] A["开源 libaom 编码器 (功耗过高，较 H.264 +14%)"] --> B["自研 Meta RTC AV1 编码器"] B --> C["引入超低复杂度预设 (计算开销看齐 H.264)"] end subgraph Decoder["解码器演进路线 (Decoder)"] D["早期方案: libaom"] --> E["开源横向对比与 A/B 测试"] E --> F["全面采用 dav1d 解码器 (高能效、高可靠、提升用户时长)"] end ``` ### [07:09](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=429s) 告别静态规则：基于机器学习的大规模设备资质判定系统选定了高效的编解码内核后，另一个核心难题浮出水面：究竟哪些 Android 设备真正有能力稳定运行 AV1？在 iOS 生态中，机型数量有限且规格统一步调清晰，只需通过标准 A/B 测试即可快速框定可用白名单；而在庞杂的 Android 阵营，仅凭硬件规格参数几乎是一场灾难。Yu Chen 透露，团队曾尝试依据硬件内存容量、芯片出厂年份或系统版本号等硬编码规则（Fixed Rules）进行判定，但没有一个指标能够准确反应真实运行性能。三年前，Meta 依靠内部硬件跑分测试系统筛选高端设备；而今天，团队构建了一套完全基于机器学习（ML）的设备资质判定基础设施。每天在 Meta 庞大的用户基数下，海量的新旧机型在真实网络环境下发起通话。这套 ML 系统的核心思想是：利用海量真实产品日志中的底层性能信号，实时评估每种机型运行 AV1 的真实表现。正如 Yu Chen 在阐述这套设计时所断言：“Instead of relying on fixed rule, could we leverage large scale product data to find the best codec setting for each device?” ```mermaid flowchart LR accTitle: 基于机器学习的设备资质预测与决策流程 accDescr: 收集真实通话中的底层性能信号并经由 ML 模型输出 RTC 分数指导编码决策 subgraph Input["数据采集层"] A["大规模真实通话日志 (Logging Pipeline)"] --> B["底层运行时性能信号 (帧率/延迟/丢包/抖动)"] end subgraph Model["模型推理层"] B --> C["ML 设备资质评估模型"] C --> D["设备综合评分 (RTC Score)"] end subgraph Policy["策略决策层"] D --> E["编解码器预设选择"] D --> F["是否启用 AV1"] D --> G["动态降级回退 H.264"] end ``` - **海量设备异构性**：老旧设备可能仅支持解码，部分设备无法承担任何 AV1 运算，盲目开启会导致体验雪崩。 - **RTC 综合性能分（RTC Score）**：模型根据设备底层的实际运行时性能特征打分，量化其处理 AV1 的实际潜力。 - **持续数据飞轮**：随着 AV1 线上流量规模不断扩大，模型持续接收新通话数据进行迭代训练，推动长尾设备上的视频质量不断优化。 ### [10:02](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=602s) 应对温控降频：运行期动态自适应与全链路协同降级即便利用机器学习模型精准锁定了支持机型，线上 A/B 测试中依然出现了意料之外的险情：部分在 2023 年刚刚发布的旗舰高配手机，在进行 AV1 实时编解码时竟然也出现了卡顿与失败。深入排查后团队发现，罪魁祸首是手机长时间通话导致的温控降频（Thermal Throttling）。设备发热后，系统主动大幅削减 CPU 核心频率，使得硬件真实计算能力骤降至基线以下。 Yu Chen 直言，单靠设备型号静态放行 AV1 存在巨大的盲区，系统必须在通话生命周期中具备动态调整复杂度的弹性能力。为此，Meta 构建了三大关键联动控制机制： 1. **自适应编码预设（Adaptive Encoding Preset）**：编码器提供从高画质到极简复杂度的宽幅预设梯度。通话过程中，客户端持续监测自身的编码延迟（Encoding Latency）。若延迟过高，立即平滑调低预设等级降算力；若延迟持续处于低水位，则适时上调预设以榨取最高画质。 2. **编解码器动态平滑降级（Codec Switch）**：当自适应预设降到最低挡位后编码延迟依旧超标，客户端将果断触发降级逻辑，无缝切换回 H.264。为了防止画面闪断，通话建立之初信令协商层就会同时就绪 AV1 和 H.264 双编解码通道。预设调节与编解码降级策略紧密耦合，彻底杜绝了在两种编解码器之间来回震荡（Switching back and forth）。 3. **基于对端反馈与设备健康的协同调度**：除了自身编码能力，解码端的算力水准同样是致命短板。当高配手机与低端机视频通话时，高配机可以轻松编码 AV1，但低配对端却无法实时解码，造成画面严重卡死。Meta 引入了反向信令机制：接收端持续向发送端反馈自身的解码延迟（Decoding Latency），一旦发现对端无法实时解码，发送端主动降级回退至 H.264。同时，系统还会实时监控本地电池电量等健康指标，当电量不足时主动回退至低功耗的 H.264，最大程度确保通话不中断。随着上述整套组合拳的落地，AV1 终于安全部署至全球绝大多数移动终端。Yu Chen 随即引出下一个核心课题——深入到码率控制（Rate Control）与抗弱网容错（Error Resilience）层面，开展严谨的端到端系统级优化。在 RTC 通话中，码率必须实施严格的恒定码率（CBR）控制，哪怕只是瞬时发生的极短暂码率过冲（Bitrate Overshoot），也会瞬间挤爆传输信道，引发严重的网络拥塞与延迟剧增。 - **温控降频陷阱**：2023 年上市的高端旗舰机型在持续高负载通话时遭遇 Thermal Throttling，实际可用 CPU 性能甚至不及正常状态的一半。 - **双编解码器预热协商**：在通话建立阶段同时协商 AV1 与 H.264，保证在极端卡顿风险下能够毫秒级热切换。 - **对端解码延迟回传**：高配手机给低端手机推流时，依据低端手机的解码能力反馈动态决定编码策略，杜绝单向卡死。 - **电量感知策略**：当电池电量处于低电平状态时，客户端自动禁用高耗能的 AV1，切换至成熟低耗的 H.264，保障通话续航。 ### [13:25](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=805s) 码率控制与 VBV 延迟管理在实时通信（RTC）中，仅仅评估平均码率是远远不够的，编码器瞬间的码率过冲（overshoot）同样可能直接引发严重的视频卡顿。为了量化与监控这种瞬间波动，Meta 引入了基于漏桶模型的 VBV（Video Buffer Verifier）延迟指标。演讲者举了一个非常直观的具体算例：假设当前分配给视频流的网络可用带宽为 100 kbps，系统要求编码器以 100 kbps 输出。此时编码器生成的第 $N$ 帧大小为 20k bits，但由于网络缓冲中仍残留着之前未发送完毕的 15k bits 数据，发送第 $N$ 帧实际上需要通过网络传输总计 35k bits。在 100 kbps 的带宽限制下，排队清空这批数据将耗时整整 350 毫秒。 ```mermaid flowchart LR accTitle: VBV 延迟与漏桶模型机制 accDescr: 展示视频编码数据进入缓冲区并以恒定网络带宽排队流出的过程及过冲对端到端延迟的影响。 subgraph Bucket["VBV 漏桶模型"] Prev["缓冲区残留数据 15k bits"] Curr["新编码第 N 帧 20k bits"] Total["总待发数据 35k bits"] end subgraph Network["网络传输"] BW["恒定带宽 100 kbps"] Time["排队发送耗时 350 ms"] end Prev --> Total Curr --> Total Total --> BW BW --> Time ``` 在严格要求端到端延迟控制在 200 毫秒以内的 RTC 场景下，350 毫秒的 VBV 延迟显然已经严重超标，这种过冲会直接堵塞网络管道并造成接收端画面冻结。为此，Meta 对编码器的速率控制进行了多项关键优化：编码器在编码过程中会实时严密监控 VBV 缓冲区状态，并依据缓冲信号在各个编码线程间动态分配比特预算；一旦某处出现码率过冲，算法会立刻调低后续视频帧的目标比特数，从而强行将 VBV 延迟拉回可控范围。在 RTC 复杂多变的环境下，速率控制还面临着三大核心工程挑战： - **目标码率频繁波动**：通话过程中客户端会根据网络估算在几秒内数次调整目标码率，编码器必须在码率剧烈跳变的同时确保 VBV 延迟不失控。 - **分辨率动态切换**：为了应对设备发热或带宽突降，客户端经常在通话中动态改变视频分辨率，编码器必须维持缓冲区的平稳过渡。 - **码率不足（Undershooting）的负面冲击**：不仅过冲有害，实际码率大幅低于目标值同样致命。码率欠冲会误导网络带宽估计（BWE）模块，导致系统误以为网络容量不足，从而延缓码率攀升速度并牺牲视频画质。正如演讲者强调的那样：“Overall, if we can have a accurate rate control generated stable B-ray without overshooting or undersshooting, it can significantly improve the video code quality.” --- ### [15:58](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=958s) 视频抗丢包与两层时间层（Temporal Layer）架构视频编码普遍依赖前序帧作为参考来预测当前帧，这种帧间依赖链一旦遇到网络丢包就会导致解码端链条断裂。当接收端无法解码后续帧且重传请求失败时，视频就会彻底卡住，迫使发送端重新编码并发出关键帧（Keyframe）。然而关键帧数据量庞大且单位比特的画质效率较低，频繁发送大体积关键帧会进一步恶化本就拥堵的网络。为了在抗丢包的同时尽量避免生成关键帧，Meta 针对 AV1 落地了时间分层（Temporal Layer）技术。 ```mermaid flowchart TD accTitle: AV1 两层时间层依赖架构 accDescr: 展示基础层与增强层之间的分层依赖关系，说明增强层丢包不影响基础层独立解码。 subgraph Layer1["增强层 (Layer 1) - 提供更高帧率"] E1["Enhancement Frame"] E2["Enhancement Frame"] end subgraph Layer0["基础层 (Layer 0) - 保障基础流畅度"] B1["Base Frame 1"] B2["Base Frame 2"] B3["Base Frame 3"] end B1 --> B2 B2 --> B3 B1 -.-> E1 B2 -.-> E1 B2 -.-> E2 B3 -.-> E2 ``` 时间层打破了“所有帧线性依赖前一帧”的传统结构，将帧依赖重构为时间维度的分层拓扑。Meta 在其实践中采用了双层结构： - **基础层（Layer 0）**：仅在基础层内部帧之间建立参考依赖，可作为一个低帧率视频流独立完成解码。 - **增强层（Layer 1）**：在基础层之上插入额外的预测帧；在网络状况良好时，增强层帧能够将视频流补充至更高的完整帧率。该结构最大的优势在于基础层的解码完全不依赖增强层。一旦增强层的数据包在传输中丢失，解码端依然可以平滑解码基础层画面而完全不会发生画面停顿卡死。围绕该特性，工程团队实施了深度系统级协同优化： - **前向纠错码（FEC）差异化倾斜**：鉴于基础层对通话连续性具有决定性意义，传输层会向基础层分配更高比例的 FEC 纠错包提供重点保护。 - **自适应丢弃重传**：当网络往返时延（RTT）过高时，对增强层帧发起重传往往为时已晚，系统会主动放弃重传增强层以避免迟滞。 - **网络状态感知自适应启闭**：时间分层由于限制了跨层预测的参考范围，客观上会降低一部分视频压缩效率。因此系统会动态检测丢包率：只有当网络丢包率抬升时才激活时间层以换取鲁棒性；一旦信道恢复顺畅便自动切回单层编码以榨取极限画质。谈及跨模块联合调优的必要性时，演讲者特别指出：“This kind of end to end system level optimization is critical to provide our user the best experience.” --- ### [18:53](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1133s) 长期参考帧（LTR）与精准故障恢复时间层虽然能将抗丢包能力内嵌进码流，但其层数切分较为固定。为了获得对帧依赖结构更细粒度的控制力，Meta 进一步结合了长期参考帧（Long-Term Reference, 简称 LTR）技术。在 LTR 机制下，编码器会将特定的历史帧长时间缓存在内存参考池中，将其标记为长期参考基准。当信道发生丢包且常规帧依赖断裂时，编码器不必昂贵地重新编码一个孤立的全新关键帧，而是直接编码生成一个专门参考此前已确认收讫之 LTR 帧的预测帧，即 LTR 预测帧（LTR Predicted Frame，简称 LTP）。接收端利用本地早已缓存好的 LTR 帧即可瞬间接续画面解码，大幅缩短视频从破损中恢复的时间。在系统层面的极端弱网对抗策略中，只要算法侦测到当前信道丢包率飙升至危险区间，发送端便会主动且前瞻性地直接插发 LTP 帧，使通信链路即便在恶劣丢包冲击下依然能维系基本的画面连续性。 - **粒度差异**：相比于双层时间层的静态时间切分，LTR 允许编码端在任意时间点指定某个关键状态帧存入长期缓冲，调度更加灵活自由。 - **故障恢复对比**：传统方案依赖请求后发送超大关键帧，恢复耗时长且容易引起二次拥塞；LTR 方案下的 LTP 预测帧体积远小于独立关键帧，能在极短延时内化解卡顿。 - **弱网用户体验落地**：主动预测性发送 LTP 帧的端到端联调，被工程团队验证为保障差网环境通话流畅度的核心手段。 --- ### [20:08](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1208s) 规模化落地总结与对话式 AI 的未来图景在回顾过去数年的技术攻坚时，演讲者感慨万千：就在三年前，Meta 甚至还完全无法在低端智能手机上运行 AV1 编码；而历经全方位的软件工程压榨、多线程流水线优化与算法自适应打磨之后，如今他们已经成功将 AV1 规模化推送到绝大多数主流移动设备上。这一迁移不仅带来了显著的画面质量提升，更在真实生产环境的 A/B 测试中直接转化为用户留存与通话时长的正向增长。基于此，演讲者给出了明确的行业判断：“We believe this is a good industry moment right now to adopt AV1 in RTC.” 面向未来生态演进，演讲者向行业提出了期待与号召： - **推进 SoC 硬件编解码普及**：强烈建议各大芯片设计厂商（SoC Vendors）加快在所有层级（包括中低端设备）的移动芯片组中普及硬件 AV1 编解码器（同时涵盖硬件编码与硬件解码能力），从而彻底解放端侧算力并优化功耗表现。 - **拥抱对话式 AI（Conversational AI）等全新场景**：实时多模态交互对端到端超低延迟与超高清画质提出了更高要求，AV1 高效的压缩表现将成为该领域关键的底层视频基建。在全场演讲的尾声，演讲者展望道，或许三年之后的行业大会上，在场或正在观看视频的同行就能登台分享 AV1 支撑对话式 AI 实时交互落地的实战经验，并诚挚感谢了所有与他们共同见证这一规模化演进历程的工程师伙伴。 ## Notable quotes > [20:08](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=1208s) We believe this is a good industry moment right now to adopt AV1 in RTC. — Yu Chen > [03:30](https://www.youtube.com/watch?v=QO1LLoMJBM8&t=210s) Picking the right encoder has been our most critical decision. — Yu Chen ## Mentioned Yu Chen, Sun Eric, Meta, Messenger, Android, iOS, Google Pixel, AV1, H.264 (AVC), libaom, dav1d, Conversational AI [Source](https://www.youtube.com/watch?v=QO1LLoMJBM8)

Source quotes

[20:08] Yu Chen: We believe this is a good industry moment right now to adopt AV1 in RTC. [3:30] Yu Chen: Picking the right encoder has been our most critical decision.

Source quotes

[20:08] Yu Chen Sun Eric: We believe this is a good industry moment right now to adopt AV1 in RTC.

Original transcript

[0:05] Hi everyone. Three years ago, I was here [0:08] to share our early work on how to use [0:10] AV1 to improve the realtime [0:12] communication. Back then, we can only [0:15] ship AV1 on high-end and mid-range [0:17] devices. Today, we can ship AV1 on [0:20] majority of mobile devices. I'm excited [0:23] to be here again and share our journey [0:25] on how we make it happen. [0:28] A video codec is one of the most [0:31] important part in real-time [0:32] communication or RTC. A codec does two [0:36] things. First is compress the video [0:38] without making it looks bad. Second is [0:41] keep the video playback robust when the [0:44] network is poor. At Meta, bring a new [0:48] codec AV1 to RTC has been a long [0:50] journey. I'm happy to be here to talk [0:53] about the breakthrough from the past [0:55] three years. [0:57] My name is Euchen. I'm with the media [1:00] foundation team at Meta. I work on [1:02] improving video call quality across the [1:04] meta family of X. [1:08] So first why are we interested in AV1 in [1:11] RTC? Let me show you quality comparison [1:14] between AV1 and 264. [1:17] Here we enable AV1 on Messenger app and [1:20] we make a video call on two Android [1:22] phones side by side. 264 is on the left. [1:26] AV1 is on the right. The bit rate is [1:28] restrict to 100 kbps and from the result [1:32] you can see that 264 video is blurry and [1:36] AV1 video is visually better. This is a [1:39] good example of benefit of adopting AV1. [1:43] And for the people who may not be [1:44] familiar with realtime communication or [1:47] RTC, let me start with a brief intro. [1:51] The most typical use case of RTC is [1:53] video calling. In RTC unlike video on [1:57] demand in RTC we want we have to deal [2:00] with end to end video latency. Typically [2:04] we want to keep the latency to be lower [2:06] than 300 millisecond. Once the latency [2:09] goes over the user start a delay. So [2:12] it's always struggle for RTC video [2:15] system to provide a high quality video [2:17] while trying to keep the latency low. In [2:20] this case we cannot do multipass [2:22] encoding. The decoder cannot do much [2:24] buffering. Any B overshoot would likely [2:27] introduce a video freeze on the code. An [2:31] ADC client has to do both real-time [2:33] encoding and decoding. Both consume a [2:36] lot of power. So lowering power [2:38] consumption is critical especially on [2:40] mobile devices. [2:43] Everyone has several advantage to RTC. [2:47] First it offers the better coding tool. [2:50] So better compression efficiency and [2:52] better video call quality. I've shown [2:54] you evidence to support this. Second, [2:57] everyone has some RTC tool which allow [3:00] encoder to quickly adapt to the network. [3:03] For example, there's a coding tool [3:05] called reference picture sampling. It [3:08] allow encoder to change resolution [3:10] without forcing a key frame. [3:13] Finally, there are several use case [3:15] specific to RTC which we have to take [3:18] care of. Screen sharing is a good [3:20] example. The fact that AV1 come with [3:23] screen sharing coding tool support in [3:25] the me profile is a huge plus. [3:30] So we have seen the benefit of adopting [3:32] AV1. Then what's the challenge? Why does [3:36] it take so many year? The biggest [3:39] challenge is capacity. [3:41] A1 usually has much higher computing [3:44] cost than H.264 ABC. A encoder can [3:48] easily cost twice as much. And in RTC, [3:52] the client has to do both real-time [3:54] encoding and a decoding. It's even [3:56] harder with good code. A client may have [3:59] to do three encoding and a decoding at [4:02] the same time. So before we enable AV1 [4:06] on a device, we have to make sure it's [4:08] capable of using AV1. If we mistakenly [4:11] enable AV1 on an illegible device, the [4:14] video may freeze and the e and the app [4:17] may even crash [4:20] and it's difficult to find the eligible [4:24] device on Android at meta. We are [4:27] looking at over thousands of Android [4:29] model and the further we push adoption [4:33] the more issue showed up. So in this [4:37] tail in this talk I'm going to talk [4:39] about how we use AI to solve this [4:42] problem [4:45] and I will first talk about how we [4:47] increase AV1 coverage. My favorite part [4:50] is how we use large scale product data [4:53] to find the best codec setting and then [4:56] I will talk about few method to improve [4:59] a code quality especially how we do end [5:02] to end optimization. [5:06] Picking the right encoder has been our [5:08] most critical decision. Three years ago, [5:11] we use M opensource encoder. As I [5:14] mentioned before, MV1 encoder usually [5:17] has higher computing cost. We did a [5:20] test. We enable AV1 code on Pixel 84 and [5:24] measure the power. We observe 14% more [5:27] power increase compared with H.264. [5:30] This make it challenging to ship the [5:33] feature on the mobile devices. [5:35] However, we think MA1 encoder should not [5:39] necessarily require high complexity [5:41] encoding because a modern codec actually [5:45] offer a larger coding tools coding a [5:48] larger set of coding tool. So a [5:51] well-defined encoder actually has more [5:53] opportunity to find a better tradeoff [5:56] between quality and complexity. [5:59] Ideally, if encoder can offer a wide [6:01] range of preset from low complexity to [6:04] high complexity and if the lowest [6:07] complexity can have a similar complexity [6:10] as H264, [6:12] it will make AV1 possible on low-end [6:14] phone. [6:16] This is what we built an internal AV1 [6:19] encoder for RTC. [6:21] It offer a new ultra low complexity [6:24] preset which match H264 in on complexity [6:28] and we build a mechanism to pick the [6:31] right preset for each device. [6:36] And the next question is decoder. [6:39] Decoder is lighter than encoder but it's [6:42] still critical especially on low form. [6:45] Three years ago we use libon decoder. [6:48] Over the past year, we compared [6:50] different opensource decoder and AP [6:53] testing that and finally we choose David [6:57] decoder because of it because of its [6:59] power efficiency and reliability. We [7:02] also saw ups engagement wing with [7:05] Davidit. [7:09] And then the next question is which [7:12] device can actually run AV1 [7:15] on iPhone is easy because they are [7:18] limited model. We can easily find the [7:21] list of eligible device by AB testing [7:23] them. It's hard on Android because there [7:27] are so many Android models. It's [7:29] challenging to know how much computing [7:32] resource each device really has. [7:35] We tried several [7:37] memory release year or Android version [7:41] but none of them were reliable. [7:44] Three years ago we leverage meta [7:47] in-house benchmarking system to help us [7:49] find a list of higher devices. Today we [7:53] build a new MLbased infrastructure to [7:55] handle this. [7:59] The idea is instead of relying on fixed [8:02] rule, could we leverage large scale [8:05] product data to find the best codec [8:07] setting for each device? At meta a lot [8:11] of device are using our product every [8:13] day at a very large scale. Some of [8:16] device are very old. Every year they are [8:19] new models. Not all of them can support [8:22] AVY in the same way. Some of them can [8:25] encode a video. Some of them may be only [8:28] able to decode AV1. Some may not be able [8:31] to support AV1 at all. So if we can [8:35] build an ML system to leverage this [8:38] signal to help us define the best [8:41] setting for each device, this can [8:44] improve our user experience a lot. [8:49] This is what we built. ML based device [8:52] eligibility system. Its job is simple. [8:56] Use the real data to find which device [9:00] can actually run AV1. Instead of rely on [9:03] fixed rule, we use the product data from [9:06] the real code. From our logging [9:09] pipeline, we collect the lowle [9:11] performance signal. Lo signal indicate [9:14] the a how AV1 perform on the each device [9:18] and then we feed this signal into a [9:21] model. The model will output a number [9:24] RTC scope. This go summarize the MV1 [9:27] performance on each device and we use [9:30] this goal to make a decision such as [9:33] which codec setting to use whether to [9:35] enable AV1 when to fall back to H264 [9:40] and with this system we can enable AV1 [9:43] on many more devices while maintain the [9:46] user experience stable. [9:48] Furthermore, since everyone traffic keep [9:51] growing, we keep updating our model with [9:55] the new data. We expect this will [9:58] improve the video quality over time. [10:02] Now with the ML based system, we can [10:06] find a list of eligible device. But we [10:09] have another challenge. During our AB [10:12] test, we found some of device cannot do [10:16] AV1 real-time encoding or decoding. It [10:20] was surprising some of those device are [10:23] actually highend for which were released [10:25] in 2023. [10:27] We suspect those device may lower their [10:30] CPU frequency during a call because of a [10:33] thermal throttling. So their real [10:35] performance is much lower than we [10:38] expected [10:40] because of that. [10:41] It's not enough to enable a rewind just [10:44] only based on the device model. We need [10:48] a dynamic way to adjust complexity [10:50] during a code. To do this, we built [10:54] three mechanism. First, adaptive [10:57] encoding preset. [10:59] We ask encoder to provide a wide range [11:02] of preset from high complexity to low [11:05] complexity and a device would monitor [11:08] the encoding latency and decide which [11:10] complexity to use. If the latency is too [11:13] high, it will be lower the preset. If a [11:17] latency stay low, it will higher the [11:19] preset for better quality. [11:22] And we also base on encoding complexity [11:25] to do codec switch. The idea is that if [11:28] lowering the preset is not enough and [11:31] encoding latency is still high then [11:33] client can fall back to H264. [11:36] To make it work, we negotiate two codec [11:39] at a code set up and during the code the [11:41] client can monitor device status and the [11:44] design to fall back to H2C4 if needed. [11:49] It's important that codec and the preset [11:52] has to be designed together. This help [11:55] improve the quality and avoid switching [11:58] back and forth. [12:01] And finally, A1 decoding also had a [12:04] cost. This especially important when the [12:07] high-end phone call low-end phone. We [12:10] see some problem. The sensor which is [12:12] high-end phone can encode AV1 but its [12:15] peer which is a low-end phone cannot [12:17] decode AV1. So to solve this we ask each [12:21] device send a feedback about its [12:23] decoding latency and if a device found [12:27] its peer cannot decode AV1 real time [12:31] then it will fall back to H264. [12:35] We also look at some device health [12:37] metrics such as battery level to make a [12:40] decision. For example if a battery is [12:43] low the client would fall back to 264. [12:47] This help the quality stable and the [12:49] code stay longer. With the method I just [12:52] talked about, we can ship AV1 on [12:55] majority of mobile devices. Now let's [12:58] talk about how to improve A1 call [13:00] quality. I going to talk about two topic [13:04] rate control and error resilience. The [13:06] part I found most interesting is [13:08] actually how we did end to end system [13:11] level optimization. here [13:14] rate control [13:16] and in RTC we need a constant B rate [13:19] control if if B overshoot even for [13:23] showtime it can cause network congestion [13:25] and video freeze because of that it's [13:29] not enough just checking the RGB rate we [13:32] also care about instant overshoot [13:35] to measure that we use VVV delay VBV is [13:39] a liquid bucket model here is Example, [13:43] assume the network bandwidth for the [13:46] video is 100 kbps. So we ask encoder to [13:50] encode a video with 100 kbps. Let's say [13:53] frame n was encoded as 20k bits. But it [13:58] previous has not been fully sent yet. [14:01] There are still 15k bits left in the [14:03] ber. So to send the frame m we have to [14:07] send 35k bits total at 100k bps it will [14:12] take 350 millisecond [14:15] in RTC we usually want to keep the [14:18] latency to be lower than 200 [14:21] millisecond. In this case RT VB delay is [14:26] too high and [14:29] this kind of overshoot can cause network [14:31] congestion and a video freeze. [14:34] This is why accurate rate control is [14:36] important in RTC. [14:39] We made several improvements to rate [14:41] control. During encoding, we ask encoder [14:44] to monitor the VPV buffer. It use this [14:48] signal to define how to allocate bits [14:51] across threats. If B overshoot, it will [14:54] be lower than the B beats of the [14:57] following friend. This help the VVB [15:01] delay to be kept under control. [15:04] And there are some challenging in rate [15:06] control. First in RTC the client may [15:10] change your target B rate many time [15:12] during a call. So a good encoder has to [15:16] keep the VV delay under control in this [15:18] case. [15:20] Second the client may change the [15:23] resolution very often as well. Again the [15:26] encoder has to keep VBV stable as well. [15:30] And finally, in addition to [15:32] overshooting, we also care about [15:34] undersshooting. Undersshooting can [15:37] confuse the network bandwidth [15:39] estimation. It can slow the B-ray ramp [15:41] up and the video quality. [15:45] Overall, if we can have a accurate rate [15:48] control generated stable B-ray without [15:50] overshooting or undersshooting, it can [15:53] significantly improve the video code [15:55] quality. [15:58] The next topic is error resilience. [16:01] To encode a video, a encoder would use [16:04] previous to predict the current friend. [16:08] And this introduce a friend dependency. [16:11] Which means that if we lost a friend, [16:14] the decoder will not be able to decode [16:16] the following thread and has to h ask a [16:19] sender to send a friend again. If it [16:22] still fail then decoding chain break [16:25] dependency chain break and the video [16:28] will freeze. At that time the encoder [16:31] has to send a new key frame. However a [16:34] key frame is usually larger than normal [16:36] frame and has worse quality. [16:39] To reduce a key frame we have to improve [16:42] the AV1 error resilient against the py [16:46] loss. [16:47] We did that by two method temporal layer [16:51] and long-term reference. [16:54] Let's talk about temporal layer. Instead [16:57] of letting all friend depend on its [17:00] previous friend, temporal layer simply [17:02] the friend dependency in the time [17:04] hierarchy. Layer zero is base layer. It [17:08] can be decoded by itself as a low frame [17:10] rate video stream. Layer one is [17:13] enhancement layer which provide extra [17:16] frame. It help us reach higher frame [17:19] rate if network is good. In AV1 setting [17:23] we are using a two-layer structure. The [17:26] key benefit of lampro layer is that the [17:30] decoder can decode the base layer [17:32] without depending on enhancement layer. [17:35] So which means that if we lost a packet [17:38] of enhancement layer then the decoder [17:40] can keep going without stalling. [17:44] And we also did some system level [17:47] optimization for temporal layer. First [17:50] we put more FEC on base layer because [17:53] it's more important and require more [17:56] protection. [17:57] And the second we adapted decide whether [18:00] to retransmit enhancement layer. The [18:03] idea is that if RTC is too high, the [18:06] retransmission is usually too late. Then [18:09] we may just skip it. Finally, enabling [18:13] temporal layer usually would hurt the [18:14] video quality because of less petition [18:17] efficiency. [18:19] So, we enable temporal layer adaptively [18:22] based on the network status. [18:25] If a network packet loss is high, we [18:28] turn on temporal layer. If a condition [18:31] recover, we turn off temporal layer. [18:34] This way, we can get error robustness [18:36] when we need that. If we don't need [18:38] that, we don't have to sacrifice the [18:40] video quality. [18:42] This kind of end to end system level [18:44] optimization is critical to provide our [18:48] user the best experience. [18:53] Then we can take this further. We can [18:56] control the friend dependency by [18:58] long-term reference or LTR. [19:02] Unlike temporal layer which split the [19:04] friend dependency into a fixed number of [19:06] layer AOTR give us final grand control [19:11] on friend dependency structure. A [19:13] encoder would keep a friend in the [19:16] buffer for a longer time. We call that [19:19] long-term reference frame or LTR. When [19:22] pecky loss happen and instead of sending [19:25] a key frame the encoder would send a new [19:28] friend which refer to previous LTR. This [19:32] would help a video recover faster. We [19:36] call this LTR predict fret or LTP. [19:41] Again we did some end to end system [19:42] optimization here. If we observe that [19:46] packet loss is high then we can further [19:50] proactively send the LTP [19:53] and this can improve the error [19:56] resilience significantly and it's turned [19:58] out to very crucial to to improve the [20:01] user experience under poor network [20:04] condition. [20:08] Now let me share the conclusion and the [20:10] takeaways. [20:12] Three years ago, we were not able to [20:14] ship AVR on low-end phone. Today, we can [20:18] ship AV on majority of mobile devices. [20:21] We deliver the meaningful quality [20:23] improvement from AV1. We also observe [20:25] user engagement win with AV1. We believe [20:29] this is a good industry moment right now [20:31] to adopt AV1 in RTC. [20:35] What's next? First we encourage SOC [20:39] vendor to invest AV1 hardware codec on [20:43] all device tier including encoding and [20:46] decoding. We believe this can improve [20:49] user experience a lot and lay a lot of [20:54] use case which can benefit from adopting [20:57] AV1 as well. Conversational AI is a good [21:00] example. [21:02] Maybe three years from now, one of you [21:04] who is watching the video can stand here [21:07] with me and talk about how to use AVY in [21:11] conversational AI. I definitely looking [21:13] forward to that opportunity and thank [21:16] you for letting me share this journey [21:18] with you.