Nvidia Backs Hybrid AI Video Layer to Beat Silicon Area Limits

AI Compression for AOM from NVIDIA | Manindra Parhy from NVIDIA — @Scale

@Scale

Because pure neural compression exceeds mobile chip budgets and conventional codecs face diminishing returns, Nvidia argues a hybrid AI extension running within 5 to 50 TOPS offers the only scalable path forward.

AI-assisted summary of AI Compression for AOM from NVIDIA | Manindra Parhy from NVIDIA — original source. Check the original for full context.

Source published:

Key ideas

  • Traditional video codec development is facing an unsustainable silicon bottleneck as Moore's Law slows, compounding hardware area across legacy standards while cutting-edge formats like AV2 yield diminishing returns of under 1% coding efficiency per tool. (0:05)
  • NVIDIA is proposing a scalable hybrid AI extension layer requiring only 5 to 50 INT8 TOPS, allowing real-time execution across modern mobile devices and PCs rather than the >200 TOPS demanded by pure end-to-end neural codecs. (9:10)
  • The framework supports two operation modes: an out-of-loop post-processing mode with minimal signaling overhead for artifact cleanup, and an in-loop residual bitstream mode adding ~10% bitrate overhead to faithfully restore high-frequency textures. (12:46)
  • NVIDIA projects the hybrid enhancement bitstream approach will achieve industry adoption within two to three years, serving as a pragmatic stepping stone toward fully learned, end-to-end AI codecs. (17:54)

Detailed notes

## Background Published on YouTube by the @Scale channel on October 30, 2026, this presentation features Manindra Parhy from NVIDIA discussing AI video compression for the Alliance for Open Media (AOMedia). Although no formal video description was provided, the talk focuses on integrating neural enhancement layers into traditional codec pipelines to overcome physical silicon constraints.

## TL;DR As the end of Moore's Law and diminishing compression returns make traditional fixed-function video codecs increasingly unsustainable, NVIDIA proposes a hybrid, scalable AI extension architecture. Rather than relying on compute-prohibitive end-to-end neural networks or expensive ASIC modifications, the approach pairs a standard base codec with an AI enhancement layer running within a practical 5 to 50 TOPS mobile hardware budget. The framework supports both out-of-loop post-processing and residual bitstream modes, using consumer neural accelerators to restore high-frequency textures like facial hair and foliage at low bitrates. This model establishes a pragmatic roadmap for the next two to three years, serving as a stepping stone toward fully learned end-to-end AI codecs.

## Atmosphere & style The presentation carries the measured, analytical cadence of a specialized semiconductor deep dive, grounding future-facing neural technology in practical silicon economics. Manindra Parhy speaks with technical clarity and quiet confidence, replacing marketing hype with disciplined hardware budget assessments and interactive visual proof. Watching the talk feels pragmatic and engineering-focused, turning an industry bottleneck into a lucid, step-by-step technological compromise.

## Bulletin - [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) Traditional video codec development is facing an unsustainable silicon bottleneck as Moore's Law slows, compounding hardware area across legacy standards while cutting-edge formats like AV2 yield diminishing returns of under 1% coding efficiency per tool. - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) NVIDIA is proposing a scalable hybrid AI extension layer requiring only 5 to 50 INT8 TOPS, allowing real-time execution across modern mobile devices and PCs rather than the >200 TOPS demanded by pure end-to-end neural codecs. - [12:46](https://www.youtube.com/watch?v=GWY5cpzKYms&t=766s) The framework supports two operation modes: an out-of-loop post-processing mode with minimal signaling overhead for artifact cleanup, and an in-loop residual bitstream mode adding ~10% bitrate overhead to faithfully restore high-frequency textures. - [17:54](https://www.youtube.com/watch?v=GWY5cpzKYms&t=1074s) NVIDIA projects the hybrid enhancement bitstream approach will achieve industry adoption within two to three years, serving as a pragmatic stepping stone toward fully learned, end-to-end AI codecs.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) Video accounts for approximately 95% of total internet traffic, driving continuous pressure for superior compression efficiency across standards like H.264, HEVC, AV1, and AV2. - [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) Codec stickiness forces chip designers to permanently support legacy standards like H.264 alongside modern formats, creating unsustainable cumulative silicon area growth as Moore's Law ends. - [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) Conventional block-based compression algorithms face diminishing returns, with new tools in AV2 often delivering less than a 1% coding gain despite requiring complex silicon implementations. - [05:53](https://www.youtube.com/watch?v=GWY5cpzKYms&t=353s) AI codecs analyze scenes at a macro structural level rather than micro-granularity 4x4 pixel blocks, retaining semantic features and edges that better align with human visual perception. - [05:53](https://www.youtube.com/watch?v=GWY5cpzKYms&t=353s) Unlike traditional codecs with static mathematical filters frozen into hardware specifications, neural models learn from large video datasets and can be retrained or fine-tuned post-deployment. - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) Consumer AI compute capacity is growing rapidly, with smartphones currently delivering 35–60 INT8 TOPS, laptops offering ~120 TOPS, and budget mobile silicon projected to exceed 100 TOPS by 2028–2029. - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) Pure end-to-end neural compression requires over 200 TOPS to outperform modern traditional codecs, exceeding the thermal and power limits of mobile platforms. - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) Embedding neural tools directly inside block-based codecs introduces prohibitive bus latencies between fixed-function ASICs and neural accelerators, or requires an impractically large dedicated ASIC footprint. - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) NVIDIA's hybrid extension preserves the traditional codec as a base layer and adds an AI enhancement pass, operating comfortably within a viable 5 to 50 TOPS compute envelope. - [12:46](https://www.youtube.com/watch?v=GWY5cpzKYms&t=766s) The post-processing mode operates out-of-loop using lightweight signaling metadata to clean up compression artifacts and apply super-resolution without noticeable bitrate penalties. - [12:46](https://www.youtube.com/watch?v=GWY5cpzKYms&t=766s) The residual bitstream mode transmits compressed residual latents at an estimated 10% bitrate overhead, significantly improving overall visual fidelity at iso-bitrate. - [14:28](https://www.youtube.com/watch?v=GWY5cpzKYms&t=868s) In the residual pipeline, base layer motion vectors warp the previous enhancement frame, and entropy coding leverages probability distributions derived directly from the base decoded video. - [15:47](https://www.youtube.com/watch?v=GWY5cpzKYms&t=947s) Low-bitrate visual evaluations at 1.3 Mbps show that AI extension layers successfully recover fine details lost in standard AV2, such as foliage, beard textures, and facial skin tones. - [17:54](https://www.youtube.com/watch?v=GWY5cpzKYms&t=1074s) The long-term industry trajectory moves from external post-processing filters toward hybrid enhancement bitstreams over the next 2–3 years, eventually culminating in fully learned end-to-end AI codecs.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) The History of Video Codecs and the Limits of Silicon

Video coding standards have evolved over nearly four decades, tracing back to the emergence of H.261 in 1988. In the years that followed, standard-setting organizations such as ITU-T and ISO developed the MPEG series, but the defining inflection point arrived in 2003 with H.264 (AVC). H.264 revolutionized media consumption by enabling high-definition television and digital streaming across a vast ecosystem of consumer devices. Today, video accounts for roughly 95% of all internet traffic. Keeping pace with the staggering demand for network bandwidth and data storage has driven relentless pressure for superior compression efficiency, yielding subsequent standards such as HEVC (H.265) and VVC (H.266). In parallel, Google launched the open-source VP8 and VP9 codecs around 2010, eventually joining forces with NVIDIA and other industry leaders to establish the Alliance for Open Media (AOMedia). This consortium created AV1 under a royalty-free licensing model, followed by the latest generation standard, AV2.

Yet for hardware manufacturers like NVIDIA, this continuous proliferation of distinct codec standards is running headlong into physical and economic barriers. Moore's Law is ending, creating deep structural bottlenecks for traditional fixed-function silicon engines. Every new standard demands dedicated silicon real estate that cannot easily be repurposed for other video formats. Compounding the issue is codec stickiness: hardware makers cannot drop support for older standards because billions of legacy devices and vast media archives still rely on them. H.264 remains pervasive despite being more than twenty years old, forcing chip designers into a scenario of compounding, cumulative silicon area growth that has become unsustainable.

At the same time, traditional standardization tools are delivering diminishing returns. Engineering teams have largely exhausted the design space of conventional block-based compression algorithms. In cutting-edge formats like AV2, individual algorithmic tools often yield less than a 1% gain in coding efficiency, yet extracting that fractional improvement requires disproportionately complex, area-hungry hardware implementations. Because modern codecs are already capable of delivering pristine video, the perceptual deltas between generations are flattening. Comparing identical footage encoded in VP9, AV1, and AV2 shows that while AV1 eliminated the heavy blockiness and facial color artifacts visible in VP9, the subsequent jump from AV1 to AV2—such as finer skin-tone preservation on a forehead—yields a much narrower subjective gain. As Manindra Parhy observed about the current standardization paradigm: “What we see is that this kind of plethora growth of codecs is not sustainable in silicon especially with Moore's law ending and it creates a structural challenges to the fixed function codec evolution.”

- H.261 debuted in 1988, followed by H.264 in 2003, HEVC, and VVC under ITU-T and ISO/MPEG. - Google introduced VP8 and VP9 starting in 2010 before co-founding AOMedia with companies like NVIDIA to build AV1 and AV2. - Video represents 95% of total internet data traffic. - Hardware implementations face rigid cycles: standardization takes roughly six years, followed by extended silicon deployment to recoup high research-and-development investments.

---

[05:53](https://www.youtube.com/watch?v=GWY5cpzKYms&t=353s) The AI Alternative: Leveraging Ubiquitous Compute and Structural Vision

To resolve the impasse between skyrocketing complexity and stagnant fixed-function silicon budgets, NVIDIA is developing an AI-driven codec strategy built around two core structural advantages: hardware ubiquity and macro-level feature perception. AI compute accelerators are no longer confined to supercomputing clusters; they have permeated personal computers, laptops, notebooks, data centers, and modern smartphones in the form of NPUs, GPUs, and TPUs. In mobile phones, NPUs already handle everyday computer vision tasks such as image cleanup or generating Genmoji. Repurposing these existing, general-purpose neural engines for video compression circumvents the need to carve out ever-larger dedicated fixed-function ASIC area for every emerging codec format.

```mermaid flowchart LR accTitle: Traditional vs AI Video Codec Processing accDescr: Diagram comparing micro-granularity block processing against macro-level AI structural analysis.

subgraph Trad["Traditional Fixed-Function Approach"] direction TB T1["Micro-Granularity<br/>(4x4 Pixel Blocks)"] --> T2["Pixel-Difference<br/>Minimization"] T2 --> T3["Hardcoded Coefficients<br/>(Static Handpicked Tools)"] end

subgraph Neural["AI Accelerator Approach"] direction TB A1["Macro-Level<br/>Structural Analysis"] --> A2["High-Level Feature<br/>& Edge Retention"] A2 --> A3["Trainable Parameters<br/>(Continual Model Updates)"] end ```

Beyond hardware efficiency, neural architectures possess fundamental algorithmic advantages over conventional codecs. Standard codecs operate at a micro-granularity level, partitioning frames into small blocks (down to 4x4 pixels) and relying on handpicked mathematical tools tuned strictly to minimize raw pixel differences between the source and the compressed output. In contrast, AI codecs analyze visual scenes at a macro structural level, recognizing continuous objects, surfaces, and semantic features to preserve structural fidelity that directly impacts perceived visual quality. Furthermore, standard codec parameters and filter coefficients are locked in stone once a specification finalizes. AI models, conversely, learn dynamic representations trained across tens of thousands of diverse video sequences and can be continually retrained or fine-tuned post-deployment. Because the physical silicon area allocated to GPUs and NPUs on modern system-on-chips dwarfs the footprint of dedicated video encoder/decoder blocks, neural codecs can tap into far deeper compute budgets to deliver superior quality. As Manindra Parhy summarized: “So essentially we can get better compression whereas not getting a proportional growth in area every time we have a new codec.”

- Traditional codecs rely on fixed, static mathematical filters established by committee standards that cannot adapt once frozen. - Neural models train against vast corpuses of diverse video data, generalizing better across disparate edge cases and content types. - General-purpose accelerators provide an expanding pool of compute without incurring dedicated, single-use silicon area penalties.

---

[09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) Evaluating AI Codec Paradigms Against Real-World TOPS Budgets

Translating AI video compression into shipping consumer technology requires designing around the lowest common denominator of hardware compute. Evaluating contemporary consumer devices shows that smartphones currently deliver between 35 and 60 INT8 TOPS on their integrated NPUs. Thin-and-light laptops and notebooks equipped with integrated GPUs provide around 120 TOPS, while desktop PCs housing discrete GPUs scale from 230 up to 1,660 TOPS. Because video streams must be universally decodable regardless of form factor, the operational ceiling is constrained by mobile devices. However, mobile AI compute is expanding rapidly, roughly doubling every 1.5 to 2 years, with low-end mobile devices projected to exceed 100 TOPS by 2028 or 2029.

```mermaid flowchart TD accTitle: Comparison of AI Video Compression Architectures accDescr: Three architectural routes for AI codecs ranging from end-to-end models to scalable hybrid extensions.

subgraph Cat1["1. Pure End-to-End Neural Codec"] direction TB E1["Full Neural Compression"] E2[">200 TOPS Required"] E3["Impractical on Mobile"] E1 --- E2 --- E3 end

subgraph Cat2["2. Hybrid Block-Tool Replacement"] direction TB H1["Embedded AI in Traditional Codec"] H2["Severe Round-Trip Bus Latency"] H3["Huge Dedicated ASIC Footprint"] H1 --- H2 --- H3 end

subgraph Cat3["3. Scalable AI Extension Layer"] direction TB S1["Traditional Base Codec"] S2["AI Enhancement Layer"] S3["5 to 50 TOPS Budget<br/>(Mobile & PC Feasible)"] S1 --- S2 --- S3 end ```

Within this operational landscape, the industry has explored three primary approaches to AI-assisted video compression: 1. **Fully End-to-End Neural Compression**: This architecture routes the entire compression pipeline through deep neural networks running exclusively on AI accelerators. To genuinely outperform state-of-the-art standards like AV1 or AV2, an end-to-end network requires more than 200 TOPS, making it completely impractical for real-time decoding on mobile chipsets. 2. **AI Tool Insertion Inside Block-Based Codecs**: This hybrid design attempts to replace individual conventional coding tools with localized neural networks within the traditional block-partitioning pipeline. However, streaming individual pixel blocks back and forth between fixed-function codec ASICs and GPU/NPU cores introduces prohibitive round-trip bus latency, destroying real-time performance. Implementing those neural sub-tools directly inside the video engine itself would require an unacceptably massive dedicated ASIC footprint. 3. **Scalable AI Extension**: Rather than disrupting the core pipeline, this architecture preserves the existing traditional codec infrastructure as an underlying base layer and introduces an AI-driven enhancement layer on top. By restricting the AI workload to an auxiliary enhancement pass, computational requirements drop to a feasible budget of 5 to 50 TOPS, allowing real-time execution across modern smartphones and PCs alike.

Detailing why NVIDIA favors this hybrid paradigm, Manindra Parhy explained: “And what we are proposing is the third approach of a scalable AI extension where the traditional codecs will be still used but we enhance this traditional video with an extension layer which is running on AI accelerators and from our work we see that it's possible to do this within 5 to 50 TOPS which means that it we can run on mobile platform and this is the approach we think is the scalable path forward which combines the 30 years of codec innovations at the same time the growth of AI accelerators.” The framework divides the task between an encoder taking the source input to emit a multiplexed bitstream and a paired decoder architecture.

- Current consumer AI capacity (INT8): - Smartphones: 35 to 60 TOPS. - Laptops / Integrated GPUs: ~120 TOPS. - Discrete Consumer GPUs: 230 to 1,660 TOPS. - AI TOPS capacity across consumer chips is doubling every 1.5 to 2 years, with budget mobile silicon expected to surpass 100 TOPS by 2028–2029. - Pure end-to-end neural codecs need >200 TOPS to beat current standards, exceeding the thermal and compute envelope of mobile devices. - The scalable AI extension approach operates comfortably within a 5 to 50 TOPS envelope, bridging traditional hardware decode engines with modern NPU/GPU hardware.

[12:46](https://www.youtube.com/watch?v=GWY5cpzKYms&t=766s) Post-Processing vs. Residual Bitstream Modes

When video data is transmitted or stored, the base bitstream goes to a standard decoder to produce video playback. In NVIDIA’s proposed framework, both the reconstructed video and the original input video are additionally fed into a neural network to generate an enhancement bitstream, which is then decoded to form an enhanced final output. Manindra Parhy outlines two fundamentally different operating approaches for this architecture:

```mermaid flowchart TD accTitle: AI Video Enhancement Modes accDescr: Diagram contrasting post-processing mode with residual bitstream mode for video enhancement. subgraph Base ["Base Layer"] A["Input & Reconstructed Video"] --> B["Neural Network"] end subgraph Modes ["Enhancement Modes"] B --> C["Post-Processing Mode"] B --> D["Residual Bitstream Mode"] end subgraph Outputs ["Decoded Results"] C --> E["Artifact Removal & Super Resolution<br>(Signaling overhead only)"] D --> F["Texture & Detail Restoration<br>(~10% Bitrate Overhead)"] end ```

The first approach is the post-processing mode, operating as an out-of-loop enhancement. Under this setup, the bitstream merely signals control metadata to the decoder side—such as specifying which model index to apply and designating which parts of the video should receive specific enhancement models. Because it transmits only signaling information, the bit rate overhead remains extremely low. This mode is primarily intended to clean up artifacts in the decoded video and scale up pixel density through super resolution.

The second approach is the residual bitstream mode, which actively transmits residual data over the wire. This mode is designed to recover high-frequency textures and intricate details that were stripped away during the base compression process. Although transmitting residual bitstreams introduces an estimated 10% overhead in bit rate, Parhy emphasizes that the trade-off is well worth it: “But if you compare that 10% increase to the quality gain that you are getting overall you get a much better compression at ISOB treat.”

- Post-processing mode uses out-of-loop execution and lightweight signaling (model indices and region indicators) to eliminate artifacts and apply super resolution. - Residual bitstream mode transmits compressed residual latents to recover lost textures, adding roughly 10% bit rate overhead while delivering superior iso-bitrate visual quality.

---

[14:28](https://www.youtube.com/watch?v=GWY5cpzKYms&t=868s) Deep Dive into the Residual Mode Architecture

Drilling deeper into the pipeline of the residual mode, Parhy explains how information flows across base and enhancement layers. The source video undergoes standard encoding and decoding along the base layer pipeline. Crucially, auxiliary information is extracted directly from this base layer—specifically motion vectors. These motion vectors are applied to the previously decoded enhancement frame to perform motion-compensated warping.

```mermaid flowchart LR accTitle: Residual Mode Architecture accDescr: Pipeline showing base layer extraction, warping, context fusion, entropy coding, and enhancement reconstruction. subgraph BaseExt ["Base Layer & Motion"] MV["Base Motion Vectors"] --> WRP["Warp Previous Enhanced Frame"] DEC["Base Decoded Video"] end subgraph Fusion ["Latent Processing"] DEC --> FUS["Context Fusion"] WRP --> FUS FUS --> Q["Quantization"] Q --> ENT["Entropy Coding"] end subgraph Output ["Enhancement Generation"] DEC -.->|"Prior Probability"| ENT ENT --> EBS["Enhancement Bitstream"] EBS --> REC["Reconstruct Latents"] REC --> OUT["Enhanced Video<br>(Display & Next Frame)"] end ```

The latents from the current base-layer decoded video and the warped, motion-compensated enhancement video are fused together to establish a shared context. Following this context fusion, the resulting latents pass through a quantization step and into entropy coding to produce the enhancement bitstream. Because entropy coding requires an accurate probability prior, the probability distribution of the latents is derived directly from the already-decoded base video. This ensures that the decoder can mirror the exact prior distribution without wasting extra transmission bits. The decoder then reconstructs the latents, combines them back into the stream, and outputs the enhanced video frame for immediate display while recycling it into the loop for the next frame.

- Motion vectors from the base layer warp previously decoded enhancement frames to provide motion-compensated temporal context. - Latents from both base decoded frames and warped references are fused and quantized before entropy coding. - Probability priors for entropy coding are inferred directly from the decoded base video, keeping transmission efficient. - Reconstructed latents regenerate the high-fidelity video frame, which serves simultaneously as the display output and the reference for subsequent frames.

---

[15:47](https://www.youtube.com/watch?v=GWY5cpzKYms&t=947s) Visual Comparisons: AV2 vs. AV2 with AI Extension

Parhy presents visual comparisons demonstrating standard AV2 compression against AV2 augmented with the AI extension. The sample clips are encoded at approximately 1.3 Mbps—a low bit rate chosen intentionally to illustrate the outer boundaries and maximum visual delta that the technology can deliver, though Parhy notes that noticeable gains persist at higher bit rates as well.

Using an interactive sliding comparison tool, Parhy walks through several distinct video sequences: - In a clip featuring trees and foliage, standard AV2 exhibits severe compression artifacts and smearing where individual leaves disappear into blocks. On the AI-enhanced side, the model reconstructs the leaf structures and fine branch details so that the scene closely mirrors the raw original footage. - In a portrait sequence of a bearded man, the standard AV2 stream obliterates the fine facial hair entirely, leaving a flat, smoothed patch. Parhy details how the neural system operates: “The network learns that feature-wise this part is missing on its own and then it puts the residual bits there so that you can see those beard is coming back.” - In a music performance clip, the hand of an instrumentalist on the right side suffers from broken, jagged lines and blurry edges in AV2, both of which are cleanly resolved by the AI enhancement. The surface of a nearby drum appears significantly sharper and cleaner, while the smeared, washed-out face of the person in the center has its skin texture faithfully restored.

---

[17:54](https://www.youtube.com/watch?v=GWY5cpzKYms&t=1074s) Roadmap: A Pragmatic Path Toward End-to-End AI Codecs

Parhy concludes by framing NVIDIA's proposal as a pragmatic bridge between legacy compression techniques and future neural paradigms. Currently, the industry relies on artifact reduction and super-resolution models that operate entirely detached from the underlying codec. NVIDIA’s strategy bridges this gap by retaining the reliable foundations of traditional codecs while embedding a neural enhancement layer directly into the bitstream, designed to run efficiently on specialized hardware accelerators.

```mermaid flowchart LR accTitle: Evolution of Video Compression Codecs accDescr: Timeline from current out-of-codec enhancement to hybrid enhancement bitstreams and finally full end-to-end AI codecs. A["Current State<br>Artifact reduction & SR<br>outside the codec"] -->|"2-3 Years"| B["Mid-Term: Hybrid<br>Traditional Codec +<br>Neural Enhancement Bitstream"] B -->|"Long-Term"| C["Future: Full End-to-End<br>Learned AI Codec<br>(Motion, Texture & Representation)"] ```

Parhy envisions this intermediate hybrid framework gaining substantial industry adoption within the next two to three years. In the longer term, the technology will evolve into fully learned, end-to-end AI codecs where all components—motion estimation, texture synthesis, and spatial representation—are learned natively within the model. Parhy reflects on the long-term outlook: “So what we are doing here is proposing a pragmatic path forward towards the learned video compression.” While acknowledging that a pure end-to-end architecture entails immense computational complexity, Parhy remains confident that future hardware acceleration will mature rapidly enough to support fully learned codecs, concluding his presentation on that forward-looking trajectory.

- **Current State:** Standalone super resolution and artifact filters operating independently outside the video codec. - **Mid-Term (2–3 years):** Hybrid architectures combining traditional base codecs with standard-compliant neural enhancement bitstreams accelerated on dedicated silicon. - **Long-Term:** Pure end-to-end AI codecs handling motion, texture, and latent representations natively as hardware accelerator throughput scales up.

## Notable quotes > [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) What we see is that this kind of plethora growth of codecs is not sustainable in silicon especially with Moore's law ending and it creates a structural challenges to the fixed function codec evolution. — Manindra Parhy

## Mentioned Alliance for Open Media (AOMedia), AV1, AV2, Google, H.261, H.264 (AVC), HEVC (H.265), ISO, ITU-T, Manindra Parhy, MPEG, NVIDIA, VVC (H.266), VP8, VP9

[Source](https://www.youtube.com/watch?v=GWY5cpzKYms)

中文简报

英伟达力推混合AI视频层以突破芯片面积限制 由于纯神经压缩超出了移动芯片的算力预算，且传统编解码器面临收益递减，英伟达认为在5至50 TOPS内运行的混合AI扩展层提供了唯一可扩展的前进路径。 随着摩尔定律的终结和压缩收益的递减使得传统固定功能视频编解码器日益不可持续，英伟达提出了一种混合式、可扩展的AI扩展架构。该方法并非依赖算力要求过高的端到端神经网络或昂贵的ASIC修改，而是将标准基础编解码器与运行在5至50 TOPS实际移动硬件预算内的AI增强层相结合。该框架支持环外后处理和残差码流两种模式，利用消费级神经加速器在低比特率下恢复面部毛发和树叶等高频纹理。该模型为未来两到三年确立了一条务实的路线图，成为迈向完全端到端学习型AI编解码器的跳板。 随着摩尔定律放缓，传统视频编解码器的开发正面临不可持续的芯片面积瓶颈，不仅在兼容传统标准时占用了更多硬件面积，而且像AV2这样的前沿格式每个工具带来的编码效率提升已低于1%，收益逐渐递减。 (0:05) 英伟达正提出一种仅需5至50 INT8 TOPS的可扩展混合AI扩展层，使其能够在现代移动设备和PC上实时运行，而纯端到端神经编解码器则需要200 TOPS以上。 (9:10) 该框架支持两种运行模式：一种是信令开销极小、用于消除伪影的环外后处理模式；另一种是增加约10%码率开销、可真实还原高频纹理的环内残差码流模式。 (12:46) 英伟达预计混合增强码流方案将在两到三年内获得行业采纳，成为迈向完全学习型端到端AI编解码器的务实跳板。 (17:54)

Reading summary

NVIDIA engineer Manindra Parhy argues that traditional fixed-function video codecs have reached physical and economic limits in silicon, requiring an evolutionary pivot toward hybrid AI extension layers that leverage general-purpose neural accelerators. Silicon Bottlenecks in Traditional Video Standardization (0:05) Video accounts for an estimated 95% of internet traffic, maintaining pressure for improved compression efficiency across successive generations from H.261 in 1988 through H.264, HEVC, VVC, AV1, and AV2. However, with Moore's Law slowing down, dedicating single-purpose silicon area for each new standard has become economically and physically unsustainable for hardware makers. Because billions of legacy devices require backward compatibility with older formats like H.264, chip designers face cumulative silicon area expansion. Simultaneously, conventional block-based compression algorithms are yielding diminishing returns. In formats like AV2, individual algorithmic tools often contribute less than a 1% gain in coding efficiency while demanding disproportionately complex hardware. Perceptual differences between generations are also narrowing, with the transition from AV1 to AV2 yielding much smaller subjective improvements than previous generational leaps. The AI Alternative and Consumer Hardware Budgets (5:53) To escape fixed-function silicon constraints, Parhy proposes tapping into the ubiquitous AI accelerators (NPUs, GPUs, and TPUs) already present across consumer devices. Unlike standard codecs that rely on static, committee-frozen mathematical filters operating on small pixel blocks, neural networks analyze macro-level visual structures and continuous semantic features. They also benefit from training across tens of thousands of video sequences and can be continually refined post-deployment. Deploying AI video compression requires designing around real-world compute envelopes. Contemporary smartphones deliver between 35 and 60 INT8 TOPS on integrated NPUs, laptops provide approximately 120 TOPS, and discrete desktop GPUs span 230 to 1,660 TOPS. Parhy notes that mobile AI capacity doubles roughly every 1.5 to 2 years, with budget mobile chips projected to exceed 100 TOPS by 2028 or 2029. Evaluating Architectural Paths to AI Video Compression (9:10) The industry has explored three main AI codec designs: pure end-to-end neural compression, hybrid block-tool replacement within conventional codecs, and scalable AI extensions. Pure end-to-end models require upwards of 200 TOPS to outperform standards like AV1 or AV2, which Parhy notes exceeds mobile thermal and compute envelopes. Meanwhile, embedding neural tools directly into traditional block pipelines introduces severe round-trip bus latency or demands impractical ASIC footprints. NVIDIA favors the third paradigm: a scalable AI extension layer built on top of traditional codecs. By preserving existing base codec hardware and offloading an auxiliary enhancement layer to AI accelerators, the computational burden drops to a workable budget of 5 to 50 TOPS. This allows real-time execution across modern mobile devices as well as PCs. Enhancement Layer Modes and Latent Processing (12:46) NVIDIA's proposed architecture operates in two configurations: post-processing mode and residual bitstream mode. The post-processing mode operates out-of-loop, transmitting only signaling metadata to direct decoder-side artifact removal and super-resolution with virtually zero bit rate overhead. The residual bitstream mode transmits compressed residual latents to recover fine, high-frequency textures lost during base compression, incurring roughly a 10% bit rate overhead while delivering superior overall visual quality at iso-bitrates. In residual mode, motion vectors extracted from the base layer warp previously decoded enhancement frames to provide temporal context. Latents from the base decoded video and the warped enhancement frames undergo context fusion, quantization, and entropy coding. Because the entropy coding prior is derived directly from the decoded base video, the decoder mirrors the probability distribution without requiring extra transmission overhead. Parhy presents the hybrid extension layer as a pragmatic two-to-three-year bridge for the industry, navigating between today's detached post-processing filters and a long-term future of fully learned, end-to-end neural codecs that handle motion, texture, and representation natively once hardware acceleration sufficiently matures.

精炼阅读摘要

NVIDIA的Manindra Parhy指出，随着摩尔定律终结与传统块级压缩算法边际效益递减，硬件视频编解码器正面临不可持续的芯片面积瓶颈；为此，NVIDIA提议采用可扩展AI扩展层方案，将传统编解码器与现代NPU及GPU算力相结合以突破物理限制。 传统编解码器的发展瓶颈与芯片面积危机 (0:05) 视频数据目前占据了全球互联网总流量的约95%。自1988年H.261面世以来，行业经历了H.264、HEVC（H.265）、VVC（H.266）以及由AOMedia推出的免版税格式AV1与AV2等迭代。然而，硬件制造商如今正遭遇严峻的物理与经济瓶颈。每一个新编解码标准都需要大约六年的标准化周期，并在硅片上占用专用的定功能ASIC面积。由于数以亿计的旧设备和海量媒体存档依赖旧格式，芯片设计无法弃用H.264等已有二十多年历史的标准，导致芯片面积呈累加性增长，在摩尔定律终结的大背景下变得不可持续。 与此同时，传统的块级压缩算法在设计空间上面临边际效益递减。在最新的AV2标准中，单个算法工具往往只能带来不足1%的压缩效率提升，却需要成倍增加复杂的硬件面积支持。由于现代格式已经具备出色的画质表现，代际间的主观感知差异正逐步收窄：从VP9跨越至AV1时尚可消除明显的块状效应与面部伪影，但从AV1进阶到AV2的视觉改善（如前额微小肤色的保留）已极为有限。Parhy指出，这种编解码标准的繁杂增长对定功能硬件构成了深层次的结构性挑战。 普及型AI算力与宏观结构视觉优势 (5:53) 为打破传统架构的死局，NVIDIA提出利用AI加速器普及与宏观特征感知的策略。当前AI计算单元已经广泛渗透至智能手机、笔记本、台式机和数据中心等设备中，以NPU、GPU和TPU的形式存在，例如手机NPU已用于日常图像修复与Genmoji生成。复用这些通用神经计算单元进行视频压缩，无需为每个新编解码器不断切割出新的专用ASIC物理区域。 在算法机制上，传统编解码器将图像分割为细小像素块（最小达4x4像素），依赖专家委员会静态固化的数学公式来最小化像素误差，一旦标准冻结便无法调整。而AI模型则通过数万条多样化视频序列进行训练，能够对画面进行宏观层面的场景和语义结构分析，保留高层次特征与边缘轮廓。此外，AI模型在部署后仍可保持持续迭代与微调，借助现代系统级芯片（SoC）上远超视频专用编解码模块的GPU与NPU算力空间，在不按比例扩张芯片面积的前提下达成更高的压缩表现。 端侧TOPS算力约束与混合扩展架构路线 (9:10) 将AI编解码推向消费级市场必须受制于端侧硬件的实际算力限制。当前智能手机的集成NPU提供约35至60 INT8 TOPS，轻薄笔记本集成GPU约为120 TOPS，而台式机独立GPU则介于230至1,660 TOPS之间。由于视频流需要跨形态通用解码，手机构成了核心算力瓶颈。尽管移动AI算力约每1.5至2年翻一番、预计2028至2029年低端设备将突破100 TOPS，但目前能够落地的技术路径依然受限。 业界探索的三种架构呈现不同可行性：第一种完全端到端的神经压缩网络需要超过200 TOPS才能超越AV1或AV2，无法在移动端实现实时解码；第二种在传统块级编解码器内部替换局部AI工具的设计，会引发ASIC与GPU/NPU之间严重的往返总线延迟，若做成全专用ASIC则面积代价过高；第三种则是NVIDIA主推的可扩展AI扩展层（Scalable AI Extension）。该方案保留传统编解码器作为基础层，仅在其上叠加AI增强层，将算力预算控制在5至50 TOPS，兼顾了传统编解码技术的积累与AI加速器的演进。 后处理模式与残差比特流架构细节 (12:46) 在该扩展架构中，输入视频与重构的基础层视频被送入神经网络以生成增强输出，主要分为两种运行模式。第一种是后处理模式，属于环外处理，码流仅向解码端发送极轻量的控制元数据（如模型索引和增强区域标识），额外码率极低，主要负责去除伪影和超分辨率放大。第二种是残差比特流模式，通过传输约10%码率开销的残差潜变量来恢复高频纹理与微观细节。Parhy指出，相对于这10%的码率增加，其在同等码率（iso-bitrate）下带来的画质增益更为显著。 在残差模式的具体数据流中，系统直接从传统基础层提取运动矢量，作用于先前已解码的增强帧以完成运动补偿扭曲（warping）。随后，基础层当前帧潜变量与扭曲后的增强帧潜变量进行上下文融合，经量化后进入熵编码生成增强比特流。为了避免额外传输开销，熵编码所需的先验概率分布直接从已解码的基础层视频中推导获得，解码端据此重建潜变量并输出高保真增强帧，同时将其作为后续帧的时域参考。 AV2增强画质实测对比与产业演进路径 (15:47) 在约1.3 Mbps的极低码率实测演示中，AI扩展层展现出超越标准AV2的效果。在树木与枝叶场景下，标准AV2出现了严重的涂抹与块状效应，叶片细节丢失，而AI增强层成功重构了树叶轮廓与树枝纹理；在胡须男性的面部特写中，标准AV2几乎抹平了细密毛发，神经网络则自主识别缺失特征并以残差比特精确恢复胡须；在乐器演奏片段中，AV2造成的演奏者手指边缘锯齿与模糊被有效修复，鼓面质感与背景人脸的皮肤纹理也得到清晰还原。 Parhy将这一方案定性为通向端到端AI编解码的务实桥梁。目前的产业阶段主要是在编解码框架外部运行独立的降噪与超分算法；NVIDIA预计在未来2至3年内，业界将逐步采纳传统编解码加标准化神经增强比特流的混合架构；长期来看，随着硬件算力吞吐量的持续扩展，视频编解码终将迈向运动估计、纹理合成与空间表征均由模型原生学习的完全端到端AI编解码形态。 NVIDIA提出的AI扩展层方案通过5至50 TOPS的适度算力消耗，为当下端侧硬件提供了可行的高清视频增强路径，但完全端到端的神经编解码普及仍高度依赖移动端算力成倍增长的长期兑现，且在跨厂商生态中推进标准化增强比特流仍需产业链协同。

Detailed notes — zh

## Background 该视频由 YouTube 频道 @Scale 于 2026 年 10 月 30 日发布，主讲人为来自 NVIDIA 的 Manindra Parhy。视频原页面未提供具体的简介与描述信息，核心内容围绕 NVIDIA 在开放媒体联盟（AOMedia）框架下探索的 AI 视频压缩技术展开。 ## TL;DR 演讲首先回顾了自 H.261 到 AV2 的视频编解码演进史，指出在摩尔定律终结、芯片硅片面积膨胀以及边际压缩收益递减的背景下，传统固定功能硬件已难以为继。NVIDIA 提出利用普及的通用 NPU/GPU 算力引入 AI 编解码，并对比了端到端纯神经压缩、块级混合替换及可扩展 AI 扩展层三种路径。团队主推仅需 5 至 50 TOPS 算力的可扩展增强层架构，涵盖轻量级后处理与残差比特流两种模式，在低码率下实现了对 AV2 细节与纹理的显著重构。最后，演讲描绘了从当前的独立后处理，到未来 2 至 3 年普及混合增强层，再到远期迈向全流程端到端 AI 编解码的务实演进路线。 ## Atmosphere & style 整场演讲呈现出典型硬核工程峰会的专业严谨与务实沉稳风格。Manindra Parhy 叙述条理分明、层层递进，从产业面临的物理算力瓶颈与标准制约切入，带有鲜明的行业问题意识。在画质实测对比环节，演讲通过特写对比直观展示算法对细节的挽救，语气从宏观分析转向令人信服的成果论证，整体观感理性客观且充满前瞻技术说服力。 ## Bulletin - [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) 随着摩尔定律终结与固定功能编解码器边际效益递减，AV2 中多数新增工具的压缩增益已不足 1%，且硬件厂商因向下兼容难以废除旧电路，导致芯片面积与成本呈不可持续的累积膨胀。 - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) NVIDIA 提出“可扩展 AI 扩展层”路线，在成熟传统编解码器底座上叠加 AI 增强层，仅需 5 到 50 TOPS 算力即可在当下移动设备（35-60 TOPS 预算）上运行，避免了端到端纯神经方案（>200 TOPS）的高算力门槛与块级混合方案的高通信延迟。 - [12:46](https://www.youtube.com/watch?v=GWY5cpzKYms&t=766s) AI 增强架构分为两种模式：极低开销的信令后处理模式（去伪影与超分辨率）与约增加 10% 码率的残差比特流模式；残差模式通过利用基础层运动矢量与潜空间上下文融合，可在同等码率下恢复大幅丢失的高频纹理与微观细节。 - [17:54](https://www.youtube.com/watch?v=GWY5cpzKYms&t=1074s) NVIDIA 规划了务实的神经视频压缩演进路线：预计未来 2 到 3 年内以“传统编解码底座 + AI 增强层”的混合架构实现规模化落地，长期则逐步过渡至运动与纹理全流程由神经网络驱动的端到端 AI 编解码器。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) 视频流量已占互联网数据流量的 95%，为了应对带宽与存储压力，业界从 H.261、H.264、HEVC、VVC 持续迭代至由 AOMedia 推动的免版税标准 AV1 与 AV2。 - [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) 传统编解码器存在极强的向下兼容粘性，诞生超 20 年的 H.264 仍需保留，导致芯片每代必须设计并累加专用 ASIC 电路，硅片面积成本急剧攀升。 - [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) 传统视频压缩技术挖掘已趋饱和，AV2 中大部分新增工具带来的压缩增益不足 1%，且长达约 6 年的标准制定周期严重拖慢了行业创新节奏。 - [05:00](https://www.youtube.com/watch?v=GWY5cpzKYms&t=300s) AI 编解码器能够直接复用智能手机、PC 和云端已普及的 NPU、GPU 等通用 AI 算力，无需在芯片中为每种新标准追加固定功能电路。 - [05:00](https://www.youtube.com/watch?v=GWY5cpzKYms&t=300s) 传统算法受限于微观像素块（小至 4×4）并追求最小化逐像素差值，而 AI 编解码器能在宏观语义层面提取结构化特征并保留高级纹理细节。 - [05:00](https://www.youtube.com/watch?v=GWY5cpzKYms&t=300s) 传统标准参数在冻结后无法更改，而 AI 神经网络不仅可利用数万条视频训练泛化，还能通过后续重新训练持续优化画质，并弹性调用芯片上更大的 GPU/NPU 算力。 - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) 编解码方案受限于消费级移动端“最低算力公分母”（35-60 TOPS），按端侧算力每 1.5 到 2 年翻番的趋势，2028 至 2029 年低端设备算力预计突破 100 TOPS。 - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) 端到端纯神经网络编解码要全面超越顶尖传统标准需消耗 200 TOPS 以上算力，短期内无法在移动端落地。 - [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) 块级嵌入式混合替换方案导致固定功能 ASIC 与 GPU/NPU 之间存在频繁的往返通信延迟，极难保证实时性且极耗带宽。 - [12:46](https://www.youtube.com/watch?v=GWY5cpzKYms&t=766s) 后处理模式属于环外处理，仅传输模型索引信令，码率开销极低，主要用于消除压缩伪影和超分辨率提升。 - [12:46](https://www.youtube.com/watch?v=GWY5cpzKYms&t=766s) 残差比特流模式虽引入约 10% 额外码率，但能在同等视觉质量（ISO bitrate）下精准恢复传统压缩过程中丢失的高频纹理细节。 - [14:28](https://www.youtube.com/watch?v=GWY5cpzKYms&t=868s) 残差架构直接复用基础层的运动矢量进行特征变形，避免了网络重复计算光流，并利用已解码视频特征推导熵编码先验分布以节省码率。 - [14:28](https://www.youtube.com/watch?v=GWY5cpzKYms&t=868s) 残差模式中当前帧生成的增强结果会反馈送入时序参考缓冲区，用于后续帧的运动变形预测。 - [15:47](https://www.youtube.com/watch?v=GWY5cpzKYms&t=947s) 在 1.3 Mbps 低带宽极限压力测试中，AI 增强扩展成功修复了 AV2 输出中的树叶涂抹模糊、胡须纹理丢失以及面部边缘锯齿破线伪影。 - [17:54](https://www.youtube.com/watch?v=GWY5cpzKYms&t=1074s) 行业中短期将走向“传统底座 + AI 增强比特流”的混合协同，全流程端到端 AI 编解码器则需等待端侧 AI 加速器算力进一步突破后成熟落地。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) 视频编解码器的历史演进与摩尔定律终结下的硬件瓶颈 演讲者开篇指出，今天将介绍一种全新的视频编解码器（video codec）方法：利用 AI 领域的创新，结合现有的传统视频标准，来满足当下的编解码需求。回顾过去三四十年的编解码器发展史，早在 1988 年国际电联（ITU-T）推出 H.261 开始，ITU-T 和 ISO/IEC 就联合制定了一系列 MPEG 标准。2003 年 H.264 的诞生是一个巨大的转折点，它直接促成了高清电视（HDTV）的普及以及视频在海量设备上的流媒体播放。自此之后，视频数据呈现爆发式增长。如今互联网 95% 的数据流量都是视频，为了跟上存储容量与网络带宽的巨大挑战，业界对压缩效率的要求越来越高。为此，业界相继推出了 HEVC（H.265）和 VVC（H.266）；与此同时，Google 在 2010 年开启了 VP8 和 VP9 系列编解码器的研发。随后，Google 与包括 NVIDIA 在内的多家行业领军企业联合成立了开放媒体联盟（Alliance for Open Media，简称 AOMedia），基于参与者的免版税政策打造了首个开放编解码器标准 AV1，而最新的 AV2 标准也在近期正式推出。 然而，这种编解码标准的飞速繁衍，给像 NVIDIA 这样的硬件芯片厂商带来了严峻挑战。随着摩尔定律的终结，传统的固定功能（fixed-function）硬件编解码器陷入了结构性的演进危机。首先是芯片硅片面积成本（silicon cost）的持续攀升：每推出一种新标准，硬件芯片就必须设计一套全新的专用硬件电路，且这些专用电路通常无法复用于其他视频标准。其次，编解码器具有极强的“粘性”（stickiness）——由于海量存量旧内容和老旧设备依然在网运行，硬件厂商根本不可能直接在芯片中废弃旧标准，必须持续向下兼容。直到今天，诞生超过 20 年的 H.264 依然被广泛保留和使用。新旧标准不断累加，导致芯片面积呈累积性膨胀，这在物理上和经济上都是不可持续的。 更为关键的是，传统视频编解码工具带来的边际收益正急剧递减。当前视频压缩技术对可能性的挖掘已趋于饱和，想要榨取额外的压缩增益变得极其艰难。在 AV2 中，绝大多数新增工具带来的增益甚至不足 1%，而要在硬件中高效实现这些微小的增益，其面积开销却极为高昂，致使每一代编解码硬件都愈发臃肿复杂。与此同时，人眼可感知的画质提升也在递减。此外，固定功能编解码硬件的研发周期极为漫长，耗费巨大的人力与时间成本，一旦流片量产，必须在市场上销售足够长的时间才能收回投资（ROI）。硬件较长的生命周期反过来拖慢了标准制定的节奏，导致大约每隔 6 年才会推出一个新标准，极大地抑制了行业创新。 - 互联网中 95% 的数据流量为视频流。 - H.264 标准制定于 2003 年，距今已超过 20 年，但仍必须集成在当前主流芯片中。 - AV2 中多数工具所能带来的压缩效率增益已小于 1%，导致硬件实现的芯片面积利用率极低。 - 硬件投资回报压力倒逼标准化周期长达约 6 年，严重阻碍技术迭代速度。 > “What we see is that this kind of plethora growth of codecs is not sustainable in silicon especially with Moore's law ending and it creates a structural challenges to the fixed function codec evolution.” —— Manindra Parhy --- ### [05:00](https://www.youtube.com/watch?v=GWY5cpzKYms&t=300s) 画质提升饱和现状与引入 AI 编解码器的双重优势 为了直观展示画质提升逐渐饱和的现象，演讲者展示了一组对比样本：同一段人脸视频分别采用 VP9、AV1 和 AV2 编码。最左侧的 VP9 画面中，人物面部出现大面积的马赛克块效应（blockiness）和色斑，伪影十分严重；过渡到 AV1 后，大部分伪影被有效清理，画质显著改善；而从 AV1 跨越到 AV2，虽然在前额肤色还原与纹理细节上有明显保留，确实存在画质提升，但对比“VP9 到 AV1”的画质跃升幅度，“AV1 到 AV2”的相对增益已经明显变小。 来自 NVIDIA 的 Manindra Parhy 随后正式介绍了自己及其团队的研究方向。面对硬件演进的结构性瓶颈，NVIDIA 给出的破局方案是引入 **AI 编解码器（AI Codec）**。AI 编解码器具备两大核心基石优势： 1. **复用无处不在的通用 AI 算力**：当今各大计算平台（从云端数据中心、PC、笔记本电脑到智能手机）中，AI 加速器已经普及化。无论是 NPU、GPU 还是 TPU，都可以直接运行 AI 编解码网络。以智能手机为例，用户设备上的 NPU 原本用于照片消除、降噪或生成 Genmoji 表情包，现在完全可以借用这部分算力进行视频压缩，从而无需在芯片中额外增加专用的固定功能编解码电路。 2. **更高维度的感知画质与宏观特征保留**：传统编解码器基于微观像素块（小至 4×4 像素）进行划分，其算法工具的核心目标是最小化输入视频与压缩视频之间的逐像素绝对差值。而 AI 编解码器能够跳出微观块的局限，在宏观层面上理解和提取结构化语义特征（structural features），并主动保留对人眼沉浸感最为关键的高级纹理细节。 3. **强大的泛化适应能力与算力缩放性**：传统标准的工具和滤波器系数是由研究人员从极为有限的测试视频集中手动挑选并固化进规范中的，一旦标准冻结（例如 H.264 的滤波器参数），在整个生命周期内就永远无法更改。而 AI 神经网络可以利用数以万计、覆盖各种复杂场景的视频样本进行充分训练，并在未来通过重新训练不断自我进化；此外，当前 SoC 芯片上分配给 GPU/NPU 的硅片面积远远大于固定功能音视频处理单元（NVENC/NVDEC），AI 编解码器能够调用规模大得多的计算资源，从而在不导致芯片面积每代线性膨胀的前提下，换取显著更优的压缩率与画质表现。 > “The AI codecs, they can learn the parameters, they can be trained later and do a better quality over time, so it's flexible that way, and also the network can be trained with thousands and thousands of video encompassing very different kind of content, so it's much more robust in different scenarios.” —— Manindra Parhy --- ### [09:10](https://www.youtube.com/watch?v=GWY5cpzKYms&t=550s) 端侧 AI 算力现状与三种 AI 编解码路径对比 为了让 AI 编解码器落地，必须首先评估现有终端消费级设备的 AI 算力预算。Manindra Parhy 列举了当前消费级设备所能提供的 INT8 AI TOPS（万亿次操作/秒）数据： | 设备类型 | 典型 INT8 AI 算力 (TOPS) | 算力特点与定位 | |:--- |:--- |:--- | | **智能手机 (Smartphones)** | 35 – 60 TOPS | 消费生态中最基础的最低算力公分母 | | **集成 GPU 笔记本/PC (Laptops/Notebooks w/ iGPU)** | ~120 TOPS | 主流移动办公设备的计算基础 | | **搭载独立 GPU 的 PC (PC w/ Discrete GPU)** | 230 – 1660 TOPS | 涵盖中高端消费级桌面设备（不含工作站/数据中心） | 演讲者强调，由于同一段视频必须能够在所有终端设备上正常播放解码，方案设计必须严格受制于“最低公分母”（即手机端 35~60 TOPS）的算力瓶颈。观察产业趋势，设备端的 AI TOPS 大约每 1.5 到 2 年翻一番；以此推算，到 2028 至 2029 年，哪怕是手机等低端设备也将具备超过 100 TOPS 的算力，这为 AI 视频编解码划定了明确的算力预算红线。 随后，Manindra Parhy 梳理了当前业内探索 AI 编解码的三种根本技术路线： 1. **端到端纯神经压缩（Fully End-to-End AI Codec）**：整套编解码流程完全依赖神经网络构建，仅针对 AI 加速器运行。虽然在较低算力下也能勉强完成压缩，但若要全面击败并超越当今最顶尖的传统视频标准（如 AV1/AV2），其算力需求预估将远超 200 TOPS，在目前的移动设备上完全无法实用化。 2. **传统工具级混合替换（Block-based Hybrid AI Codec）**：在传统编解码器框架内部，将微观块预测或滤波等具体工具替换为 AI 模型。这种做法在硬件协作上存在严重的性能瓶颈（Poor Exploitability）：当固定功能专用集成电路（ASIC）解码完某个像素块后，必须将其传输至 GPU 执行神经网络计算，然后再把结果回传给 ASIC 处理下一个块。高频的往返通信延迟（roundtrip latency）使得这种方案极难实现实时编解码，且极度消耗带宽；如果直接将 AI 工具全部固化成全固定功能 ASIC，又会导致芯片面积极度膨胀。 3. **可扩展 AI 扩展层（Scalable AI Extension）**：这是 NVIDIA 提出的兼顾架构与实际落地的第三种路线。该方案依然保留成熟的传统编解码器作为基础底图（Base Layer），并在其之上叠加一个运行于 AI 加速器上的扩展增强层（Extension Layer）。实验表明，该扩展层仅需 **5 到 50 TOPS** 的轻量级算力即可运行，完全能够适配当下的移动平台，既继承了过去 30 年传统编解码器的工程沉淀与硬件兼容性，又完美搭上了 AI 算力指数级增长的快车。 ```mermaid flowchart TD accTitle: Comparison of Three AI Codec Architecture Approaches accDescr: Diagram contrasting End-to-End Neural Compression, Block-level Hybrid Codec, and Scalable AI Extension. subgraph E2E["路线一：端到端纯神经压缩 (End-to-End AI)"] A1["纯神经网络压缩架构"] --> B1["算力需求 >200 TOPS"] B1 --> C1["超越现代标准极难在移动端落地"] end subgraph Hybrid["路线二：块级嵌入式混合替换 (Block Hybrid)"] A2["传统固定 ASIC 块解码"] <-->|"高往返通信延迟"| B2["GPU/NPU 神经网络推理"] B2 --> C2["无法保证实时性 / ASIC 面积爆炸"] end subgraph Scalable["路线三：可扩展 AI 扩展层 (Scalable AI Extension)"] A3["传统硬件编解码基础层 (Base Layer)"] B3["AI 算力扩展层 (Extension Layer)"] A3 --> C3["5~50 TOPS 超轻量算力"] B3 --> C3 C3 --> D3["通吃手机/移动端，兼顾成熟生态与超高画质"] end ``` > “What we are proposing is the third approach of a scalable AI extension where the traditional codecs will be still used, but we enhance this traditional video with an extension layer which is running on AI accelerators, and from our work, we see that it's possible to do this within 5 to 50 TOPS which means that it we can run on mobile platform.” —— Manindra Parhy ### [12:46](https://www.youtube.com/watch?v=GWY5cpzKYms&t=766s) 两种 AI 增强流模式：后处理模式与残差比特流模式 在标准编解码管线中，原始码流经传输或存储后送入解码器输出视频。NVIDIA 提出的增强架构在此基础上引入了神经网络：将重构视频与原始输入视频一同输入神经网络，生成独立的“增强比特流”（enhancement bitstream），并在解码端解码生成高质量的增强视频。Manindra Parhy 明确指出，基于这一思路可以衍生出两种根本不同的技术路线： 第一种是**后处理模式（post-processing mode）**，属于环外（out-of-loop）处理。该模式的码流仅负责向解码端传递信令（signaling），指示针对特定视频区域应当调用哪一个模型索引（model index）。由于仅传输指示信令，其码率开销非常微小。后处理模式的核心目标是消除视频压缩失真（artifact removal），并通过超分辨率（super-resolution）提升像素密度。 第二种则是**残差比特流模式（residual bitstream mode）**。与单纯传递信令不同，该模式在码流中实际传输残差比特。这种方式旨在恢复传统压缩过程中丢失的高频纹理与微观细节。尽管残差码流会引入约 10% 的额外码率开销，但带来的画质提升幅度极为显著。正如 Manindra Parhy 所强调的：“But if you compare that 10% increase to the quality gain that you are getting overall you get a much better compression at ISO bitrate.” - **后处理模式特点**：信令开销极低、专注于去除压缩伪影、提升像素分辨率。 - **残差比特流模式特点**：携带约 10% 的残差码率开销、精准还原被丢弃的纹理与细节、在同等视觉质量（ISO bitrate）下提供更优的综合压缩效率。 --- ### [14:28](https://www.youtube.com/watch?v=GWY5cpzKYms&t=868s) 残差模式技术架构与潜空间特征融合 Manindra Parhy 进一步深入解析了残差模式的底层技术架构。系统分为上下两条主要处理管线：顶层是传统编解码链路，底层则是 AI 残差提取与增强链路。 在残差模式下，系统从基础层（base layer）提取运动矢量（motion vectors），并将其施加在前一帧已解码的视频上，对增强帧进行运动补偿变形（warping）。接着，基础层解码视频的潜变量（latents）与运动补偿后的特征潜变量进行上下文融合（context fusion）。融合生成的潜变量经过量化（quantization）后，利用熵编码（entropy coding）生成最终的增强比特流。为了完成高效的熵编码，系统直接从已解码视频中提取潜变量概率分布作为先验知识（prior），确保解码端具备一致的先验信息。在解码端，利用该先验重建潜变量并叠加回视频流，生成最终展示给用户的增强视频，同时该帧视频也会被反馈送入管线，作为下一帧预测的参考。 ```mermaid flowchart TD accTitle: 残差模式增强编解码架构 accDescr: 展示基础层传统编解码与AI残差层在编码端与解码端的运动补偿、上下文融合及熵编码流程。 subgraph BaseLayer["传统编解码基础层"] Src["原始视频 Source"] --> Enc["基础编码器"] Enc --> Dec["基础解码器"] Dec --> RecBase["已解码基础帧"] end subgraph MotionWarp["运动补偿与变形"] MV["提取运动矢量 (MV)"] PrevRef["上一帧增强视频参考"] MV --> Warp["特征变形处理 (Warping)"] PrevRef --> Warp end subgraph ResidualCore["残差潜空间融合与编码"] RecBase --> Fusion["上下文融合 (Context Fusion)"] Warp --> Fusion Fusion --> Quant["潜变量量化 (Quantization)"] RecBase -.->|"先验概率 (Prior)"| Entropy["熵编码 (Entropy Coding)"] Quant --> Entropy Entropy --> EnhStream["增强比特流"] end subgraph OutputStage["增强视频生成与回环"] EnhStream --> Recon["潜变量重建与叠加"] RecBase --> Recon Recon --> FinalOut["最终增强视频输出"] FinalOut -.->|"反馈下一帧"| PrevRef end ``` - **运动补偿特征融合**：重用传统编解码层提取的运动矢量，避免神经网络重复计算稠密光流。 - **自适应先验引导**：熵编码所需的概率分布完全由已解码视频特征推导，无需在码流中传输额外的分布描述开销。 - **帧间时序依赖**：当前帧生成的增强结果会存入缓冲区，用于后续帧的运动变形与时序参考。 --- ### [15:47](https://www.youtube.com/watch?v=GWY5cpzKYms&t=947s) AV2 与 AI 扩展增强效果对比 为了直观展现技术潜力，Manindra Parhy 展示了多组 AV2 原始视频与结合 AI 增强扩展（AV2 + AI extension）后的画质对比演示。测试码率设定在约 1.3 Mbps 的低带宽场景，以此检验技术在极端压力下的极限质量增益（max quality delta）。 在第一组树木外景对比中，画面中央带有滑动对比条，左侧为纯 AV2 解码输出，右侧为 AI 增强结果。左侧 AV2 画面中树叶区域细节严重模糊退化，并伴随大量块效应和失真；而右侧 AI 扩展模型准确重建出纷繁细密的树叶结构，观感高度还原原始未压缩视频。 第二组演示为人物特写镜头。由于码率过低，左侧纯 AV2 处理后的下巴胡须纹理被完全抹平（“the beard is completely gone”）。Manindra Parhy 针对此现象深度剖析了网络的工作机制：神经网络在特征层面自发学习并感知到底层信息中缺失了这部分高频纹理，随即利用残差比特流中的信息进行定向弥补，从而完整重构出了清晰根根分明的胡须细节。 在第三组舞台人物与乐器演示中： - 画面右侧人物手部边缘在 AV2 下呈现锯齿断线与涂抹伪影（broken lines and smear），经 AI 处理后轮廓线条变得清晰平滑。 - 画面中的打击乐器（鼓）在增强后表面质感更加纯净利落。 - 画面中央的人物面部在左侧呈现明显的过饱和与涂抹感（washed out and smeared），右侧增强画面则成功恢复了自然的皮肤质感与立体纹理。 --- ### [17:54](https://www.youtube.com/watch?v=GWY5cpzKYms&t=1074s) 务实演进路径：从混合增强迈向端到端 AI 编解码 在演讲结尾，Manindra Parhy 总结了 NVIDIA 针对神经视频压缩所规划的务实技术演进路线（pragmatic path forward）。当前行业通常将超分辨率与伪影去除视为独立于编解码器之外的播放端插件；而 NVIDIA 倡导将这一能力深度整合进编解码生态中。 ```mermaid flowchart LR accTitle: 神经视频编解码演进路线图 accDescr: 从当前的传统编解码外挂滤镜，跨越到两至三年的混合残差增强层，最终在长期实现全流程端到端AI编解码。 Stage1["现状：外挂后处理<br>· 独立超分与降噪<br>· 缺乏码流级深度协同"] Stage2["中期（2-3年）：混合架构<br>· 传统编解码底座 (如 AV2)<br>· AI 增强比特流 / 硬件加速"] Stage3["远期路线：端到端 AI Codec<br>· 运动/纹理全流程端到端学习<br>· 算力成熟支撑端侧硬件运行"] Stage1 --> Stage2 Stage2 --> Stage3 ``` Manindra Parhy 阐述了分阶段迈向学习型压缩的战略构想：“So what we are doing here is proposing a pragmatic path forward towards the learned video compression.” 他预测中期方案（传统编解码底座 + AI 增强比特流 + 硬件加速器后处理）将在未来 2 到 3 年内获得产业界的广泛采纳与规模化落地。而对于终极的端到端神经编解码器（end-to-end AI codec）——即运动估计、纹理表征及熵模型全流程均由深度网络驱动的架构——由于其计算复杂度极高，需等待端侧 AI 加速器算力进一步突破。待硬件具备充沛算力后，行业将全面进入全流程端到端 AI 视频压缩时代。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=GWY5cpzKYms&t=5s) This kind of plethora growth of codecs is not sustainable in silicon especially with Moore's law ending and it creates a structural challenges to the fixed function codec evolution. — Manindra Parhy ## Mentioned NVIDIA, Manindra Parhy, AOMedia, Google, ITU-T, ISO/IEC, H.261, H.264, HEVC, VVC, VP8, VP9, AV1, AV2 [Source](https://www.youtube.com/watch?v=GWY5cpzKYms)

Source quotes

[0:05] Manindra Parhy: This kind of plethora growth of codecs is not sustainable in silicon especially with Moore's law ending and it creates a structural challenges to the fixed function codec evolution.

Source quotes

[0:05] Manindra Parhy: What we see is that this kind of plethora growth of codecs is not sustainable in silicon especially with Moore's law ending and it creates a structural challenges to the fixed function codec evolution.

Original transcript

[0:05] Hi everyone. Today I'm going to talk [0:07] about a novel video codec approach which [0:11] uses the innovation in AI and using the [0:15] traditional video standards of today to [0:18] meet the codec demands. [0:21] Before I go into the details, let me [0:23] start with a brief history of the video [0:25] codecs. standards have been there for [0:28] last three four decades started with [0:30] H.261 in 1988 and since then I2T and ISO [0:36] have created a lot of MEX series of [0:38] standards. The inflection point came [0:41] with H.264 264 in 2003 which enabled [0:47] streaming to a wide range of devices and [0:50] the highdeinition TV and since then [0:53] there is a increasing growth in video [0:57] today 95% of the internet data is video [1:01] and to catch up with the storage and the [1:04] network bandwidth requirement [1:06] what is needed is better and better [1:08] compression efficiency and there have [1:11] been a series of standards to to cater [1:14] to that. So there was HVC and VVC and in [1:19] the parallel Google started in 2010 a [1:22] VP8 and VP9 series of codecs and after [1:26] that [1:28] Google and other industry companies [1:31] including Nvidia formed an alliance [1:33] called alliance for open media AOM media [1:37] to create the first open codec in AV1 [1:41] which was based on a license free [1:45] royalty policy by the participants and [1:50] lately like this year only another [1:53] standard of AV2 which is the latest [1:56] standard just came and what it means for [2:00] a hardware company like Nvidia. [2:03] What we see is that this kind of [2:06] plethora growth of codeex is not [2:10] sustainable in silicon especially with [2:14] M's law ending and it creates a [2:18] structural challenges [2:20] to the fixed function codec evolution. [2:23] Let me explain in a little bit more [2:25] details. So the first one is the growing [2:29] silicon cost. Each new codec requires [2:32] new hardware and this hardware mostly [2:36] cannot be used by other video standards. [2:39] And also another fact is that the codecs [2:41] are sticky is because we just can't [2:44] throw older standards because somebody [2:47] may have legacy content or some some [2:50] devices may have those codecs. So we [2:52] have to make sure that we are able to [2:54] decode or encode to those devices. And [2:58] it's a fact that we still are sticking [3:00] around with H.264 which is more than 20 [3:03] year old standard. So the codecs are [3:06] there we can't take it away and they are [3:08] basically more and more new and newer [3:11] standards coming. So this is creating a [3:13] cumulative area growth which is [3:15] obviously not sustainable [3:18] for us and also the standards are giving [3:22] diminishing return in terms of codec [3:25] complexity because the technology is [3:28] kind of saturating in terms of exquisite [3:31] most of the possibilities of [3:33] compression. So it's becoming lot lot [3:35] harder to get for the gain and the [3:38] standard tools in any codec they give [3:41] less and less gain. In fact in AV2 most [3:45] of the codecs give less than 1% gain and [3:50] this tiny amount of gain is very [3:53] difficult to realize in hardware in area [3:56] efficient way. So the codecs are [3:58] becoming every generation is becoming [4:00] more and more complex. And another thing [4:02] is that there is a the visual quality [4:06] itself is also the the visual [4:08] improvement are diminishing with every [4:11] successive codec because the codecs are [4:13] becoming so good now. [4:15] And then lastly [4:18] it's basically a long development cycle [4:21] to implement a codec into the hardware. [4:24] It takes a lot of time and effort and [4:27] once we have in that in the silicon we [4:30] can't just move to another codec. We [4:32] need to get the return on investment for [4:34] that. So the we need to have it in the [4:36] product for some time. So this puts a [4:38] pressure on the standardization cycle [4:40] and slows down the innovation. So as you [4:43] saw in my first slide it's roughly [4:46] around 6 years when a new standard comes [4:48] in. So it's stunting innovation. And so [4:52] what we need here is a more flexible [4:56] codec approach. [5:00] And also to illustrate my point on the [5:02] visual quality getting saturated, here [5:05] is one example where we have the same [5:08] video being encoded with BP9, AV1 and [5:10] AV2. And the leftmost is VP9. And as you [5:15] can see like there is lot of blockiness [5:18] on the person's face, lot of color [5:20] patches and all. And it looks very very [5:22] artifacty. Now with AV1, most of these [5:25] artifacts are cleaned up. Looks much [5:28] much good. And going from AV1 to AV2, [5:32] obviously there is also a big amount of [5:34] improvement. Like if you look at the [5:35] forehead, the skin tone is preserved in [5:38] AV2 compared to AV1. So there is an [5:41] improvement. But if you compare VP9 to [5:43] AV1 quality delta versus AV1 to AV2 [5:45] quality delta, the amount of quality [5:47] data we got from VP9 is much smaller. [5:50] when you go to AV2. [5:53] My name is Manindra Parari. I work at [5:55] Nvidia and we looked into this issue and [6:00] I'm working on how to solve this as a [6:03] hardware company and the answer is an AI [6:07] codec and an AI codec actually it has [6:10] two fundamental advantage. [6:12] The first one is that we can leverage [6:15] the existing AI compute on the various [6:17] platforms. So these days the ax letters [6:20] are getting ubitus. So be it data [6:23] center, be it PC, laptop, notebooks or [6:27] smartphones. So there is some kind of a [6:29] letter in the form of NPU, GPU or TPU [6:33] and we can use this for AI codecs and [6:38] like if you look at smartphones, people [6:39] use it for all kind of things. The NPUs [6:42] they use it for cleaning up the images [6:45] and all that create creating genoji and [6:49] we can use the same NPU for for video [6:52] compression. So that way we are not [6:55] adding any extra dedicated hardware to [6:57] it. And there is another advantage to [7:00] using AI codeex that's the higher [7:03] quality compared to a traditional codec. [7:06] And why is that? Because the traditional [7:08] quirks they work on blocks and these [7:11] blocks can be very small like a 4x4 [7:13] pixel. And the way the standard tools [7:16] are designed is to minimize the pixel [7:18] differences between the input video and [7:21] the compressed video and they look into [7:24] micro granularity like blocks. Whereas [7:26] an AI codec can look into structures [7:30] features at a macro level. they can [7:32] differentiate that and they can preserve [7:35] the structural details and all that [7:37] which is much more important for immers [7:39] quality point of view. And the other [7:41] thing is that when the standards are [7:43] created these are created by researchers [7:46] who handpicked these tools and they have [7:50] they do this from a limited set of video [7:53] and whatever parameters are baked into [7:56] the standard they are going going never [7:58] going to change like whatever is there [8:00] in H.264 H264 when the product is [8:02] standardized whatever filter coefficient [8:04] it it remains all the time H.264 H264 is [8:07] there whereas the AI networks the AI [8:11] codeex they can learn the parameters [8:14] they can be trained later and do a [8:17] better quality over time so it's [8:19] flexible that way and also the network [8:22] can be trained with thousands and [8:25] thousands of video and compassing very [8:27] different kind of content so it's much [8:29] more robust in different scenarios [8:33] and lastly the the amount amount of area [8:36] dedicated to NPUs, GPUs in a platform, [8:40] it's much bigger than the fixed function [8:43] video encoders and decoder. That means [8:46] the AI codec can access to lot of more [8:49] compute than in a fixed fixed function [8:52] AC and using those lot more compute it [8:56] can do a better job in terms of quality. [8:59] So essentially we can get better [9:01] compression whereas not getting a [9:05] proportional go growth in area every [9:07] time we have a new codec. [9:10] So to get this AI codec we need to first [9:13] look into what kind of AI accelator [9:16] capacities are there. And here I'm [9:19] showing [9:20] the int8 AI tops available on many of [9:24] the consumer devices of today. So [9:27] smartphone there is 35 to 60 tops. Then [9:31] you have the laptops, notebooks and all [9:34] with the integrated GPU they are around [9:36] 120 tops. Now a discrete GPU can be [9:39] added to a PC device and they have much [9:42] bigger range like 230 to 1660. I'm [9:46] talking about here only the consumer [9:47] devices not the the professional [9:49] workstation or data center. And so we [9:53] need to make sure the that we have the [9:55] the capability to run the video on the [9:59] lowest denominator which is like phones [10:01] and all because the same video has to [10:03] run on all sort of devices. Another [10:06] trend we see is that the AI tops is [10:09] basically doubling every 1.5 to 2 years [10:12] and if you continue this projection by [10:14] 28 29 we expect more than 100 tops on [10:18] the low-end devices like phone and that [10:21] puts the budget that we have for the AI [10:24] compute [10:26] and there are three fundamental [10:29] different ways the AI codec can happen [10:32] based on the work done in the industry. [10:35] The first one is the fully end to end AI [10:38] codec based on completely neural [10:41] compression and these are targeted only [10:43] at AI excelators. the amount of tops [10:46] needed is more than 200 tops and even [10:50] with less number of tops you can always [10:52] have the video compression happening but [10:55] here I'm talking about improving further [10:58] than the existing latest video standards [11:02] to to basically beat those and to make [11:05] better than the existing standards when [11:07] we estimate it to be much more than 200 [11:10] tops which are impractical to run on a [11:12] mobile platform. Now the the second [11:15] approach is that we take the traditional [11:16] codec and the blockbased tools and they [11:19] are embed the AI tools. So the problem [11:22] with that approach is the poor expl. So [11:25] imagine you have a block of pixels being [11:28] decoded on a on a fixed function as now [11:32] this has to go to a GPU and then come [11:34] back. So the amount of roundtrip latency [11:37] is very very high and we can't create a [11:41] real-time codec with with this kind of [11:43] approach and the performance will be [11:45] really bad. So we can put this into [11:48] fully fixed function ASIC but the ASIC [11:52] area will be pretty pretty big. And what [11:56] we are proposing is the third approach [11:58] of a scalable AI extension where the [12:01] traditional codeex will be still used [12:03] but we enhance this traditional video [12:07] with an extension layer which is running [12:09] on AI accelerators and from our work we [12:12] see that it's possible to do this within [12:14] 5 to 50 tops which means that it we can [12:17] run on mobile platform and this is the [12:19] approach we think is the the scalable [12:22] path forward which combines the 30 years [12:25] of codec innovations at the same time [12:28] the growth of AI AI accelerators. [12:33] Let me talk about a little bit at a high [12:34] level how this scalable AI extension [12:37] look like and here is the encoder and [12:39] decoder view. So on the left side there [12:41] is a encoder which takes a input video [12:44] and generates a bitstream and this [12:46] bitstream gets transmitted or stored and [12:48] then when it is played back it goes to a [12:50] decoder which outputs a video and [12:53] additionally what we're doing is that [12:55] here the reconstructed video and the [12:57] input video goes to a neural network [12:59] which will create an enhancement [13:01] bitstream and then this enhancement [13:03] bitstream will be decoded on a a [13:07] we basically create the enhanced video [13:10] and with this there are fundamentally [13:12] two different approaches can happen. So [13:14] the first one is the post-processing [13:15] mode where there's out of loop [13:18] post-processing of the video. But what [13:21] is being done here is the bitstream will [13:23] tell which model index to be used and [13:27] which part of the video will use what [13:29] kind of video and basically this is [13:31] signaling to the decoder side and [13:33] because of the signaling the signaling [13:36] overhead is not very high. So the bit [13:38] rate overhead is not much. [13:41] The other approach is the residual [13:42] bitstream mode approach. And here [13:45] actually we are putting some residual [13:47] bitstream. And if you look at the [13:49] post-processing versus residual [13:50] bitstream, we expect the post-processing [13:52] mode to do remove the artifacts in the [13:54] video and then improve the pixel density [13:58] to do a super resolution. Whereas in the [14:01] residual bitstream mode, we expect the [14:03] textures and details that were lost [14:06] during the compression to come back. And [14:09] since there is a residual bit stream [14:10] involved then there is a small amount of [14:13] bit rate overhead maybe around 10%. But [14:17] if you compare that 10% increase to the [14:19] quality gain that you are getting [14:22] overall you get a much better [14:24] compression at ISOB treat. [14:28] So let me explain go to a little bit [14:32] deeper into this especially the residual [14:35] mode. So we have the source video and it [14:38] is being encoded and decoded on the top [14:41] line which I like explained in the [14:43] previous slide. And here from the base [14:45] layer we are extracting additional [14:47] information. For example the motion [14:50] vector and the motion vector is applied [14:53] to the previously decoded video does a [14:56] warping on the enhancement video. Now [14:58] this decoded video from the base layer [15:01] and the motion compensated video the [15:04] latence of this are fused together. So [15:07] create the context and now after the [15:09] context fusion so this latence they go [15:12] through a quantization process and after [15:14] that we do the entropy coding to [15:16] generate the enhancement bit stream. Now [15:18] to do this entropy coding we need a [15:21] prior. So basically the the probability [15:24] of the latence that's actually coming [15:27] from the decoded video already decoded [15:29] video so that the decoder knows what [15:32] prior to use and using that the [15:36] reconstructed latence and then added [15:38] back and creates the enhancement video [15:41] which goes which is output to display [15:44] and also goes back for the next frame. [15:47] So next I'm going to show some visual [15:51] examples of AV2 versus AV2 plus AI [15:55] extension how the quality looks like. [15:58] All right. So in this example I am [16:01] showing a visual comparison of AV2 and [16:05] AV2 video being enhanced with AI [16:08] extension. The bit rate is around 1.3 [16:11] Mbps. This is kind of low bit rate but I [16:14] wanted to show the max potential of this [16:17] technology the max quality delta we can [16:19] get obviously at higher bit rate you [16:22] will see also visual delta and as you [16:25] can see here on the there's a moving [16:27] slider on the left side is AV2 right [16:29] side is enhanced and as the slider is [16:31] moving you can see that the the tree [16:34] area if you focus on that on the AV2 is [16:37] basically lot of details are lost and a [16:41] lot artifacts are there but on the right [16:43] side it is recreating those leaves and [16:47] all and it's looking more like more like [16:49] the tree as in the original video. Here [16:53] is one more example where you can see on [16:56] the left side the beard is completely [16:59] gone but the right side is the AI [17:03] enhancement is reconstructing those [17:05] beards. The network learns that feature- [17:07] wise this part is missing on its own and [17:11] then it puts the residual bits there so [17:13] that you can see those beer is coming [17:16] back. [17:17] Here is a third video again on the left [17:20] enhancement on the right and as you can [17:23] see the hand on the person on the right. [17:27] So this is like like lot of broken lines [17:31] and the edges are not very clear smear [17:33] and all and that's basically cleaned on [17:36] the right side and even if you look at [17:39] the drum the drum is looking lot lot [17:42] cleaner and the person on the center the [17:46] face on the left is very smeared is [17:49] washed out the right side tries to bring [17:51] back the texture. [17:54] Now you have seen the videos. So let me [17:57] summarize. So what we are doing here is [18:00] proposing a pragmatic path forward [18:03] towards the learned video compression. [18:06] So as of today people use artifact [18:10] reduction super resolution not as part [18:13] of the codec and what we are proposing [18:17] is to use the traditional codec and also [18:22] add an enhancement layer in bitstream [18:26] and which will be best processed in ax [18:29] later to improve the quality and in the [18:31] mid in the middle term and we expect [18:34] this to be [18:36] basically getting industry traction in 2 [18:39] to 3 years time and eventually [18:43] in the long term it will go to end to [18:45] end AI codec where every representation [18:47] in terms of motion texture the full [18:50] pipeline will be learned obviously this [18:53] will take lot more complexity [18:56] and by the time the accelator becomes [18:59] powerful enough so this can be run on a [19:02] excelator full end to end codec but that [19:04] that's the direction that we are [19:05] heading. This concludes my talk. Thank [19:08] you very much for listening.