Microsoft Brings Neural Video Codec to Teams, Slashing Bitrate 60%

@Scale

By transmitting explicit scale indices to eliminate cross-platform arithmetic errors, Microsoft enabled real-time neural video compression on consumer NPUs that outperforms H.265 at a fraction of the bandwidth.

AI-assisted summary of ML Video Codecs: From Research Breakthroughs to Product-Scale Impact | Nabakumar Khongbantabam — original source. Check the original for full context.

Source published: 2026-10-30

Key ideas

Microsoft is productizing a Machine Learning Video Codec (MLVC) directly inside Microsoft Teams to replace traditional silicon-bound video codecs. (0:05)
By executing on consumer laptop NPUs rather than waiting for dedicated fixed-function hardware, MLVC slashes the conventional 8-to-12-year standard-to-hardware adoption cycle to software update speeds. (3:07)
In production testing, MLVC achieves comparable or superior perceptual visual quality with up to a 60% to 65% bitrate reduction compared to H.265. (12:19)
To prevent decoding crashes across heterogeneous NPU hardware architectures, Microsoft engineered around floating-point bit-exactness by transmitting scale indices directly over the network. (9:17)
The entire MLVC stack—including model checkpoints, the VCD2 benchmark dataset, and a C++ integration library—has been open-sourced ahead of a planned public preview in early 2027. (16:11)

Detailed notes

## Background This presentation was published on YouTube by the @Scale channel on October 30, 2026. Featuring Nabakumar Khongbantabam from Microsoft, the session details the technical journey of translating research-grade machine learning video codecs into a production-scale deployment inside Microsoft Teams.

## TL;DR Conventional video codec standards require an 8- to 12-year cycle from specification to hardware ubiquity, prompting Microsoft to develop a software-updatable ML Video Codec (MLVC) running directly on consumer Neural Processing Units (NPUs). Bridging research models like DCVC-RT to production required solving severe cross-platform bit-exactness failures and temporal drift across heterogeneous silicon from Apple, Intel, and Qualcomm. By transmitting explicit scale indices and pairing 540p neural encoding with client-side super-resolution, the team achieved a 60% to 65% bitrate reduction over H.265 at sub-16 millisecond frame latencies. Microsoft is validating the codec in internal Teams test rings ahead of a 2027 Copilot+ PC rollout, while releasing the models, VCD2 benchmark dataset, and C++ runtime library to the open-source community.

## Atmosphere & style The presentation carries the polished, authoritative rhythm of a pragmatic engineering post-mortem mixed with a high-stakes product roadmap reveal. Khongbantabam speaks with calm, measured technical confidence, focusing candidly on real-world engineering compromises—such as accepting bitrate overhead to overcome floating-point calculation mismatches across vendor silicon—rather than abstract research ideals. Side-by-side artifact comparisons and concrete performance metrics give the talk an engaging, problem-solving momentum that illustrates what it genuinely takes to commercialize deep learning breakthroughs.

## Bulletin - [00:05](https://www.youtube.com/watch?v=oD5z32RR4mk&t=5s) Microsoft is productizing a Machine Learning Video Codec (MLVC) directly inside Microsoft Teams to replace traditional silicon-bound video codecs. - [03:07](https://www.youtube.com/watch?v=oD5z32RR4mk&t=187s) By executing on consumer laptop NPUs rather than waiting for dedicated fixed-function hardware, MLVC slashes the conventional 8-to-12-year standard-to-hardware adoption cycle to software update speeds. - [12:19](https://www.youtube.com/watch?v=oD5z32RR4mk&t=739s) In production testing, MLVC achieves comparable or superior perceptual visual quality with up to a 60% to 65% bitrate reduction compared to H.265. - [09:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=557s) To prevent decoding crashes across heterogeneous NPU hardware architectures, Microsoft engineered around floating-point bit-exactness by transmitting scale indices directly over the network. - [16:11](https://www.youtube.com/watch?v=oD5z32RR4mk&t=971s) The entire MLVC stack—including model checkpoints, the VCD2 benchmark dataset, and a C++ integration library—has been open-sourced ahead of a planned public preview in early 2027.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=oD5z32RR4mk&t=5s) Conventional video codec standards (such as H.264, H.265, and H.266) take 8 to 12 years from specification to widespread commercial adoption due to standardization, silicon manufacturing, and hardware refresh cycles. - [03:07](https://www.youtube.com/watch?v=oD5z32RR4mk&t=187s) Neural video codecs running on client NPUs decouple video compression advances from fixed-function silicon hardware cycles, enabling rapid deployment via standard application updates. - [03:07](https://www.youtube.com/watch?v=oD5z32RR4mk&t=187s) MLVC delivers operational backward compatibility across diverse existing consumer hardware, running on Apple Silicon dating back to M1 as well as newer Copilot+ PCs. - [06:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=377s) Deploying academic models like DCVC-RT to consumer devices surfaced two major bottlenecks: severe real-time compute limits on laptop NPUs and the near-impossibility of achieving cross-vendor bit-exactness. - [06:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=377s) Divergent floating-point arithmetic across different vendor NPUs breaks entropy model synchronization, causing immediate decode failures when streaming across heterogeneous platforms like Apple M3 and Intel Lunar Lake. - [09:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=557s) Transmitting scale indices explicitly across the network resolves cross-platform bit-exactness failures at the cost of minor bitrate overhead. - [09:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=557s) Temporal decoder drift caused by compounding rounding errors in recurrent layers is mitigated using Long-Term Reference frames, periodic intra-frame resets, and standardized operators like ReLU. - [12:19](https://www.youtube.com/watch?v=oD5z32RR4mk&t=739s) Direct engineering partnerships with silicon vendors enabled real-time execution achieving at least 30 fps with sub-16 ms end-to-end frame latency across Intel, Qualcomm, and Mac platforms. - [12:19](https://www.youtube.com/watch?v=oD5z32RR4mk&t=739s) The Teams production pipeline pairs 540p MLVC encoding with receiver-side super-resolution upscaling to 720p, achieving roughly 60% bitrate savings compared to native 720p H.265 streams. - [14:22](https://www.youtube.com/watch?v=oD5z32RR4mk&t=862s) Microsoft built the VCD2 dataset to evaluate conferencing codecs, expanding capture volume fivefold over VCD1 using lossless raw footage across diverse sensors and lighting conditions. - [14:22](https://www.youtube.com/watch?v=oD5z32RR4mk&t=862s) MLVC is undergoing internal employee ring testing in Microsoft Teams, with targeted rollout for public preview in early 2027 across Copilot+ PCs and MacBooks. - [16:11](https://www.youtube.com/watch?v=oD5z32RR4mk&t=971s) An open-source release provides the core model checkpoints, training scripts, the VCD2 benchmark suite, and an optimized C++ wrapper library to facilitate integration into third-party media engines.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=oD5z32RR4mk&t=5s) Introduction to Video Codecs and the Slow Pace of Conventional Standards

Nabakumar Khongbantabam, a product manager at Microsoft, introduces the joint initiative between Microsoft Research and engineering teams to commercialize and ship a machine learning video codec (MLVC) directly inside Microsoft Teams. Video codecs underpin virtually every modern digital video experience, from web streaming to everyday video conference calls, by dramatically shrinking the massive volume of raw data emitted by cameras so it can be transmitted and stored over standard network infrastructure. During playback, the decoder reconstructs the video stream as close to the original source fidelity as possible. Because video compression is inherently lossy, different codecs achieve varying levels of compression efficiency—trading off bitrate reductions against computational complexity. Demonstrating this disparity, Nabakumar shares a side-by-side comparison between MLVC and the ubiquitous legacy standard H.264, both streaming at a constrained 130 kilobits per second (kbps), highlighting that MLVC reproduces a dramatically sharper, higher-quality image where H.264 visibly breaks down.

The broader problem with traditional video codecs is the glacial pace of standard development and commercial adoption. Historical video coding advances occur roughly once every decade: H.264 emerged over twenty years ago and remains widespread; H.265 (HEVC) arrived roughly ten years later promising half the bitrate; and H.266 (VVC) was finalized recently and is only beginning the initial steps toward productization. Developing a conventional codec requires a multi-stage process spanning standard body debates, dedicated fixed-function silicon architecture design by hardware vendors, device production, and finally the multi-year cycle of consumer adoption. Nabakumar emphasizes this protracted timeline:

> “So overall it can easily take 8 to 12 years from spec to scale.”

- **H.264 standard**: Developed ~20 years ago and remains in active commercial use. - **H.265 (HEVC)**: Arrived ~10 years after H.264, delivering roughly a 50% bitrate reduction for equivalent quality. - **H.266 (VVC)**: Recently standardized and only now entering early productization stages. - **Spec-to-scale cycle**: 8 to 12 years required across international standardization, silicon fabrication, OEM hardware distribution, and end-user hardware refresh cycles.

---

[03:07](https://www.youtube.com/watch?v=oD5z32RR4mk&t=187s) Neural Video Codecs on NPUs: Overcoming Silicon Lag

To circumvent the decade-long silicon lag, Microsoft's strategy pivots around Neural Processing Units (NPUs), which are now integrated into modern laptops and mobile devices. Rather than relying on expensive server-grade GPUs or waiting for fixed-function silicon encoders, ML video codecs execute directly on consumer NPUs using end-to-end trained deep neural networks designed specifically for reconstructive fidelity. This flexibility allows Microsoft to tailor neural compression models to specific communication workloads and offers three fundamental operational advantages over conventional silicon-bound codecs.

First, neural codecs achieve substantially superior compression efficiency, delivering higher perceived visual quality at drastically reduced bitrates. In production testing, MLVC achieves equivalent or superior fidelity using 65% less bitrate than H.265. Second, software-driven neural models allow instant, frictionless deployment; an updated model can be delivered overnight within a standard Microsoft Teams application update and run immediately on existing client devices, as demonstrated by MLVC deploying straight to new Copilot+ PCs. Third, NPU execution provides broader backward compatibility across varied hardware tiers without requiring new silicon, running successfully on platforms dating back to Apple's M1 chips. Highlighting this software-driven delivery velocity, Nabakumar notes:

> “The second big advantage is that we can deploy it much faster when we have an update of the model.”

```mermaid flowchart TD accTitle: Conventional Silicon vs ML NPU Codec Deployment accDescr: Diagram contrasting the slow 8-12 year cycle of conventional video codecs with the rapid software update model of NPU-based ML codecs.

subgraph Conventional ["Conventional Codecs (8-12 Years)"] A["Standardization Bodies"] --> B["Silicon Architecture Design"] B --> C["Hardware OEM Device Release"] C --> D["Consumer Fleet Refresh"] end

subgraph Neural ["ML Video Codec on NPUs (Weeks/Months)"] E["End-to-End Deep Model Training"] --> F["Client App Software Update"] F --> G["Instant Deployment on Consumer NPUs"] end ```

- **Compression efficiency**: MLVC delivers comparable or better visual reconstruction using 65% less bitrate than H.265. - **Deployment speed**: Instant software delivery through application patch cycles without waiting for hardware refreshes (active on Copilot+ PCs). - **Backward reach**: Operational across diverse existing hardware, including Apple Silicon (M1 and newer).

---

[06:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=377s) Research to Product: The Dual Bottlenecks of Bit-Exactness and Compute

Neural video codecs are deeply rooted in academic literature, with Microsoft Research pioneering the Deep Context Video Compression (DCVC) family. The team developed multiple iterative generations, culminating in DCVC-RT (Real-Time DCVC), which introduced critical real-time features including adaptive rate control, NPU runtime suitability, and substantial compute efficiency gains over prior academic benchmarks. However, transitioning DCVC-RT from a controlled laboratory setting to a consumer application like Microsoft Teams uncovered two major technical obstacles: computational complexity and bit-exactness.

While research models run unconstrained on high-end desktop GPUs, real-world teleconferencing demands real-time frame processing on standard consumer laptop NPUs under tight thermal and power envelopes. The more insidious blocker, however, was the requirement for bit-exactness. In neural codecs, the learned entropy model between the encoder and decoder must synchronize mathematically to mirror probability distributions and decode symbols accurately. In a consumer ecosystem as fragmented as Microsoft Teams—spanning multiple hardware manufacturers, NPU instruction sets, and divergent floating-point execution implementations—achieving bit-exact numerical parity proved virtually impossible. To illustrate the catastrophic nature of this breakdown, Nabakumar presents a cross-platform test where a frame encoded with DCVC-RT on an Apple M3 processor was sent to an Intel Lunar Lake laptop:

> “When we attempted to decode it on the lunar lake device it completely failed and that is because of the issue that I described earlier.”

- **DCVC-RT (Real-Time DCVC)**: State-of-the-art neural codec developed by Microsoft Research featuring dynamic rate control and runtime optimizations. - **Compute disparity**: Transitioning research workflows off power-hungry discrete GPUs onto lightweight consumer laptop NPUs. - **Cross-platform failure**: Floating-point calculation differences across vendor architectures (e.g., Apple M3 encoding decoded by Intel Lunar Lake) break entropy model probability sync, leading to instant decoding crashes.

---

[09:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=557s) Engineering Around Bit-Exactness: Scale Indices and Drift Mitigation

To solve cross-platform decoding crashes, Microsoft abandoned the academic ideal of universal bit-exactness in favor of a pragmatic engineering workaround: transmitting the scale indices directly over the network to the decoder rather than forcing the decoder's local entropy model to infer them. Transmitting these explicit indices introduces a minor bitrate overhead penalty, but it successfully sidesteps the requirement for identical floating-point arithmetic across heterogeneous client hardware.

```mermaid flowchart LR accTitle: MLVC Heterogeneous Stream and Drift Recovery accDescr: Flow diagram showing scale index transmission and drift mitigation techniques across diverse hardware.

subgraph Sender ["Encoder (e.g., Apple M3)"] ENC["Neural Encoder"] --> OUT["Transmit Compressed Frame"] IDX["Extract Scale Indices"] --> SIG["Explicit Index Transmission"] end

subgraph Receiver ["Decoder (e.g., Intel Lunar Lake)"] SIG --> DEC["NPU Neural Decoder"] OUT --> DEC REC["Mitigation: Long-Term References & I-Frame Resets"] -.-> DEC end ```

Eliminating hard decode crashes revealed a secondary challenge: temporal decoder drift. Because the recurrent neural network (RNN) layers in the model feed past state representations into future predictions, minute numerical rounding errors across differing NPUs gradually compound frame after frame, causing image quality to degrade into noise over time. To stabilize long-running calls, the team introduced three architectural mitigations: 1. **Long-Term Reference (LTR) recovery**: Periodically binding active decoding passes back to known stable reference frames. 2. **Proactive intra-frame (I-frame) resets**: Refreshing the temporal state chain at regular intervals, mirroring conventional video codecs. 3. **Operator sanitization**: Replacing arbitrary non-linear activation functions—which hardware vendors implement inconsistently—with numerically robust, standardized operators such as ReLU and Leaky ReLU.

Demonstrating the efficacy of this production-hardened pipeline, Nabakumar shares a visual test between MLVC and H.265, both operating at 200 kbps. Where H.265 exhibits heavy compression artifacts and noise around the speaker's face and background, MLVC maintains razor-sharp facial details and clean backgrounds. Nabakumar underscores that the added overhead needed for platform stability does not compromise the codec's value proposition:

> “So overall despite all the enhancements and increase in the bit rate because of the additional overhead we are sending it is still significantly better than H.265.”

With interoperability stabilized, the engineering team turned its focus toward optimizing real-time execution speeds directly on consumer NPUs through deep vendor-specific hardware collaborations.

[12:19](https://www.youtube.com/watch?v=oD5z32RR4mk&t=739s) Real-Time Hardware Optimization and Rate-Distortion Performance

To bring machine-learning-based video coding out of research environments and into production, Microsoft partnered directly with silicon vendors to optimize the models specifically for each hardware target. These joint efforts made real-time execution possible across Intel, Qualcomm, and Mac platforms. For production interactive video, the engineering bar was clear: the system required at least 30 frames per second with an end-to-end frame latency under 16 milliseconds. Nabakumar Khongbantabam confirms that internal engineering hit these strict operational budgets, showing internal ring call testing where employees dogfood the codec inside Microsoft Teams prior to any public preview.

When comparing MLVC against standard H.265 (HEVC), the production setup in Microsoft Teams departs slightly from the baseline configuration published in the original research paper. While the academic paper compared both codecs natively at 540p, the Teams implementation targets 720p video delivery. Under this pipeline, MLVC encodes at 540p and reconstructs to 720p on the receiving end using an upscaler such as a neural super-resolution model or Lanczos filtering, whereas H.265 operates directly at native 720p. The resulting rate-distortion curves demonstrate substantial efficiency gains, with the MLVC curve tracking well above H.265 across comparable perceptual quality metrics. Demonstrating the magnitude of the improvement, Nabakumar Khongbantabam highlights: “This roughly corresponds to about 60% reduction in bit rate by MLVC when we maintain the same perceptual video quality between the two codecs.”

```mermaid flowchart LR accTitle: MLVC versus H265 Teams Pipeline accDescr: Diagram contrasting the Teams MLVC 540p super-resolution pipeline with native H265 720p encoding.

subgraph MLVC_Pipeline["MLVC Teams Pipeline"] A["Capture 540p"] --> B["MLVC Encode & Transmit"] B --> C["Super Resolution / Lanczos"] C --> D["720p Perceptual Output"] end

subgraph H265_Pipeline["Conventional Pipeline"] E["Capture 720p"] --> F["H.265 Native Encode"] F --> G["720p Output"] end ```

- Hardware execution targets real-time video delivery at $\ge$30 fps with a frame latency of $\le$16 ms on Intel, Qualcomm, and Apple Mac hardware. - The Teams production pipeline pairs 540p MLVC encoding with receiver-side super-resolution or high-quality upscalers to outperform native 720p H.265 streams. - The pre-trained model weights reflecting this configuration are published in the open-source repository for public evaluation.

---

[14:22](https://www.youtube.com/watch?v=oD5z32RR4mk&t=862s) The VCD2 Evaluation Dataset and Teams Rollout Roadmap

Rigorous verification of the codec required realistic, modern benchmarking data. Microsoft previously released VCD (Video Conferencing Dataset), which helped evaluate earlier computer vision algorithms, but production-grade testing demanded an updated suite: VCD2. This upgraded second generation expands the capture footprint to five times the size of VCD1, incorporating a broader spectrum of camera sensors, lighting scenarios, and subject variety. Critically, VCD2 stores every raw video capture with lossless compression, ensuring that no prior encoder artifacts or compression noise contaminate benchmark evaluations.

Deploying MLVC into Microsoft Teams follows a staged ring deployment model. Having worked around core machine learning and systems hurdles, Microsoft advanced the codec to internal test rings to polish real-world edge cases. The roadmap calls for introducing MLVC to public preview early next year in 2027, followed by general availability across Copilot+ PCs powered by Intel and Qualcomm silicon, later expanding to AMD platforms as well as Apple MacBooks. Outlining the platform's long-term trajectory, Nabakumar Khongbantabam states: “So overall our aim is to migrate the video codec used in teams to our ML video codec over time.”

- **VCD2 Benchmark Dataset**: Contains 5× more footage than VCD1, broader camera model diversity, and completely lossless raw source captures. - **Deployment Stage**: Operating inside internal employee testing rings (step two) before opening to broader audiences. - **Rollout Schedule**: Targeted public preview rollout in early 2027 across Copilot+ PCs (Intel, Qualcomm, and eventually AMD) and MacBooks, with wider platform support slated for late 2027.

---

[16:11](https://www.youtube.com/watch?v=oD5z32RR4mk&t=971s) Open-Source Ecosystem and C++ Codec Library

To spur industry adoption and community experimentation, Microsoft open-sourced the complete MLVC technical stack rather than reserving it solely for Teams infrastructure. The open-source deliverables encompass the core codec architectures alongside fully trained model checkpoints that developers can deploy out of the box. The repository includes end-to-end training pipelines, step-by-step guidance on curating training data, and the full VCD2 evaluation benchmark suite.

To bridge the gap between Python-heavy research environments and real-world software engineering, the release provides an optimized C++ wrapper library. This runtime layer abstracts the internal deep learning inference engine, providing standard coding interfaces that allow engineers to plug the machine learning codec into native media pipelines and external video applications without rewriting model management code. Summarizing the intent behind the release, Nabakumar Khongbantabam explains: “We have also released the C++ codec library that wraps the model and makes it really easy for any developer to try it out and integrate it in an application.” Extensive technical documentation and previously published academic papers accompany the release to give engineers insight into the underlying mathematical formulations and systems-level tradeoffs.

## Notable quotes > [00:05](https://www.youtube.com/watch?v=oD5z32RR4mk&t=5s) So overall it can easily take 8 to 12 years from spec to scale. — Nabakumar Khongbantabam

## Mentioned Nabakumar Khongbantabam, Microsoft, Microsoft Research, Microsoft Teams, MLVC, H.264, H.265 (HEVC), H.266 (VVC), Copilot+ PC, Apple, Apple M1, Apple M3, Intel Lunar Lake, Intel, Qualcomm, AMD, DCVC, DCVC-RT, VCD, VCD2

## Video description Nabakumar Khongbantabam from Microsoft

[Source](https://www.youtube.com/watch?v=oD5z32RR4mk)

中文简报

微软将神经视频编解码器引入Teams，码率降低60% 通过传输显式缩放索引以消除跨平台算术误差，微软在消费级NPU上实现了实时神经视频压缩，以极低的带宽占用超越了H.265。传统视频编解码标准从规范确立到硬件普及通常需要8至12年的周期，这促使微软开发出可直接在消费级神经处理单元（NPU）上运行且支持软件更新的机器学习视频编解码器（MLVC）。将DCVC-RT等研究模型转化为生产应用，需要解决来自苹果、英特尔和高通等异构芯片之间严重的跨平台位精确度缺失及时间漂移问题。通过传输显式缩放索引，并将540p神经编码与客户端超分辨率相结合，该团队在低于16毫秒的帧延迟下，相比H.265实现了60%至65%的码率降低。微软目前正在Teams内部测试通道中验证该编解码器，计划于2027年随Copilot+ PC推出，同时已将相关模型、VCD2基准数据集以及C++运行时库向开源社区发布。微软正在将机器学习视频编解码器（MLVC）直接产品化引入Microsoft Teams，以取代传统依赖固定芯片的视频编解码器。 (0:05) 通过直接在消费级笔记本NPU上运行，而非等待专用的固定功能硬件，MLVC将传统8至12年的“从标准到硬件普及”周期缩短至软件更新的速度。 (3:07) 在生产测试中，与H.265相比，MLVC能以高至60%至65%的码率降幅，实现相当甚至更优的主观视觉质量。 (12:19) 为防止在异构NPU硬件架构上发生解码崩溃，微软通过直接在网络上传输缩放索引，解决了浮点运算位精确度的问题。 (9:17) 在计划于2027年初推出公开预览版之前，整个MLVC技术栈（包括模型检查点、VCD2基准数据集和C++集成库）均已开源。 (16:11)

Reading summary

Microsoft is working to replace legacy video standards in Microsoft Teams with an in-house machine learning video codec (MLVC) running on consumer Neural Processing Units (NPUs). Product manager Nabakumar Khongbantabam explains how the team overcame cross-platform numerical drift and silicon lag to deliver up to a 60% to 65% bitrate reduction over H.265. Circumventing Decade-Long Silicon Adoption Cycles (0:05) Traditional video codecs evolve at a sluggish pace dictated by standardization bodies, chip fabrication, and device replacement cycles. While H.264 remains widespread after twenty years and H.265 (HEVC) arrived a decade later, the overall 'spec-to-scale' journey takes between 8 and 12 years. Even recently finalized formats like H.266 (VVC) are only beginning productization. Rather than waiting for dedicated fixed-function hardware encoders, Microsoft executes deep neural network models directly on consumer NPUs across modern laptops. This software-driven architecture enables updates to deploy overnight via routine application patches—such as direct distribution to Copilot+ PCs—while providing backward compatibility across existing hardware architectures, including Apple Silicon dating back to M1 chips. Resolving Cross-Platform Bit-Exactness and Temporal Drift (6:17) Commercializing Microsoft Research's Deep Context Video Compression (DCVC-RT) exposed severe interoperability hurdles. In neural codecs, the learned entropy model between the sender and receiver must match mathematically to decode properly. In real-world multi-vendor environments, differing floating-point execution across NPU architectures caused instant decoding crashes when sending streams between mismatched hardware, such as an Apple M3 encoding to an Intel Lunar Lake decoder. To bypass the impossible requirement of universal bit-exactness, the engineering team opted to transmit scale indices explicitly over the network. Although this adds minor bitrate overhead, it decouples the decoder from vendor-specific floating-point variations. To prevent compounding rounding errors across recurrent neural network (RNN) layers from turning streams into visual noise, the architecture incorporates regular intra-frame (I-frame) resets, Long-Term Reference (LTR) recovery frames, and sanitized activation functions like ReLU and Leaky ReLU. Real-Time Latency Budgets and Production Rate-Distortion (12:19) Deploying MLVC in Microsoft Teams required hitting a real-time budget of at least 30 frames per second with an end-to-end frame latency under 16 milliseconds across Intel, Qualcomm, and Apple hardware. To maximize compression efficiency, the Teams production pipeline captures and encodes video at 540p before reconstructing it at the receiver into 720p using super-resolution models or Lanczos upscaling. Rate-distortion benchmarks show that MLVC outperforms native 720p H.265 streams, achieving equivalent visual quality with roughly a 60% to 65% reduction in bitrate. In low-bandwidth tests at 130 kbps and 200 kbps, MLVC preserved sharp facial features and clean backgrounds where H.264 and H.265 displayed heavy compression artifacts and breakdown. Validation Datasets, Open-Source Tooling, and Rollout Roadmap (14:22) To evaluate performance against real-world video conferencing conditions, Microsoft built the VCD2 (Video Conferencing Dataset 2) benchmark. VCD2 expands capture volume to five times that of VCD1, incorporating diverse camera sensors and lighting environments with lossless raw captures to avoid pre-existing compression artifacts. To drive broader industry adoption, Microsoft open-sourced the model weights, training pipelines, the VCD2 benchmark, and an optimized C++ wrapper library that simplifies embedding ML models into native media pipelines. Currently running in internal Microsoft employee testing rings, MLVC is scheduled for a public preview in early 2027 on Copilot+ PCs (Intel and Qualcomm) and MacBooks, with subsequent expansion planned for AMD silicon and broader platforms later in 2027. Microsoft aims to migrate Teams entirely to its ML video codec over time, but the rollout remains gated by phased testing rings and depends on widespread consumer adoption of NPU-equipped devices through 2027.

精炼阅读摘要

微软产品经理 Nabakumar Khongbantabam 介绍了将机器学习视频编解码器（MLVC）落地于 Microsoft Teams 的技术突破。该方案利用终端设备的神经处理单元（NPU）运行端到端深度神经网络，绕过了传统编解码器长达 8 至 12 年的专用硅芯片研发周期，在同等感知质量下较 H.265 节省了约 60% 至 65% 的码率。突破硬件周期：NPU 驱动的神经编解码 (3:07) 传统视频编解码标准依赖于极其漫长且成本高昂的演化路径。Khongbantabam 指出，从 H.264、H.265（HEVC）到近期的 H.266（VVC），技术跨度往往以十年计；一个新方案从国际标准确立、芯片厂商设计专用固定功能硅架构、OEM 硬件生产，到最终在消费者端普及，通常需要经历 8 到 12 年的滞后周期。这种以固定硅芯片为核心的模式，严重限制了视频压缩技术的快速迭代与创新分发。为彻底规避硬件滞后，微软将运行平台转向现代 PC 和移动设备内置的 NPU。MLVC 摆脱了昂贵的服务器端 GPU 和专用编码硅芯片，直接在终端 NPU 上运行端到端训练的深度神经网络。这种软件驱动模式不仅在生产测试中以比 H.265 低 65% 的码率实现了同等或更优画质（例如在 130 kbps 极低码率下图像依旧锐利），更使技术部署周期缩短至数周：工程师只需推送一次 Teams 应用程序更新，即可让 Copilot+ PC 以及可追溯至 Apple M1 的既有硬件设备即刻获得最新的编解码能力。攻克工程瓶颈：浮点精度失效与时域漂移 (6:17) 神经编解码技术由微软研究院的 DCVC（深度上下文视频压缩）家族发展而来，其最新成果 DCVC-RT 针对实时场景进行了优化。然而，将实验室模型迁移至工业级应用时，团队遇到了算力约束与跨硬件“完全位精确（Bit-Exactness）”两大阻碍。在神经编解码中，编码器与解码器的熵模型必须在概率分布上精确同步；但因各硬件厂商 NPU 的浮点指令集和计算实现存在微小差异，跨芯片解码时概率分布会迅速失真。在实际测试中，由 Apple M3 编码的数据流送至 Intel Lunar Lake 笔记本解码时会发生彻底崩溃。微软并未执着于理论上的绝对位精确，而是采取了务实的工程妥协：通过网络直接向解码器显式传输“尺度索引（Scale Indices）”，以轻微的码率开销绕过了跨硬件浮点计算不一致的问题。此外，针对循环神经网络（RNN）长时状态传递引发的数值误差累积（即图像逐步劣化为噪点的“时域漂移”），团队实施了三重加固方案：周期性引入长期参考帧（LTR）绑定基准、引入常规视频中的周期性 I 帧重置状态链，并将各厂商实现差异大的非线性激活函数规范化为数值稳定的 ReLU 和 Leaky ReLU。现场测试证明，在 200 kbps 极端码率下，MLVC 即便承担额外开销，其面部细节与画质仍远胜噪点明显的 H.265。实时性能调优与 540p 超分架构 (12:19) 为了让模型在多供应商硬件上平稳运行，微软与芯片厂商展开了针对性优化，使模型在 Intel、Qualcomm 以及 Apple Mac 平台上均能满足交互式视频会议的工业级门槛：帧率达到每秒 30 帧以上，端到端单帧延迟控制在 16 毫秒以内。目前，该系统已在微软内部员工测试环中展开全面验证与体验。在具体的工程管线设计上，Teams 生产环境针对主流 720p 视频输出采用了与学术论文不同的创新设计：MLVC 在发送端先以 540p 分辨率进行高效编码传输，接收端解码后再通过神经超分辨率模型或 Lanczos 滤波器重建放大至 720p；相比之下，对照组 H.265 则在原生 720p 下编解码。率失真曲线显示，在此架构下维持相同主观感知视频质量时，MLVC 管线相较原生 720p 的 H.265 实现了大约 60% 的显著码率缩减。 VCD2 评估数据集与 Teams 部署路线 (14:22) 为保障系统验证的可靠性，微软构建了第二代视频会议数据集 VCD2。相比早期用于算法验证的 VCD1，VCD2 的数据量扩大了 5 倍，覆盖了更多样化的摄像头传感器型号、复杂的室内光照条件与丰富的被测主体，并且全程采用无损格式捕获存储原始帧，确保评估不会受到任何既有压缩伪影的干扰。微软计划通过分阶段环形发布推进 MLVC 的商业化：在完成当前内部员工测试后，预计于 2027 年初在搭载 Intel 和 Qualcomm 芯片的 Copilot+ PC 以及 MacBook 上开启公开预览，随后逐步支持 AMD 平台并在 2027 年晚些时候扩展到更广泛的硬件生态。Khongbantabam 明确表示，微软的长期战略是逐步将 Teams 的底层视频流全面迁移至自研的机器学习编解码器。为加速生态共建，微软已将 MLVC 的架构、预训练模型权重、训练管线、VCD2 数据集以及高度优化的 C++ 包装库完整开源，方便外部开发者将其直接接入原生多媒体管线。微软验证了将神经视频编解码器推进至亿级用户生产环境的可行性，但该方案当前深度依赖终端设备的 NPU 算力与专用底层驱动支持。在缺乏神经加速硬件的老旧机型及碎片化的低端移动设备上，如何平衡算力消耗与画质收益，仍是 Teams 彻底取代传统编解码器必须面对的长期课题。

Detailed notes — zh

## Background 本内容由 YouTube 频道 @Scale 于 2026 年 10 月 30 日发布，页面简介显示该演讲来自微软的 Nabakumar Khongbantabam，重点分享机器学习视频编解码器从前沿研究突破走向工业级规模应用的技术实践。 ## TL;DR 微软产品经理 Nabakumar Khongbantabam 系统阐述了为 Microsoft Teams 打造机器学习视频编解码器（MLVC）的全过程。针对传统视频编码标准迭代周期长达 8 至 12 年的困境，团队转而利用主流设备中的 NPU 算力，摆脱了对专用硬件的依赖并实现向下兼容。面对异构硬件带来的跨平台解码器崩溃与时间累积漂移两大落地瓶颈，团队通过直接网传尺度索引、长期参考帧机制、主动 I 帧重置及算子重塑成功突围。实测显示，MLVC 在单帧延迟低于 16 ms、30 fps 的实时性能下，相比 H.265 实现了高达约 60% 的码率节省，整套方案与评测数据集现已完全开源，并计划于 2027 年推向公开预览。 ## Atmosphere & style 整场演讲呈现出浓厚而扎实的工程技术底色，风格严谨克制、条理清晰且直击工业痛点。Nabakumar Khongbantabam 以产品研发亲历者的从容姿态娓娓道来，没有过多空洞的行业宣传，而是坦诚拆解学术模型在异构硬件生产环境中遭遇的崩溃死结与工程破局过程。配合真实通话抓帧、速率失真对比曲线与视频实测录屏，给观众带来一种高度务实、见证前沿深度学习技术真正攻克硬核工程难关并落地亿级产品的硬核沉浸感。 ## Bulletin - [00:05](https://www.youtube.com/watch?v=oD5z32RR4mk&t=5s) 微软正在将其自研的机器学习视频编解码器（MLVC）落地到 Microsoft Teams 中，打破了传统视频编解码标准从制定到硬件普及长达 8 至 12 年的漫长研发周期。 - [03:07](https://www.youtube.com/watch?v=oD5z32RR4mk&t=187s) MLVC 基于消费级设备自带的 NPU 运行，在相同主观视频感知质量下相比主流的 H.265 标准可缩减多达 60% 至 65% 的码率。 - [09:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=557s) 为解决异构硬件由于浮点运算差异导致的比特不一致与解码器崩溃，团队通过网传尺度索引（Scale Indices）绕过跨平台浮点反推，并结合长期参考帧与周期性 I 帧有效抑制了递归漂移。 - [12:19](https://www.youtube.com/watch?v=oD5z32RR4mk&t=739s) MLVC 在 Intel、Qualcomm 及 Apple Mac 平台均已实现 30 fps 以上、单帧延迟低于 16 ms 的实时性能，预计将于 2027 年初面向公众推出预览版，整套模型权重与 C++ 库已全面开源。 ## Key takeaways - [00:05](https://www.youtube.com/watch?v=oD5z32RR4mk&t=5s) 传统视频编解码器（如 H.264、H.265、H.266）依赖严苛的国际标准制定与专用 ASIC 芯片研发，从规范确立到规模化普及往往耗时长达 8 至 12 年。 - [03:07](https://www.youtube.com/watch?v=oD5z32RR4mk&t=187s) 神经视频编解码器放弃了专用硬件芯片绑定，直接利用现代笔记本电脑与手机普遍内置的 NPU 进行推理运算，摆脱了对高功耗独立 GPU 的依赖。 - [03:07](https://www.youtube.com/watch?v=oD5z32RR4mk&t=187s) 基于软件模型部署的 MLVC 具备极速迭代优势，无需更换物理硬件即可通过 Teams 客户端日常软件升级分发，并能向下兼容运行在 Apple Silicon M1 等老款存量设备上。 - [06:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=377s) 学术界研发的神经编解码器（如 DCVC-RT）落地工业界的最大障碍，在于异构硬件因底层浮点舍入与算子实现差异而无法实现百分之百的“比特级一致性（Bit Exactness）”。 - [06:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=377s) 微软跨平台实测发现，在苹果 M3 芯片上编码的视频流在英特尔 Lunar Lake 设备上解码时会因熵模型计算出的索引不匹配而直接崩溃报错。 - [09:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=557s) 为彻底消除跨芯片解码崩溃，微软改由编码端直接将尺度索引随数据包打包传输给解码端，虽增加少量开销但成功解耦了对跨平台精确浮点反推的依赖。 - [09:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=557s) 针对 RNN 递归网络在跨芯片运算时随时间滚雪球累积的“解码器长期漂移”，团队组合采用了长期参考帧恢复（LTR）、周期性 I 帧重置以及替换数值更稳健算子（如 ReLU/Leaky ReLU）等多重防护机制。 - [12:19](https://www.youtube.com/watch?v=oD5z32RR4mk&t=739s) 通过与英特尔、高通和苹果的深度联合优化，MLVC 在轻薄本低功耗环境下成功达成了视频会议所需的硬性门槛：帧率至少 30 fps 且单帧延迟不超过 16 毫秒。 - [12:57](https://www.youtube.com/watch?v=oD5z32RR4mk&t=777s) Teams 实际传输管线采用 MLVC 端侧 540p 编码配合超分辨率/Lanczos 上采样至 720p 输出的方案，对标原生 720p 编码的 H.265 实现了约 60% 的码率节省。 - [12:57](https://www.youtube.com/watch?v=oD5z32RR4mk&t=777s) 微软推出了规模达到初代 5 倍的全新视频会议评测数据集 VCD2，全流程采用无损压缩源文件，有效消除了传统编码压缩噪声对画质评测的干扰。 - [15:13](https://www.youtube.com/watch?v=oD5z32RR4mk&t=913s) MLVC 已在微软内部测试环路运行验证，计划于 2027 年初上线 Public Preview 公开预览版，其终极目标是在 Teams 中全面逐步替代现有的传统编解码器。 - [15:13](https://www.youtube.com/watch?v=oD5z32RR4mk&t=913s) 微软已将涵盖预训练权重、端到端训练管线、VCD2 基准数据集以及生产级 C++ 编解码器封装库在内的整套生态完全向社区开源。 ## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=oD5z32RR4mk&t=5s) 视频编解码器基础与传统标准的演进困境演讲者 Nabakumar Khongbantabam 是微软的产品经理，负责推动用于 Microsoft Teams 的机器学习视频编解码器（ML Video Codec，简称 MLVC）的研发与产品化。他与微软研究院（MSR）以及工程开发团队紧密协作，致力于将这项技术正式落地到 Teams 中。演讲伊始，Nabakumar 向观众普及了视频编解码器的基本概念：任何近期参与过视频会议或流媒体播放的用户都体验过编解码器的作用。摄像机采集的原始视频数据量极其庞大，编解码器的核心功能就是将这些数据压缩到便于网络传输或本地存储的大小；在回放时，解码端再从压缩码流中重建视频，并尽可能将画质还原到接近原始输入的水平。然而，由于压缩必然伴随着信息损失，解出来的画质永远不可能和原始源完全一致，这也导致不同编解码器在压缩效率上存在差异——有的压缩率极高，有的虽然码率偏高但在其他工程维度上更具优势。为了直观展示这一差异，现场展示了一组同在 130 kbps 极低码率下的对比视频：左侧的 MLVC 重建出的画面质量显著高于右侧广泛使用的传统 H.264 编解码器。 ```mermaid flowchart LR accTitle: 传统视频编解码标准研发周期流程 accDescr: 展示传统编解码器从制定国际标准到硬件落地再到大规模用户采纳的长达8至12年的漫长过程。 subgraph Phase1 ["标准确立"] A["标准制定 (Standardization)<br/>耗时数年制定技术规范"] end subgraph Phase2 ["硬件实现"] B["芯片设计与制造 (Silicon Design)<br/>硬件厂商开发专属硬编解码单元"] end subgraph Phase3 ["生态普及"] C["设备出货与用户采纳 (Adoption)<br/>终端逐步更新换代至规模化"] end A --> B B --> C ``` Nabakumar 回顾了传统视频编解码器的发展历史。传统标准大约每十年才完成一次代际更替：H.264 诞生于大约 20 年前，至今仍被大规模使用；大约十年后出现的 H.265 在同等画质下将码率降低了约 50%；而最新的 H.266 刚刚定稿，目前尚处于早期的产品化阶段。传统编解码器的迭代之所以如此缓慢，是因为其研发阶段极其繁重： - **制定标准**：需要各大标准化组织经过数年争辩与测试来制定严格规范。 - **硅片设计与流片**：硬件供应商根据标准规范设计专用的专用集成电路（ASIC）硬件电路并随新芯片发货。 - **终端普及周期**：终端消费者和企业需要数年时间采购并更换新设备，正如 Nabakumar 指出的：“So overall it can easily take 8 to 12 years from spec to scale.” ### [03:07](https://www.youtube.com/watch?v=oD5z32RR4mk&t=187s) 基于 NPU 的神经视频编解码器及其核心优势在概述整场演讲的四大板块（前沿神经编解码方法、Teams 落地遭遇的两大难题、工程解决策略、最新评估数据与开源资源）后，Nabakumar 重点阐述了微软正在采取的新型研发路径。该方案放弃了对传统 ASIC 专用编解码硬件的依赖，转而全面利用近年来主流消费级设备（如笔记本电脑、智能手机）中普遍搭载的神经网络处理单元（NPU）。这一架构转变让神经视频编解码器无需昂贵且高功耗的独立 GPU 就能直接运行在普通用户的个人终端上。通过端到端训练深度学习模型来重构视频质量，团队获得了前所未有的工程灵活性，能够根据实际视频会议场景进行定向模型调优。 Nabakumar 总结了利用 NPU 运行神经视频编解码器相比传统固定硬件方案的三大显著优势： 1. **更高的压缩效率**：神经编解码器在本质上能实现更优的视频压缩表现。在实际测试中，MLVC 相比 H.265 可节省多达 65% 的码率，在同等带宽下带来画质的大幅跃升。 2. **极速的应用层部署能力**：传统硬件编解码器升级必须更换物理芯片，而神经编解码器作为深度学习模型，只要算法更新，就可以直接打包进 Teams 的常规客户端软件更新中，实现对海量终端的近乎即时分发。例如，最新发售的所有 Copilot+ PC 设备已经全量支持 MLVC。 3. **更广泛的存量设备覆盖**：由于 NPU/AI 算力已经在不同架构中沉淀，算法可以向下兼容老一代设备。例如 MLVC 能够顺利运行在苹果 Apple Silicon M1 及更新的机型上，打破了传统新编码格式必须依赖最新一代专属硬件芯片的桎梏，让海量存量老设备用户也能立即享受新技术红利。 ### [06:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=377s) 落地 Teams 面临的两大严峻工程瓶颈虽然神经视频编解码在学术界已经研究了数年，微软研究院自身也研发出了学术界首屈一指的深度上下文视频压缩（DCVC）系列模型，并且已演进至针对低延迟场景优化的实时版本 DCVC-RT（具备码率自适应控制、NPU 适配优化及更高推理效率等特性），但要将其从实验室 Demo 搬进支撑数亿用户的实时通讯产品 Teams 中，团队依然撞上了两堵坚硬的高墙。团队遭遇的第一个核心阻碍是编解码两端的“精确匹配处理要求”。神经编解码网络中的熵编码（Entropy Coding）模块依赖熵模型在编码器与解码器之间实现百分之百的概率状态对齐，学术界通常试图通过强制要求“比特级完全一致（Bit Exactness）”来规避错误。然而 Nabakumar 坦言，这在真实生产环境中近乎不可能实现：“many researchers try to solve that by explicitly achieving bit exactness but we feel that it's very difficult to achieve in a very heterogeneous environment like we have in teams where we have a mix of different vendors and different NPU types.” 这种比特不一致在跨平台交互时带来了灾难性的跨平台兼容崩溃。现场展示的一组故障样例显示：一段视频帧在苹果 M3 芯片上通过 DCVC-RT 完成编码，当尝试在英特尔 Lunar Lake 芯片设备上解码时，解码过程直接彻底报错崩溃。其根源在于两家硬件厂商在底层 NPU 算子实现、浮点运算（Floating-Point Operations）的舍入与截断规则存在微妙差异，导致熵模型计算出的索引（Index）无法完全复现。第二个阻碍则是算力鸿沟：学术界论文中的亮眼指标几乎全部基于高端且功耗巨大的桌面级独立 GPU 跑出来的，而 Teams 必须保证算法能在功耗与散热严苛受限的常规轻薄消费级笔记本电脑上以毫秒级实时运行。 ### [09:17](https://www.youtube.com/watch?v=oD5z32RR4mk&t=557s) 跨平台互通破局：旁路索引、漂移抑制与算子重塑为了彻底解决导致解码器直接崩溃的兼容性死结，微软团队设计了一种绕过学术界固有思路的工程解法：不再依赖解码端通过复杂的熵模型去反推概率规模索引，而是由编码端直接将尺度索引（Scale Indices）随网络数据包打包发送给解码端。这种做法虽然因为引入了额外的开销（Overhead）而略微牺牲了一定的理论压缩效率，但彻底摒弃了对跨平台计算比特绝对一致的要求，使得异构设备之间的基本解码流程得以稳定建立。 ```mermaid flowchart TD accTitle: 神经编解码跨平台漂移与崩溃解决机制 accDescr: 描述如何通过传输尺度索引解决解码崩溃，并通过长期参考帧、I帧重置以及稳定算子解决时间累积漂移。 subgraph Interop ["解决解码器崩溃 (Crash Fix)"] A["编码器端计算尺度"] -->|"直接网传尺度索引 (Over the air)"| B["解码器直接读取索引"] B -.->|"避开跨芯片浮点熵推断"| C["消除 Bit Exactness 强约束"] end subgraph DriftCtrl ["解决长期解码漂移 (Drift Fix)"] D["RNN 递归网络误差随时间累积"] D --> E["长期参考帧恢复 (LTR Recovery)"] D --> F["周期性主动插入 I 帧 (I-Frame Resets)"] D --> G["替换非线性算子 (采用更稳定的 ReLU / Leaky ReLU)"] end ``` 然而，解决了当场崩溃只是第一步。由于模型内部采用了递归神经网络（RNN）结构，跨硬件平台浮点运算的微小精度误差会随着时间推移在隐状态中不断累加，最终引发严重的“解码器长期漂移（Decoder Drift）”。展示画面显示，一段视频刚开始解码时非常清晰完美，但随着时间推移，微小误差像滚雪球一样扩散，整张画面最终彻底花屏损坏。为了扑灭这种漂移，团队组合实施了多套系统级防护机制： - **长期参考帧恢复机制（LTR Recovery）**：周期性地强制将当前帧的重建绑定回此前确定的长期基准参考帧，截断误差传递链。 - **主动周期性 I 帧重置（Proactive I-Frame Resets）**：借鉴传统视频编解码器的成熟机制，主动向流中插入完整帧刷新全局状态。 - **模型算子层面的稳定性重塑**：彻底剔除并替换那些不同 NPU 厂商在底层固件中实现各异的非线性数学算子，全面改用在各大硬件架构上表现均高度一致且数值稳健的算子，例如 ReLU 或 Leaky ReLU。在应用了这一整套加固措施后，Nabakumar 播放了一段 MLVC 与 H.265 在同等 200 kbps 极低码率下的实测对比录屏：左侧的 MLVC 输出画面锐利分明，人脸轮廓和复杂背景细节均保留完好；而右侧的 H.265 则布满了因高压缩率与复杂解码引发的明显噪点与块状伪影。事实证明，即便为了换取工业级跨平台兼容性而增加了少量网络比特开销，MLVC 的最终画质表现仍然显著超越了 H.265。在此基础上，团队开始深入攻克下一个核心难关——与各大硬件厂商展开深度联合优化，把模型推入消费级 NPU 的严苛实时运行环境之中。 ### [12:19](https://www.youtube.com/watch?v=oD5z32RR4mk&t=739s) 跨硬件平台实时优化与内部环路测试为了将机器学习视频编解码器（MLVC）真正推向工业界落地，团队必须针对不同的具体硬件平台进行深度调优。Nabakumar Khongbantabam 指出，通过与各大硬件厂商紧密协作，团队成功实现了在 Intel、Qualcomm 以及 Mac 平台上的实时运行。对于视频通话而言，严苛的性能门槛是不可逾越的底线：系统必须达到至少 30 fps 的帧率，且单帧延迟必须严格控制在 16 毫秒以内。演讲者自豪地表示，他们通过这一系列协同攻坚顺利达成了目标。演讲现场展示了一张来自微软内部测试环路（internal testing rings）的真实通话截图。MLVC 已经在微软内部员工群体中先行部署测试，用于在向公众推送公开预览版（Public Preview）之前，全面排查和验证实际生产环境中的表现与稳定性。 Nabakumar Khongbantabam 总结道：“We needed at least 30 fps the frame latency no more than 16 millisecond and we have been able to achieve that through this effort.” - **帧率与延迟指标**：达到至少 30 fps，帧延迟不高于 16 ms。 - **已适配硬件平台**：Intel、Qualcomm 芯片设备以及 Apple Mac。 - **部署阶段**：已进入微软内部测试环路（Internal Testing Rings）实测。 --- ### [12:57](https://www.youtube.com/watch?v=oD5z32RR4mk&t=777s) 速率失真曲线表现与 VCD2 评测数据集在展示集成至 Microsoft Teams 的实际收益时，Nabakumar Khongbantabam 剖析了 MLVC 与传统主流编解码器 H.265 的速率失真（Rate-Distortion）对比曲线。他解释道，Teams 中的工程落地实现与之前发表的学术论文存在细微差异：论文中是在原生 540p 分辨率下对比两者，而 Teams 实际管线中双方均输出 720p。MLVC 采用在端侧从 540p 通过高质量超分辨率（Super Resolution）算法或 Lanczos 算法上采样至 720p 的策略，而 H.265 则直接在原生 720p 下编码传输。测试曲线清晰地显示代表 MLVC 的曲线上移，表明在相同或更低码率下能提供更出色的画质。在主观感知视频质量（Perceptual Video Quality）保持一致的前提下，MLVC 相比 H.265 实现了高达约 60% 的码率缩减。为了精准评估算法在视频会议场景下的表现，微软升级并推出了全新的视频会议数据集 VCD2（Video Conferencing Dataset 2）。相比早期助力计算机视觉研究的第一代 VCD1，新发布的 VCD2 实现了全面扩容与升级：包含更多视频片段、覆盖更广泛的摄像头硬件型号、整体数据规模扩大至 VCD1 的 5 倍，并且所有原始采集视频均采用无损压缩（losslessly compressed）存储与分发，彻底杜绝了压缩噪声对画质评测的干扰。 Nabakumar Khongbantabam 评价道：“This roughly corresponds to about 60% reduction in bit rate by MLVC when we maintain the same perceptual video quality between the two codecs.” ```mermaid flowchart LR accTitle: MLVC vs H265 Pipeline Comparison accDescr: Comparison of resolution workflows and bandwidth savings between MLVC and H.265 in Teams subgraph MLVC_Pipeline["MLVC 传输管线"] A["540p 输入编码"] --> B["超分辨率 / Lanczos 上采样"] B --> C["720p 高质量输出"] end subgraph H265_Pipeline["传统 H.265 管线"] D["原生 720p 编码"] --> E["原生 720p 输出"] end C -.->|"节省约 60% 码率 (相同主观画质)"| E ``` - **码率节省**：在维持相同主观感知质量下，码率相比 H.265 降低约 60%。 - **管线方案**：MLVC（540p 编码 + 超分/Lanczos 放大至 720p）对标 H.265（原生 720p 编码）。 - **VCD2 数据集特性**： - 数据集规模达到 VCD1 的 5 倍。 - 覆盖更多镜头型号与更多真实会议片段。 - 全流程无损压缩源文件，防止编码噪声引入。 --- ### [15:13](https://www.youtube.com/watch?v=oD5z32RR4mk&t=913s) Teams 落地路线图与全栈开源生态谈及用户何时能在 Microsoft Teams 中真正用上该技术，Nabakumar Khongbantabam 梳理了清晰的时间表。目前团队已经跨过最艰难的模型研发与生产级打磨阶段，正处于内部测试环路的 Step 2。根据规划，团队计划在 2027 年初将该编解码器推向公共预览版（Public Preview），随后逐步向普通大众用户推送。首波受支持的硬件设备将覆盖 Intel、Qualcomm 乃至未来的 AMD Copilot+ PC 以及 Apple MacBook 设备，并计划在次年下半年进一步拓宽支持的硬件与平台范围。从长远来看，微软的终极目标是逐步将 Teams 中的传统编解码器全面迁移至自研的 ML 视频编解码器。此外，微软已将整套工作成果完全开源。开源资产不仅包含完整的编解码器模型及其预训练权重（直接开箱可用），还涵盖了端到端的训练流程、训练数据生成指南以及前文提到的 VCD2 评测数据集。为了让开发者能够毫无阻碍地将模型集成进各类应用中，团队还专门开源了封装底层模型的 C++ 编解码器库，并配套提供了详尽的工程文档和多篇学术论文供行业参考学习。 Nabakumar Khongbantabam 坚定地阐述了这一演进愿景：“So overall our aim is to migrate the video codec used in teams to our ML video codec over time.” - **发布时间表**： - 当前阶段：内部测试环路（Step 2）。 - 2027 年初：上线 Public Preview 公开预览版，后续推进至正式版。 - 2027 年下半年：进一步扩大支持平台范围。 - **设备支持规划**：Intel、Qualcomm 及 AMD 的 Copilot+ PC，以及 Apple MacBooks。 - **开源生态交付物**： - 编解码器模型架构与预训练模型权重。 - 完整训练管线与训练数据生成文档。 - 官方评测基准 VCD2 数据集。 - 生产级 C++ 编解码器封装库（C++ codec library）及学术论文支撑。 ## Notable quotes > [00:05](https://www.youtube.com/watch?v=oD5z32RR4mk&t=5s) So overall it can easily take 8 to 12 years from spec to scale. — Nabakumar Khongbantabam ## Mentioned Nabakumar Khongbantabam, Microsoft, Microsoft Teams, Microsoft Research (MSR), MLVC, H.264, H.265, H.266, DCVC, DCVC-RT, Copilot+ PC, Apple, Apple Silicon M1, Apple M3, Intel, Intel Lunar Lake, Qualcomm, AMD, MacBook, VCD1, VCD2, Lanczos ## Video description Nabakumar Khongbantabam from Microsoft [Source](https://www.youtube.com/watch?v=oD5z32RR4mk)

Source quotes

[0:05] Nabakumar Khongbantabam: So overall it can easily take 8 to 12 years from spec to scale.

Source quotes

[0:05] Nabakumar Khongbantabam: So overall it can easily take 8 to 12 years from spec to scale.

Original transcript

[0:05] Hello everyone. I'm Navakumar, a product [0:08] manager at Microsoft helping drive the [0:10] development of ML video codec for teams. [0:13] We have been working with a research [0:16] team and our development team to [0:19] productize ML video codec to ship in [0:22] teams. [0:23] Let me begin by introducing video codec. [0:26] Most people have already experienced it. [0:29] If you have streamed a video recently [0:32] such as in a conference call, you have [0:35] already experienced it. It helps reduce [0:37] the huge amount of data that comes from [0:40] the source video such as a camera to [0:43] manageable size which can be transmitted [0:46] or stored conveniently. At the time of [0:48] playback, the decoder part of the video [0:51] collect would reproduce the video from [0:53] the compressed stream and try to restore [0:56] the quality to roughly the same level as [0:59] the original source. [1:01] It's never the exact quality because [1:04] there is a loss in video quality during [1:07] the compression process. Because of [1:10] that, many different video codecs have [1:13] different level of compression [1:14] efficiency. Some are better at [1:16] compressing it. Some are less good at [1:20] reducing the bit rate but are better in [1:23] other aspects. [1:25] To illustrate that point, I have a demo [1:28] here comparing ML video codec to one of [1:31] the many popular video codecs available [1:33] currently H.264. Both the streams on the [1:37] left and right are streamed at 130 [1:40] kilobits per second. And you can see how [1:44] MLVC reproduces the video with much [1:47] higher quality compared to the right [1:50] video reproduced by H.264. [1:54] When we talk about conventional codecs, [1:56] they have existed for a very long time [1:58] and every decade or so there's a new [2:01] generation coming up. On the screen [2:03] here, you can see a timeline of Meek [2:06] video codecs that have been developed [2:08] over time. H.264 [2:10] was developed about 20 years ago and we [2:13] still continue to use it. H.265 which is [2:16] next generation and roughly half the bit [2:18] rate came about a decade later and very [2:21] recently H.266 was finalized and it's [2:24] just being protized. So as you can see [2:28] with conventional codex each generation [2:30] can take a long time and this time is [2:33] because of intensive development stages [2:37] involved in creating a new codec. It [2:39] starts with standardization which can [2:41] take several years to formulate the [2:43] exact standard. Then there comes the [2:46] design for silicon implementation and [2:49] the hardware vendors would then produce [2:51] the specific design for the hardware and [2:54] and start shipping the devices. The [2:56] customers would take several years [2:58] further to adopt and start using it. So [3:00] overall it can easily take 8 to 12 years [3:03] from spec to scale. [3:07] In this talk I would like to cover four [3:09] areas. Firstly, I'd like to tell a bit [3:12] about this noble approach to video [3:14] coding. We'll obviously be talking about [3:17] ML video codec and neural video codex in [3:20] general. In the second section, I would [3:22] like to highlight two major problems [3:24] that we had when we tried to utilize a [3:27] neural video codec in a real product [3:31] teams in this case. And in the third [3:34] section, I'll describe how we overcame [3:37] those problems and eventually managed to [3:40] productize the codec. [3:42] And then in the last section, I'd like [3:44] to share the results from our latest [3:47] evaluation of MLB video codec and [3:50] various resources available in [3:52] opensource for collaborating and [3:56] working together if you are into open [3:59] source development. [4:01] So the approach that we are taking here [4:05] involves utilizing the neural processing [4:08] units which are pieces of hardware [4:11] available in recent consumer devices, [4:14] laptops, mobile phones. They all now [4:16] come with the NPU device on board. This [4:19] allows us to deploy neural video codecs [4:22] on those devices without having to [4:24] require expensive and powerful GPUs. So [4:29] in this development we are essentially [4:32] trying to see how we can run ML video [4:35] codec for customers on their general [4:38] consumer devices and the models are [4:40] trained end to end [4:42] to reconstruct the quality. So in that [4:46] sense it allows us the flexibility of [4:48] ensuring that we can train the model [4:50] specific to our use cases. There are [4:53] several advantages to using NPU compared [4:56] to dedicated silicon that conventional [4:59] video codex use. [5:02] The first big advantage is that neural [5:04] codecs are generally better in [5:06] compressing the video which results in [5:08] better video quality or less bandwidth [5:11] whichever way you like to take advantage [5:13] of it. As an example, ML video codec [5:17] uses 65% less bit rate than H.65 as I [5:21] demonstrated in previous example that [5:24] was comparing with H.264 where overall [5:26] it has better compression efficiency. [5:29] The second big advantage is that we can [5:32] deploy it much faster when we have an [5:34] update of the model. We can ship it in a [5:37] new update for the application and it [5:39] will be nearly instantaneously available [5:41] to all the devices. As an example, MLVC [5:46] now runs on old copilot plus species [5:49] that were shipped recently. [5:51] The third big advantage is that it [5:55] provides a much wider coverage. This is [5:58] because we can make it run on older [6:00] hardware as well. As an example, MLVC [6:03] runs on Apple M1 and above devices. So, [6:07] it helps us reach more customers. more [6:10] consumers can benefit from their older [6:12] devices and overall providing a wider [6:14] coverage. [6:17] Neural video codex is not a new thing. [6:20] It's been going on in research for [6:23] several years already. Our own Microsoft [6:26] research has produced a very famous [6:28] perhaps the best neural video codec so [6:31] far in the literature. It's called DCVC [6:34] codec and we have had several [6:36] generations of it already. The latest [6:38] being the CVC RT which stands for [6:41] realtime version of DCVC. It [6:44] incorporates some really important [6:45] features such as rate control [6:47] adaptability for NPU devices and much [6:51] more efficient compared to previous [6:52] generations. [6:54] However, we still needed more work to [6:56] make it reliably work on an actual [6:59] product. And that's what I would like to [7:02] cover in the next section. I'll be [7:04] describing two very hard problems that [7:06] we came across when we tried to [7:09] incorporate DCBCRT [7:12] in actual application. [7:15] The first problem is related to the [7:18] exact processing requirement for encoder [7:20] and decoder to work correctly. the [7:22] neural network codecs and the entropy [7:25] coding part there's entropy model that [7:28] needs to match exactly between the [7:30] encoder and decoder and that has been a [7:32] big challenge for researchers in general [7:36] many researchers try to solve that by [7:39] explicitly achieving bit exactness [7:43] but we feel that it's very difficult to [7:46] achieve in a very heterogeneous [7:48] environment like we have in teams where [7:51] we have a mix of different vendors and [7:52] different NPU types. So that's one big [7:55] problem we hit earlier. The second issue [7:58] is more straightforward. [8:00] The compute required for the neural [8:03] video codec is much more higher than [8:06] what's generally available on consumer [8:08] devices. Research has been using [8:12] powerful GPUs [8:14] for their their development. However, [8:16] for actual product use, we need to [8:20] consider how we can run it on regular [8:24] consumer laptops for instance. So that's [8:26] the other part that we have been [8:28] solving. Now let me explain the big [8:31] exactness issue why it's such a problem. [8:35] It's actually preventing us from [8:37] crossplatform compatibility when we [8:39] encode from encode on one platform and [8:41] try to decode it on the other platform. [8:43] Here in this example you can see one [8:45] frame was encoded using the CBC RT on [8:48] Apple M3. When we attempted to decode it [8:51] on the lunar lake device it completely [8:53] failed and that is because of the issue [8:56] that I described earlier. The entropy [8:58] model between the incor and decor needs [9:01] to exactly reproduce the indexes which [9:04] which is very difficult when the two [9:06] platforms are processing floatingoint [9:10] operations or the NPU operations [9:13] differently. [9:17] Next, I'd like to tell a bit about how [9:19] we overcome these problems [9:22] both solving the interoperability issue [9:25] as well as improving the performance so [9:28] that we can actually use it in a [9:29] real-time application like teams. [9:33] So in order to overcome the decoding [9:36] failure I described previously, our team [9:39] has developed a workaround which [9:41] involves sending the scale indices [9:44] directly to the decoder instead of [9:46] relying on inferencing it from the [9:48] entropy model. This obviously sidesteps [9:51] the bit exact requirements and simply [9:54] uses the indices sent over the year. [9:57] However, it also reduces the compression [10:00] efficiency because we're now sending [10:02] extra bits, but that's a drawback to [10:04] make the decoding process more stable [10:06] across multiple platforms. Although this [10:09] prevents the decoder from failing [10:11] completely, it however doesn't prevent [10:15] drifting the decoder over time because [10:18] of the floating point operations and the [10:20] ether that accumulates in the recurrent [10:23] neural network part of the model. So [10:27] in this slide we can see the example of [10:30] a clip which started out very nicely [10:32] decoding and then over time the model [10:34] would drift resulting in failure of the [10:37] decoding. To get around this issue, our [10:40] team has implemented special handling [10:43] such as long-term reference recovery. So [10:46] occasionally we would tie up frames to [10:48] long-term references. In Edison, we are [10:51] introducing proactively iframes resets [10:54] which is also something that happens in [10:56] conventional video codex as well. So [10:58] nothing new there. Reasonally we try to [11:01] use more stable operators like relu or [11:04] leaky relu and replace the more [11:07] nonlinear operators that different npu [11:11] apparently implement differently. So by [11:14] adjusting [11:16] the model how we implement it for use in [11:21] teams we are able to make it work. [11:25] Here's a example clip demonstrating MLVc [11:29] with all the enhancements that I [11:30] mentioned earlier. Now comparing with [11:32] H.265 correct both the streams are [11:35] running at 200 kilobits per second and [11:38] you can see how the quality compares [11:40] between the two. On the left you can see [11:42] the video is much sharper. The face is [11:44] clear. Background is clear. On the right [11:47] you can observe the noise introduced [11:49] during the decoding process. So overall [11:52] despite all the enhancements and [11:55] increase in the bit rate because of the [11:56] additional overhead we are sending it is [11:59] still significantly better than H.265. [12:04] Now the second challenge was to run it [12:07] real time on consumer NPU. This involved [12:10] working closely with hardware vendors [12:13] adopting the model to each NPU with [12:16] specific implementations that help to [12:19] optimize it for that particular [12:21] platform. So through the work we have [12:24] done together with our hardware vendors, [12:26] we have managed to make it run real time [12:29] on Intel Qualcomm as well as on Macs. We [12:33] needed at least 30 fps the frame latency [12:37] no more than 16 millisecond and we have [12:39] been able to achieve that through this [12:40] effort. The screenshot you see here is [12:43] an actual call in our internal testing [12:45] rings. So we have it deployed for a set [12:48] of employees internally who would be [12:51] testing the roll out before we go to [12:54] public preview. [12:57] Next, I would like to share the results [12:59] of what we have achieved, especially [13:02] when it comes to integrating it teams [13:06] and provide you with information on the [13:09] open-source repositories and [13:11] documentation. [13:13] So here we can see the rate distortion [13:17] curves of MLVC and H265. This is a bit [13:21] different from what's been published in [13:23] the paper. paper compares 540p [13:26] resolution between these two codecs but [13:29] here this is how we use in teams [13:31] slightly implemented differently here we [13:33] are using both for 720p resolution for [13:36] MLVC we'll upscale it from 540p [13:39] using good upscaler or we have super [13:41] resolution doing it or it could be lan [13:44] source in in other applications for H265 [13:47] we're using native 720p resolution and [13:50] when comparing the rate distortion [13:51] curves you can see the orange curve is [13:54] higher which indicates that it's [13:55] performing better than the lower curve [13:58] as we see on the screen here. This [14:00] roughly corresponds to about 60% [14:03] reduction in bit rate by MLVC when we [14:06] maintain the same perceptual video [14:08] quality between the two codecs. So I I [14:11] believe the train model for this version [14:13] should be available already in the [14:15] repository if anyone is interested to [14:18] try it out. [14:22] So the evaluation I described earlier it [14:24] was performed using the video [14:27] conferencing data set that we re very [14:30] recently upgraded. So VCD is a data set [14:33] that we published some time ago already [14:35] the first version and that was very [14:37] helpful in evaluating different computer [14:39] vision developments. VCD2 is a newer [14:42] generation a newer upgrade to that data [14:44] set. Here we have many more clips and a [14:47] lot more variety of camera models. We [14:50] are also ensuring that all the original [14:52] source captures are losslessly [14:55] compressed and made available so that [14:57] there's no encoding noise introduced [14:59] when we are sharing it. It's also a much [15:02] larger data set five times larger than [15:05] what we shared previously in VCD1 [15:07] and yeah it's open source and it's [15:10] available in the link provided. [15:13] So then the question is when can people [15:16] try it out in teams? As I mentioned [15:18] previously, we are already at step two [15:21] here. Testing it in our early rings. We [15:23] have developed the model. We worked [15:25] around all the major issues. We polished [15:27] it out for production use. And now we're [15:30] testing it in in our internal rings. [15:32] Early next year in 2027, we hope to ship [15:36] it to public preview and then eventually [15:39] to general users. and it will work on [15:42] all copilot plus PCs from Intel, [15:45] Qualcomm and eventually AMD as well in [15:48] future as well as Apple MacBooks. Then [15:51] we'll aim to expand the supported [15:54] platforms in the later half of next [15:57] year. So overall our aim is to migrate [16:01] the video codec used in teams to our [16:07] ML video codec over time. [16:11] Lastly, we have open sourced all our [16:14] work that I have described previously. [16:18] We have open sourced the codec models [16:21] including the train models. So you can [16:23] directly pick them up and experiment [16:25] with it. All the training information, [16:27] training pipeline, they are also made [16:29] available. We have documentation about [16:31] how to create the training data. We have [16:34] also open sourced the evaluation test [16:36] set VCD2. We have also released the C++ [16:40] codec library that wraps the model and [16:42] makes it really easy for any developer [16:44] to try it out and integrate it in an [16:47] application. Obviously, they all come [16:49] with documentation and various research [16:52] papers that we published. So that can [16:55] also be used to understand all the [16:58] details about the work we have done. [17:01] With that, I would like to conclude my [17:03] talk. Thank you.