Face2Face Reenacts Web Video Faces in Real Time Without Depth Sensors

Face2Face: Real-time Face Capture and Reenactment of RGB Videos (CVPR 2016 Oral) — Matthias Niessner

Matthias Niessner

Retrieving mouth interiors directly from target footage eliminates synthetic artifacts, allowing standard webcams to match the fidelity of depth-assisted and offline rendering pipelines.

AI-assisted summary of Face2Face: Real-time Face Capture and Reenactment of RGB Videos (CVPR 2016 Oral) — original source. Check the original for full context.

Source published:

Key ideas

  • Face2Face introduces real-time facial reenactment operating strictly on monocular RGB input from standard webcams, allowing arbitrary online video footage such as YouTube clips to be manipulated live on a commodity PC. (0:05)
  • The pipeline overcomes synthetic mouth artifacts by retrieving matching mouth interior frames from target training footage and applying 2D warping, rather than relying on geometric teeth models. (0:05)
  • Optical flow cross-validation benchmarks demonstrate low reconstruction error when re-rendering query mouth interiors from candidate training splits. (1:47)
  • Tracking evaluations show the monocular RGB method achieves fidelity competitive with dedicated RGB-D systems like Faceshift while matching the visual quality of complex offline dubbing pipelines in real time. (4:40)

Detailed notes

## Background Published on YouTube by Matthias Niessner on March 17, 2016, this video serves as the CVPR 2016 oral paper presentation for Face2Face. The show notes outline a research-focused demonstration of real-time facial capture and reenactment from monocular RGB video streams on commodity PCs, contextualized within visual effects history alongside mentions of synthetic edit detection efforts.

## TL;DR Face2Face introduces an end-to-end framework capable of real-time facial reenactment using solely monocular RGB webcams and target footage like YouTube videos. The system tracks expressions on both source and target using dense photometric energy minimization, remapping facial geometry through subspace deformation transfer. Instead of relying on artificial geometric teeth proxies, it retrieves and warps real mouth interior frames from target training footage, achieving photo-realistic speech animation. Benchmarks against state-of-the-art offline methods and RGB-D depth trackers demonstrate comparable fidelity and tracking stability while running live on standard consumer hardware.

## Atmosphere & style The presentation carries a crisp, measured, and understated academic tone that contrasts sharply with the startling nature of the visual demonstrations. Watching a researcher pull exaggerated faces on an ordinary webcam while world leaders on pre-recorded clips mirror every smirk and brow raise in real time feels both uncanny and technologically revelatory. The pacing remains clinical and objective, grounding the surreal visual dubbing in rigorous split-screen evaluations, comparative error grids, and quantitative optical flow benchmarks.

## Bulletin - [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) Face2Face introduces real-time facial reenactment operating strictly on monocular RGB input from standard webcams, allowing arbitrary online video footage such as YouTube clips to be manipulated live on a commodity PC. - [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) The pipeline overcomes synthetic mouth artifacts by retrieving matching mouth interior frames from target training footage and applying 2D warping, rather than relying on geometric teeth models. - [01:47](https://www.youtube.com/watch?v=ohmajJTcpNk&t=107s) Optical flow cross-validation benchmarks demonstrate low reconstruction error when re-rendering query mouth interiors from candidate training splits. - [04:40](https://www.youtube.com/watch?v=ohmajJTcpNk&t=280s) Tracking evaluations show the monocular RGB method achieves fidelity competitive with dedicated RGB-D systems like Faceshift while matching the visual quality of complex offline dubbing pipelines in real time.

## Key takeaways - [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) The framework performs real-time facial reenactment using monocular RGB video feeds without requiring specialized depth sensors or RGB-D hardware. - [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) The system can dynamically animate pre-recorded target subjects sourced from arbitrary online videos, including public figures on YouTube. - [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) Source and target facial expressions are tracked using a dense photometric consistency energy minimization approach. - [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) Expression dynamics are mapped onto the target identity model using an efficient subspace deformation transfer algorithm. - [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) Mouth interior resynthesis retrieves matching frames from target video footage based on photometric and temporal similarity rather than rendering synthetic cavity proxies. - [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) Retrieved mouth interiors undergo 2D non-rigid warping to align accurately with target face meshes and real-world scene illumination. - [01:47](https://www.youtube.com/watch?v=ohmajJTcpNk&t=107s) Real-time reenactments demonstrate robust expression transfer across diverse target subjects, extreme poses, and varying lighting conditions. - [01:47](https://www.youtube.com/watch?v=ohmajJTcpNk&t=107s) The mouth synthesis pipeline is quantitatively validated through an optical flow cross-validation benchmark dividing video sequences into candidate and query sets. - [04:40](https://www.youtube.com/watch?v=ohmajJTcpNk&t=280s) The proposed tracker outperforms existing monocular RGB methods like Cao et al. during challenging facial movements. - [04:40](https://www.youtube.com/watch?v=ohmajJTcpNk&t=280s) Tracking accuracy matches depth-based real-time systems such as Faceshift (2014) and Thies et al. using RGB data alone. - [04:40](https://www.youtube.com/watch?v=ohmajJTcpNk&t=280s) The real-time tracker demonstrates geometric alignment competitive with offline methods like Shi et al. that rely on shape-from-shading refinements. - [06:02](https://www.youtube.com/watch?v=ohmajJTcpNk&t=362s) Face2Face matches the visual dubbing fidelity of multi-pass offline systems like Garrido et al. while running interactively at video framerates. - [06:02](https://www.youtube.com/watch?v=ohmajJTcpNk&t=362s) Replacing geometric teeth proxies with retrieved video frames eliminates unnatural mouth boundaries and geometric artifacts present in earlier reenactment architectures.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=ohmajJTcpNk&t=5s) Live Facial Reenactment and Mouth Interior Resynthesis

The demonstration introduces a novel real-time facial reenactment framework capable of operating entirely on RGB data captured from standard commodity webcams. By restricting the required input strictly to monocular color video for both the source driver and the target subject, the system unlocks the ability to manipulate arbitrary pre-recorded footage, such as online YouTube videos, on the fly. In a live laboratory demonstration, a source actor sits in front of a standard webcam on the right, whose facial movements directly drive the target person displayed on the monitor to the left. The presenter highlights the core technical breakthrough that sets this approach apart: “A significant difference to previous methods is the rerendering of the mouth interior.” Rather than relying on simple geometric approximations or synthetic cavity models, the pipeline resynthesizes the target actor's mouth interior by retrieving matching video frames from an initial training sequence based on combined temporal and photometric similarity metrics, producing photo-realistic lip interiors and teeth alignment.

```mermaid flowchart TD accTitle: Face2Face Real-Time Reenactment Pipeline accDescr: Pipeline diagram illustrating the flow from RGB webcam capture and monocular target tracking through deformation transfer, mouth interior retrieval, and final video compositing.

subgraph InputTracking["Tracking & Capture"] SRC["Source Webcam (RGB)"] --> TRK_S["Dense Photometric Tracking"] TGT["Target Video (Monocular RGB)"] --> TRK_T["Model-Based Bundling & Tracking"] end

subgraph Deformation["Reenactment Engine"] TRK_S --> DT["Subspace Deformation Transfer"] TRK_T --> DT DT --> MESH["Modified Target Face Mesh"] end

subgraph Synthesis["Rendering & Blending"] MESH --> MOUTH["Mouth Interior Retrieval & Warping"] MOUTH --> COMP["Photometric Re-rendering & Seamless Composite"] TGT --> COMP end

COMP --> OUT["Final Manipulated Video"] ```

In a close-up analysis of the live footage, the input video stream is inspected side-by-side with the reenacted output. The system tracks the facial states of both actors through a dense photometric energy minimization strategy. Expressions captured from the source actor are mapped to the target model using a specialized subspace deformation transfer technique, yielding a dynamically modified face template tailored to the target subject's unique facial identity. This adapted geometry is subsequently re-rendered directly on top of the original target sequence to substitute the original facial expressions while seamlessly preserving natural skin tones and ambient scene illumination.

- **Sensor Requirements:** Monocular RGB only (commodity webcams and web video streams; no depth sensors or RGB-D hardware required). - **Tracking Core:** Dense photometric consistency energy minimization. - **Expression Transfer:** Novel subspace deformation transfer mapping source expressions onto the target identity model. - **Mouth Handling:** Frame retrieval from target training footage filtered by photometric and temporal consistency, followed by 2D warping.

---

[01:47](https://www.youtube.com/watch?v=ohmajJTcpNk&t=107s) Live Sequences on YouTube Footage and Quantitative Flow Evaluation

The researchers present an extensive series of reenactments performed live on various YouTube video sequences featuring well-known public figures. Across diverse target actors, lighting variations, and head poses, the system reliably tracks the driving actor's mouth movements, eyebrow raises, smirks, and conversational expressions, re-rendering them seamlessly onto the target footage in real time.

To quantitatively evaluate the quality and physical plausibility of the mouth synthesis, the authors implement an optical flow cross-validation benchmark:

> “In order to evaluate our approach, we perform a cross validation based on optical flow.”

In this validation setup, the target video sequence is split into two halves: the mouth interior candidate database is populated exclusively from the first half of the clip, while the second half provides test queries. By comparing the synthesized reenactments against the actual ground-truth frames across the query sequence using dense optical flow metrics, the authors verify that the re-rendering reconstruction error remains remarkably low, validating both the frame selection strategy and the subsequent non-rigid alignment warps.

---

[04:40](https://www.youtube.com/watch?v=ohmajJTcpNk&t=280s) RGB Face Tracking Comparisons with State-of-the-Art

The video highlights the system's underlying RGB face tracking pipeline, contrasting its tracking stability against contemporary state-of-the-art real-time and offline algorithms. First, tracking performance is compared against the RGB method of Cao et al. and the RGB-D approach of Thies et al. While Thies et al. benefit from dedicated depth sensor streams, the proposed method matches their tracking robustness and fidelity using monocular RGB input alone, noticeably outperforming the monocular tracker of Cao et al. under challenging facial articulations.

``` +---------------------+-------------------+---------------------+-------------------------+ | Method | Input Modality | Execution Mode | Geometric Refinement | +---------------------+-------------------+---------------------+-------------------------+ | Proposed Method | Monocular RGB | Real-time | Dense photometric model | | Cao et al. | Monocular RGB | Real-time | Linear regression/blend | | Thies et al. | RGB-D (Depth) | Real-time | Depth-assisted tracking | | Faceshift (2014) | RGB-D (Depth) | Real-time | Depth ICP + RGB blend | | Shi et al. | Monocular RGB | Offline | Shape from shading cues | +---------------------+-------------------+---------------------+-------------------------+ ```

A subsequent side-by-side evaluation against commercial software Faceshift (2014) reinforces this capability. The presenter observes: “Although our method is RGB only, we achieve similar tracking quality.” In addition to real-time solutions, the pipeline is evaluated against the offline tracking framework of Shi et al. Even though Shi et al. operate without interactive runtime constraints and incorporate complex geometric shape-from-shading refinements, the proposed real-time tracker demonstrates competitive geometric expression alignment across rapid and subtle facial transitions.

---

[06:02](https://www.youtube.com/watch?v=ohmajJTcpNk&t=362s) Comparison to Prior Reenactment Systems

The presentation concludes by benchmarking Face2Face against existing facial reenactment architectures. One scenario examines a video translation application, where a live translator animates another individual speaking in a different context. The presenter notes the crucial runtime difference: “Note that our approach runs in real time while Garrido et al. works offline.” While Garrido et al. require extensive multi-pass offline computation to dub and warp target faces, Face2Face achieves comparable visual realism instantaneously.

Finally, the method is compared to the real-time RGB-D reenactment pipeline of Thies et al. While both systems deliver comparable global facial expression transfer, Thies et al. rely on an explicit geometric proxy for the teeth and inner cavity. The presenter emphasizes the visual artifacts that result from this parametric model:

> “However, note that Thies et al. uses a geometric teeth proxy which leads to artificially shaped mouth regions.”

By discarding rigid geometric teeth proxies in favor of an image-based retrieval and blending framework drawn directly from original target footage, Face2Face eliminates synthetic boundaries and achieves natural oral cavity synthesis during continuous speech.

## Notable quotes > [06:02](https://www.youtube.com/watch?v=ohmajJTcpNk&t=362s) Thies et al. uses a geometric teeth proxy which leads to artificially shaped mouth regions. — Presenter

## Mentioned Face2Face, CVPR 2016, Matthias Niessner, YouTube, The Lord of the Rings, The Curious Case of Benjamin Button, Cao et al., Thies et al., Faceshift, Shi et al., Garrido et al.

## Video description CVPR 2016 Paper Video (Oral) Project Page: http://niessnerlab.org/projects/thies2016face.html

IMPORTANT NOTE: This demo video is purely research-focused and we would like to clarify the goals and intent of our work. Our aim is to demonstrate the capabilities of modern computer vision and graphics technology, and convey it in an approachable and fun way. We want to emphasize that computer-generated videos have been part in feature-film movies for over 30 years. Virtually every high-end movie production contains a significant percentage of synthetically-generated content (from Lord of the Rings to Benjamin Button). These results are hard to distinguish from reality and it often goes unnoticed that the content is not real. The novelty and contribution of our work is that we can edit pre-recorded videos in real-time on a commodity PC. Please also note that our efforts include the detection of edits in video footage in order to verify a clip’s authenticity. For additional information, we refer to our project website (see above). Hopefully, you enjoyed watching our video, and we hope to provide a positive takeaway :)

Paper Abstract We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.

[Source](https://www.youtube.com/watch?v=ohmajJTcpNk)

中文简报

Face2Face无需深度传感器即可实时重现网络视频中的人脸 直接从目标视频素材中检索口腔内部信息消除了合成瑕疵，使标准网络摄像头能够达到深度辅助和离线渲染流程的逼真度。 Face2Face推出了一种仅使用单目RGB网络摄像头和YouTube视频等目标素材即可实现实时面部重现的端到端框架。该系统利用稠密光度能量最小化跟踪源和目标上的表情，并通过子空间形变传递重新映射面部几何形态。它没有依赖人造几何牙齿代理模型，而是从目标训练素材中检索并变形真实的口腔内部帧，从而实现了逼真的言语动画。与最先进的离线方法和RGB-D深度跟踪器的基准对比表明，该方法在标准消费级硬件上实时运行的同时，展现出了相当的逼真度和跟踪稳定性。 Face2Face引入了仅依靠标准网络摄像头的单目RGB输入运行的实时面部重现，允许在普通PC上实时操纵YouTube片段等任意在线视频素材。 (0:05) 该流程克服了合成口腔瑕疵，它从目标训练素材中检索匹配的口腔内部帧并应用2D变形，而不是依赖几何牙齿模型。 (0:05) 光流交叉验证基准表明，在从候选训练拆分中重新渲染查询口腔内部时，重建误差较低。 (1:47) 跟踪评估显示，该单目RGB方法达到了与Faceshift等专用RGB-D系统相当的逼真度，同时在实时运行中匹敌复杂的离线配音流程的视觉质量。 (4:40)

Source quotes

[6:02] Presenter: Thies et al. uses a geometric teeth proxy which leads to artificially shaped mouth regions.

Original transcript

[0:05] We present a novel real-time facial [0:06] reenactment method that works with any [0:08] commodity webcam. Since our method only [0:11] uses RGB data for both the source and [0:13] target actor, we are able to manipulate [0:15] YouTube videos in real time. Here we [0:18] demonstrate our method in a live setup. [0:20] On the right, a source actor is captured [0:22] with a standard webcam. This input [0:24] drives the animation of the face in the [0:26] video shown on the monitor. to the [0:28] left. A significant difference to [0:30] previous methods is the rerendering of [0:32] the mouth interior. To this end, we [0:34] reynthesize the mouth interior of the [0:36] target actor using video footage from [0:38] the training sequence based on temporal [0:39] and phototric [0:41] similarity. As we can see, we are able [0:44] to generate a realistic and convincing [0:45] reenactment [0:47] result. Here we show a close-up of the [0:50] footage from the previous live [0:52] reenactment. The input video stream is [0:54] shown on the left. Note that the target [0:56] actor is rerendered in a neutral [0:58] pose. On the right, we can see the final [1:01] output of our [1:12] method. Our system reconstructs and [1:15] tracks both source and target actors [1:16] using a dense phototric energy [1:19] minimization. Using a novel subspace [1:21] deformation transfer technique, we [1:22] transfer the expressions from the source [1:24] to the target actor. This allows us to [1:27] obtain a modified face template of the [1:28] target actor according to the [1:30] expressions of the source [1:31] actor. We now rerender the modified face [1:34] on top of the target sequence in order [1:36] to replace the original facial [1:42] expressions. [1:47] Here we show additional live sequences [1:50] where we reenact various YouTube videos. [4:18] In order to evaluate our approach, we [4:20] perform a cross validation based on [4:21] optical flow. To this end, we retrieve [4:24] mouth interiors from the first half of [4:26] the video. The second half is used for [4:28] evaluation [4:29] queries. As we can see, our rerendering [4:32] error is very [4:34] low. [4:40] Our method introduces a new RGB face [4:43] tracking pipeline which we compare [4:45] against state-of-the-art real-time face [4:46] tracking [4:48] methods. Here we show a comparison [4:51] against cow at all and TE's at all. Note [4:54] that TATL is based on RGBD data whereas [4:56] cow at all and our method require only [4:58] RGB input. [5:18] Here we show another tracking comparison [5:21] to phase shift 2014 which relies on RGBD [5:25] data. Although our method is RGB only, [5:28] we achieve similar tracking quality. [5:45] In contrast to real-time tracking [5:46] methods, we also compare against the [5:48] offline tracking algorithm of she at [5:50] all. Note that she atall perform [5:52] additional geometric refinement using [5:54] shading cues. [6:02] We now compare our approach against [6:04] previous reenactment approaches. Here we [6:07] show the scenario of a translator [6:08] animating another [6:10] person. Note that our approach runs in [6:12] real time while Gerrito at all works [6:16] offline. Here we show a comparison to [6:18] TE's at all who rely on RGBD [6:21] data. Both methods produce similar [6:23] reenactment results. However, note that [6:25] tease at all uses a geometric teeth [6:27] proxy which leads to artificially shaped [6:29] mouth regions. Thank you for watching.