Talk3D: High-Fidelity Talking Portrait Synthesis via Personalized 3D Generative Prior

Introduction

This paper presents a novel audio-driven talking head synthesis framework called Talk3D. The task of audio-driven talking portrait synthesis aims to generate a video of a human face with lip movements synchronized to the input audio. This poses several challenges, such as accurately capturing phonemes, generating realistic facial dynamics, and achieving high-fidelity facial image synthesis.

Prior approaches have used 2D generative models to reconstruct lip motion, but they often struggle with controlling the head pose. Recent studies have integrated neural radiance fields (NeRF) to leverage its capability in rendering realistic and multi-view consistent images. However, directly constructing dynamic facial NeRF from monocular videos remains challenging due to the lack of diverse head poses and 3D information.

To address these limitations, the paper proposes leveraging 3D-aware generative priors, which can generate high-fidelity 3D-aware images. The key strategies are: 1) using a 3D-aware GAN prior for realistic geometry, and 2) directly editing the NeRF space beyond the GAN latent space. Specifically, the model predicts a deltaplane offset to the personalized triplane, which represents the precise lip movements conditioned on the input audio. Additionally, the architecture employs an attention-based module and accommodates extra features to disentangle local facial variations, improving lip-sync accuracy.

The paper claims that the proposed Talk3D framework demonstrates state-of-the-art talking head generation results through extensive experiments.

Related Work

The text discusses the challenges of audio-driven talking portrait synthesis, which aims to generate lip motions synchronized with audio while maintaining realistic facial structure and head poses. Early deep learning methods used 2D generative adversarial networks (GANs) to synthesize audio-synchronized lip motions but lacked control over head poses. Subsequent works attempted pose control using facial landmarks and 3D facial models, but this led to information loss. AD-NeRF first incorporated neural radiance fields (NeRF) to address the challenges of 3D head structure. Later works like SSP-NeRF, RAD-NeRF, and ER-NeRF made improvements in visual quality and efficiency using techniques like semantic sampling and Instant-NGP. However, these NeRF-based methods struggle to achieve multi-view consistency and realistic geometries, as they learn facial representations from only a single monocular video.

The text also discusses extending NeRF for generating images with multi-view consistency, using techniques like conditioning NeRF on sampled random vectors or semantic codes. Some works enabled high-resolution image synthesis by using super-resolution modules or hybrid 3D representations that integrate NeRF with explicit representations. EG3D, which is used as the base representation in this work, has achieved state-of-the-art image generation quality while maintaining 3D consistency.

Lastly, the text mentions how subsequent works have taken advantage of EG3D's efficient representation and diverse latent space for 3D facial reconstruction, using techniques like GAN inversion and deformation fields to enable facial animation from the triplane representation.

Methodology

This section describes the components of the proposed Talk3D framework for audio-driven high-fidelity talking portrait synthesis. The key points are:

• Talk3D uses a personalized generator based on EG3D to produce an identity triplane 𝐏ID. This is done through a fine-tuning strategy to adapt the 3D-aware GAN to a specific person.

• An audio-guided attention U-Net is used to predict a deltaplane Δ⁢𝐏n that represents the dynamic scene variation from the audio input. This deltaplane is combined with the identity triplane to form the final feature plane 𝐏n' that is input to the neural renderer.

• The attention mechanism in the U-Net allows it to effectively capture localized facial dynamics from the audio input. Additional conditioning tokens related to head pose, eye movements, and background are also incorporated.

• The model is trained using a combination of reconstruction, identity, and lip synchronization loss terms to generate high-quality talking portraits that match the input audio.

• Quantitative results show Talk3D outperforming prior NeRF-based methods in terms of image fidelity and lip sync accuracy across different head poses.

Experiments

The paper discusses the experimental settings and evaluation of their proposed audio-driven talking head synthesis method, Talk3D. Key points:

Dataset: The authors use datasets from previous NeRF-based works, comprising person-centric videos with audio tracks. They also use a pre-trained Wav2Vec model to extract audio features.

Comparison baselines: The method is compared to 2D talking head methods like Wav2Lip and PC-AVS, as well as NeRF-based models like AD-NeRF, RAD-NeRF, and ER-NeRF. Additional comparisons to GeneFace and HFA-GP are provided in the supplementary material.

Quantitative evaluation: Three settings are used - novel-view synthesis, self-driven, and cross-driven. Metrics include PSNR, SSIM, LPIPS, FID, landmark distance, lip sync score, and action unit error. The authors also use an identity similarity metric.

Qualitative evaluation: The results show Talk3D outperforming previous NeRF-based methods in maintaining performance across diverse viewing angles and providing robust and accurate results in self-driven and cross-driven settings.

User study: A user study with 31 participants was conducted, which showed Talk3D outperforming other methods in terms of lip-sync accuracy, image quality, and video realism.

Ablation study: The ablation study validates the importance of the sync loss function and the augmented feature tokens used by Talk3D.

Conclusion

The paper introduces Talk3D, a novel framework for high-fidelity 3D talking head synthesis. Talk3D incorporates a 3D-aware GAN prior and a region-aware motion model to produce realistic 3D talking head avatars with accurate lip movements and explicit rendering viewpoint control.

The framework uses a personalized generator fine-tuned with the VIVE3D framework, which enables the synthesis of 3D-aware talking heads with realistic geometry. Additionally, the proposed audio-guided attention U-Net architecture enhances the disentanglement of local variations within the image frames, such as background, torso, and eye movements.

Extensive experiments demonstrate that the Talk3D model not only generates accurate lip movements synchronized with the input audio, but also enables rendering from novel viewpoints, addressing limitations of previous state-of-the-art approaches. The authors anticipate that this work will significantly impact digital media experiences, virtual interactions, and find applications in film-making, virtual avatars, and video conferencing.

The supplementary material provides implementation details, additional experiments, and further analyses, including ablation studies, attention map visualizations, and semantic manipulation of the generated portraits. The limitations and ethical considerations of the research are also briefly discussed.

Appendix 0.A Additional Implementation Details

The paper describes the pre-processing and feature extraction steps used in their model:

Pre-processing: The method follows the same image cropping approach as VIVE3D [22], which detects 6 facial landmarks in each video frame using an off-the-shelf detector, applies Gaussian smoothing to stabilize the landmarks over time, and calculates affine transformation matrices to crop the images. The cropping boundaries are slightly wider than those used in EG3D [8].

Augmenting feature extraction: For audio features, the method uses a pre-trained Wav2Vec [2] model followed by 1D convolutional layers. Low-dimensional augmenting features like eye movement, head rotation, and landmarks are upsampled using positional encodings and further encoded with multilayer perceptrons (MLPs). Each feature is encoded into a 64-dimensional token.

Network architecture: The deltaplane predictor encodes the 256-resolution triplane input into a 32-resolution feature map. The cross-attention layer takes the flattened image feature and conditioning tokens, and predicts a low-resolution feature map using learned query, key, and value representations.

The paper also mentions that the original super-resolution module in EG3D [8] was replaced with GFPGAN [57] to enhance rendering quality, but all quantitative evaluations were performed using the original EG3D super-resolution module for a fair comparison.

Appendix 0.B Additional Results and Comparisons

The provided text discusses the performance of the Talk3D model in generating images from extreme viewpoints and comparing it to previous NeRF-based methods. Key points:

• Talk3D is able to generate images and depth maps for the entire synthesized portrait, including the background, unlike previous NeRF methods that lack depth information outside the head/torso region.

• The authors also demonstrate the generalizability of Talk3D by conducting experiments on additional in-the-wild datasets from HDTF.

• Comparisons are made to two related works, HFA-GP and GeneFace, though the authors note limitations in accurately reproducing the results of HFA-GP.

• Overall, the results show Talk3D outperforming these prior NeRF-based approaches across various quantitative metrics for novel view synthesis and depth estimation.

Appendix 0.C Further Analysis

The paper provides several analyses and ablation studies to demonstrate the efficacy of the proposed attention-based network and triplane representation.

0.C.1 Analysis of attention: The attention maps show that the conditioning tokens successfully disentangle the local movements of the low-resolution feature map. Angles and landmarks capture different attention patterns, as angles are related to torso movement while landmarks are suitable for capturing background motion.

0.C.2 Analysis of triplane: The visualizations of the generated images and their corresponding triplanes (identity plane and deltaplane) confirm the method's ability to precisely manipulate specific regions like the lips, eyes, torso, and background.

0.C.3 Ablation studies:

1. Importance of each token selection: The ablation study demonstrates the crucial role of each feature token in controlling the corresponding regions, such as eye movement, torso, and background.
2. Deltaplane predictor: The evaluation highlights the importance of the design choices in the deltaplane predictor for optimizing image quality and lip-synchronization accuracy.
3. Effect of the personalized generator: The incorporation of the personalized generator leads to improved image quality, as shown by the higher performance metrics and the reduced noise around the nose and eye regions.

Appendix 0.D Facial Editing

This section introduces a new feature of the Talk3D model that distinguishes it from other NeRF-based methods - facial attribute manipulation. Talk3D is built on a pre-trained generative model, EG3D, which has a rich and diverse latent space that enables semantic editing by adding predefined style vectors to the input latent code.

The researchers exploit InterFaceGAN to find style vectors that represent semantic editing directions within the EG3D latent space. Since Talk3D directly predicts the triplane representation instead of a latent code, the researchers slightly alter the InterFaceGAN methodology by replacing the identity triplane with the edited triplane.

Specifically, the researchers construct the edited triplane by adding the editing style vector to the identity latent code and feeding it into the personalized generator. They then replace the identity triplane with the edited triplane to generate the edited image.

The results, shown in Fig. 15, demonstrate consistent manipulation of attributes like age and hair length without disrupting lip synchronization.

Appendix 0.E Supplementary Video

The supplementary video provides a comprehensive visualization of the effectiveness of the proposed method for talking facial video synthesis. The video showcases the facial animations generated by the method and includes detailed comparisons with other relevant techniques. It also demonstrates the versatility of the approach across different languages and its robustness in rendering performance at extreme viewpoints. Additionally, the video includes an ablation study that examines the individual contributions of the components within the methodology.

Appendix 0.F Broader Impact

The text discusses the ethical considerations and limitations of the Talk3D system, which generates highly realistic talking portrait videos with accurate lip-audio synchronization.

The key ethical consideration is the potential for misuse of such technology to create "deepfake" content that is challenging for individuals to distinguish from authentic videos. To address this, the authors emphasize the importance of informing users about the authenticity of videos and supporting the enhancement of deepfake detection mechanisms. They also advocate for protective measures, such as digital watermarks, and the need for regulatory frameworks to prevent unintended negative consequences.

In terms of limitations, while Talk3D excels at high-fidelity talking portrait synthesis, especially from extreme viewpoints, it does not generalize well outside of photorealistic human faces like some other talking face synthesis works. The system's reliance on GAN inversion also introduces technical complexities in data preparation, such as the need for precise alignment and cropping of video frames. Future development will focus on overcoming these limitations to increase the method's adaptability and ensure consistent performance across a wider range of training data.