Audio-driven talking face generation is the task of creating a lip-synchronized, realistic face video from given audio and reference frames. This involves two major challenges: overall visual quality of generated images on the one hand, and audio-visual synchronization of the mouth part on the other hand. In this paper, we start by identifying several problematic aspects of synchronization methods in recent audio-driven talking face generation approaches. Specifically, this involves unintended flow of lip and pose information from the reference to the generated image, as well as instabilities during model training. Subsequently, we propose various techniques for obviating these issues: First, a silent-lip reference image generator prevents leaking of lips from the reference to the generated image. Second, an adaptive triplet loss handles the pose leaking problem. Finally, we propose a stabilized formulation of synchronization loss, circumventing aforementioned training instabilities while additionally further alleviating the lip leaking issue. Combining the individual improvements, we present state-of-the art performance on LRS2 and LRW in both synchronization and visual quality. We further validate our design in various ablation experiments, confirming the individual contributions as well as their complementary effects.

翻译：音频驱动说话人脸生成是一项任务，旨在从给定音频和参考帧中创建唇形同步且逼真的人脸视频。这涉及两大挑战：一方面生成图像的整体视觉质量，另一方面嘴部区域的视听同步。在本文中，我们首先识别了近期音频驱动说话人脸生成方法中同步方法的若干问题。具体而言，这包括唇部和姿态信息从参考帧意外流向生成图像，以及模型训练过程中的不稳定性。随后，我们提出了多种规避这些问题的技术：首先，一个静默唇部参考图像生成器可防止唇部信息从参考帧泄漏到生成图像；其次，一种自适应三元组损失处理姿态泄漏问题；最后，我们提出了一种同步损失的稳定化公式，该公式既能规避前述训练不稳定性，还能进一步缓解唇部泄漏问题。结合各项改进，我们在LRS2和LRW上展示了同步质量和视觉质量方面的最先进性能。我们进一步通过多种消融实验验证了设计，确认了各项贡献及其互补效应。