Given one reference facial image and a piece of speech as input, talking head generation aims to synthesize a realistic-looking talking head video. However, generating a lip-synchronized video with natural head movements is challenging. The same speech clip can generate multiple possible lip and head movements, that is, there is no one-to-one mapping relationship between them. To overcome this problem, we propose a Speech Feature Extractor (SFE) based on memory-augmented self-supervised contrastive learning, which introduces the memory module to store multiple different speech mapping results. In addition, we introduce the Mixed Density Networks (MDN) into the landmark regression task to generate multiple predicted facial landmarks. Extensive qualitative and quantitative experiments show that the quality of our facial animation is significantly superior to that of the state-of-the-art (SOTA). The code has been released at https://github.com/Yaxinzhao97/MACL.git.

翻译：给定一张参考人脸图像和一段语音作为输入，说话人脸生成旨在合成一段逼真的说话人脸视频。然而，生成具有自然头部运动的唇形同步视频颇具挑战性。同一段语音片段可生成多种可能的唇部和头部运动，即二者之间不存在一一映射关系。为解决此问题，我们提出了一种基于记忆增强自监督对比学习的语音特征提取器（SFE），该提取器引入记忆模块以存储多种不同的语音映射结果。此外，我们在地标回归任务中引入混合密度网络（MDN），以生成多个预测人脸地标。大量定性和定量实验表明，我们的人脸动画质量显著优于当前最先进（SOTA）方法。代码已发布于https://github.com/Yaxinzhao97/MACL.git。