Talking head video generation aims to animate a human face in a still image with dynamic poses and expressions using motion information derived from a target-driving video, while maintaining the person's identity in the source image. However, dramatic and complex motions in the driving video cause ambiguous generation, because the still source image cannot provide sufficient appearance information for occluded regions or delicate expression variations, which produces severe artifacts and significantly degrades the generation quality. To tackle this problem, we propose to learn a global facial representation space, and design a novel implicit identity representation conditioned memory compensation network, coined as MCNet, for high-fidelity talking head generation.~Specifically, we devise a network module to learn a unified spatial facial meta-memory bank from all training samples, which can provide rich facial structure and appearance priors to compensate warped source facial features for the generation. Furthermore, we propose an effective query mechanism based on implicit identity representations learned from the discrete keypoints of the source image. It can greatly facilitate the retrieval of more correlated information from the memory bank for the compensation. Extensive experiments demonstrate that MCNet can learn representative and complementary facial memory, and can clearly outperform previous state-of-the-art talking head generation methods on VoxCeleb1 and CelebV datasets. Please check our \href{https://github.com/harlanhong/ICCV2023-MCNET}{Project}.

翻译：说话头视频生成旨在利用目标驱动视频中的运动信息，将静态图像中的人脸以动态姿态和表情进行动画化，同时保持源图像中的人物身份。然而，驱动视频中的剧烈复杂运动会导致生成模糊，因为静态源图像无法为遮挡区域或细微表情变化提供充足的外观信息，从而产生严重伪影并显著降低生成质量。为解决这一问题，我们提出学习全局人脸表征空间，并设计了一种新颖的隐式身份表征条件记忆补偿网络（简称MCNet），用于高保真说话头生成。具体而言，我们设计了一个网络模块，从所有训练样本中学习统一的空域人脸元记忆库，该库可提供丰富的人脸结构及外观先验，以补偿经变形处理的源人脸特征用于生成。此外，我们基于从源图像离散关键点习得的隐式身份表征，提出了一种高效的查询机制。该机制能极大促进从记忆库中检索更相关补偿信息的过程。大量实验表明，MCNet能够学习具有代表性且互补的人脸记忆，并在VoxCeleb1和CelebV数据集上明显优于先前最先进的说话头生成方法。请查阅我们的项目页面：\href{https://github.com/harlanhong/ICCV2023-MCNET}{项目}。