The task of lip synchronization (lip-sync) seeks to match the lips of human faces with different audio. It has various applications in the film industry as well as for creating virtual avatars and for video conferencing. This is a challenging problem as one needs to simultaneously introduce detailed, realistic lip movements while preserving the identity, pose, emotions, and image quality. Many of the previous methods trying to solve this problem suffer from image quality degradation due to a lack of complete contextual information. In this paper, we present Diff2Lip, an audio-conditioned diffusion-based model which is able to do lip synchronization in-the-wild while preserving these qualities. We train our model on Voxceleb2, a video dataset containing in-the-wild talking face videos. Extensive studies show that our method outperforms popular methods like Wav2Lip and PC-AVS in Fr\'echet inception distance (FID) metric and Mean Opinion Scores (MOS) of the users. We show results on both reconstruction (same audio-video inputs) as well as cross (different audio-video inputs) settings on Voxceleb2 and LRW datasets. Video results and code can be accessed from our project page ( https://soumik-kanad.github.io/diff2lip ).

翻译：唇形同步（lip-sync）任务旨在将人脸嘴唇动作与不同音频进行匹配，在电影工业、虚拟化身创建及视频会议等领域具有广泛应用。该任务具有挑战性，因为需要在保持身份特征、姿态、情绪和图像质量的同时，生成细致逼真的唇部动态。现有方法常因缺乏完整上下文信息导致图像质量退化。本文提出Diff2Lip——一种基于音频条件的扩散模型，能够在自然场景下实现唇形同步的同时保留上述质量特性。我们使用VoxCeleb2（包含自然场景说话人脸视频的数据集）训练模型。大量研究表明，本方法在Fr´echet初始距离（FID）指标和用户平均意见分（MOS）上均优于Wav2Lip、PC-AVS等主流方法。我们在VoxCeleb2和LRW数据集上展示了重建（输如音视频一致）和交叉（输如音视频不同）两种设置的结果。视频结果与代码可通过项目页面获取（ https://soumik-kanad.github.io/diff2lip ）。