Despite numerous completed studies, achieving high fidelity talking face generation with highly synchronized lip movements corresponding to arbitrary audio remains a significant challenge in the field. The shortcomings of published studies continue to confuse many researchers. This paper introduces G4G, a generic framework for high fidelity talking face generation with fine-grained intra-modal alignment. G4G can reenact the high fidelity of original video while producing highly synchronized lip movements regardless of given audio tones or volumes. The key to G4G's success is the use of a diagonal matrix to enhance the ordinary alignment of audio-image intra-modal features, which significantly increases the comparative learning between positive and negative samples. Additionally, a multi-scaled supervision module is introduced to comprehensively reenact the perceptional fidelity of original video across the facial region while emphasizing the synchronization of lip movements and the input audio. A fusion network is then used to further fuse the facial region and the rest. Our experimental results demonstrate significant achievements in reenactment of original video quality as well as highly synchronized talking lips. G4G is an outperforming generic framework that can produce talking videos competitively closer to ground truth level than current state-of-the-art methods.

翻译：尽管已有大量研究，但在任意音频下实现高度同步唇部运动的高保真说话人脸生成仍是该领域的一项重大挑战。现有研究的不足持续困扰着众多研究者。本文提出G4G，一种基于细粒度模态内对齐的高保真说话人脸生成通用框架。无论给定音频的音调或音量如何，G4G均能在生成高度同步唇部运动的同时，再现原始视频的高保真度。G4G成功的关键在于采用对角矩阵增强音频-图像模态内特征的常规对齐，显著提升了正负样本间的对比学习效果。此外，本文引入多尺度监督模块，在强调唇部运动与输入音频同步性的同时，全面再现面部区域原始视频的感知保真度，并通过融合网络对面部区域与其余部分进行进一步融合。实验结果表明，该方法在原始视频质量再现及高度同步唇部运动方面均取得显著成果。G4G作为一种性能优越的通用框架，生成的说话视频在接近真实水平方面优于当前最先进方法。