Despite numerous completed studies, achieving high fidelity talking face generation with highly synchronized lip movements corresponding to arbitrary audio remains a significant challenge in the field. The shortcomings of published studies continue to confuse many researchers. This paper introduces G4G, a generic framework for high fidelity talking face generation with fine-grained intra-modal alignment. G4G can reenact the high fidelity of original video while producing highly synchronized lip movements regardless of given audio tones or volumes. The key to G4G's success is the use of a diagonal matrix to enhance the ordinary alignment of audio-image intra-modal features, which significantly increases the comparative learning between positive and negative samples. Additionally, a multi-scaled supervision module is introduced to comprehensively reenact the perceptional fidelity of original video across the facial region while emphasizing the synchronization of lip movements and the input audio. A fusion network is then used to further fuse the facial region and the rest. Our experimental results demonstrate significant achievements in reenactment of original video quality as well as highly synchronized talking lips. G4G is an outperforming generic framework that can produce talking videos competitively closer to ground truth level than current state-of-the-art methods.

翻译：尽管已有大量研究，但实现与任意音频高度同步的唇部运动并生成高保真说话人脸仍是该领域的重大挑战。现有研究成果的局限性仍困扰着众多研究者。本文提出G4G——一种基于细粒度模态内对齐的高保真说话人脸生成通用框架。无论输入音频的音调或音量如何，G4G均能重建原始视频的高保真度，同时生成高度同步的唇部运动。其成功关键在于使用对角矩阵增强音频-图像模态内特征的普通对齐，显著提升正负样本间的对比学习效果。此外，本文引入多尺度监督模块，在全身网络全面重建面部感知保真度的同时，强调唇部运动与输入音频的同步性；随后通过融合网络对面部区域与其余部分进行进一步融合。实验结果表明，本方法在原始视频质量重建和高度同步的说话唇部运动两方面均取得了显著成效。G4G作为一种性能卓越的通用框架，能够生成比当前最先进方法更接近真实水平的说话视频。