For few-shot learning, it is still a critical challenge to realize photo-realistic face visually dubbing on high-resolution videos. Previous works fail to generate high-fidelity dubbing results. To address the above problem, this paper proposes a Deformation Inpainting Network (DINet) for high-resolution face visually dubbing. Different from previous works relying on multiple up-sample layers to directly generate pixels from latent embeddings, DINet performs spatial deformation on feature maps of reference images to better preserve high-frequency textural details. Specifically, DINet consists of one deformation part and one inpainting part. In the first part, five reference facial images adaptively perform spatial deformation to create deformed feature maps encoding mouth shapes at each frame, in order to align with the input driving audio and also the head poses of the input source images. In the second part, to produce face visually dubbing, a feature decoder is responsible for adaptively incorporating mouth movements from the deformed feature maps and other attributes (i.e., head pose and upper facial expression) from the source feature maps together. Finally, DINet achieves face visually dubbing with rich textural details. We conduct qualitative and quantitative comparisons to validate our DINet on high-resolution videos. The experimental results show that our method outperforms state-of-the-art works.

翻译：针对小样本学习，在高分辨率视频中实现照片级真实感的面部视觉配音仍是一项关键挑战。现有方法未能生成高保真度的配音结果。为解决上述问题，本文提出一种用于高分辨率面部视觉配音的形变修复网络（DINet）。与依赖多个上采样层从潜在嵌入直接生成像素的现有方法不同，DINet对参考图像的特征图进行空间形变，以更好地保留高频纹理细节。具体而言，DINet包含一个形变模块和一个修复模块。在第一部分，五张参考面部图像自适应地进行空间形变，生成编码每帧嘴部形态的形变特征图，从而与输入的驱动音频及源图像头部姿态对齐。在第二部分，为生成面部视觉配音，特征解码器负责将形变特征图中的嘴部运动与源特征图中的其他属性（即头部姿态和上半部分面部表情）自适应融合。最终，DINet实现了具有丰富纹理细节的面部视觉配音。我们在高分辨率视频上通过定性与定量对比验证了DINet的性能。实验结果表明，本方法优于现有最优工作。