Generating talking person portraits with arbitrary speech audio is a crucial problem in the field of digital human and metaverse. A modern talking face generation method is expected to achieve the goals of generalized audio-lip synchronization, good video quality, and high system efficiency. Recently, neural radiance field (NeRF) has become a popular rendering technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video. However, there still exist several challenges for NeRF-based methods: 1) as for the lip synchronization, it is hard to generate a long facial motion sequence of high temporal consistency and audio-lip accuracy; 2) as for the video quality, due to the limited data used to train the renderer, it is vulnerable to out-of-domain input condition and produce bad rendering results occasionally; 3) as for the system efficiency, the slow training and inference speed of the vanilla NeRF severely obstruct its usage in real-world applications. In this paper, we propose GeneFace++ to handle these challenges by 1) utilizing the pitch contour as an auxiliary feature and introducing a temporal loss in the facial motion prediction process; 2) proposing a landmark locally linear embedding method to regulate the outliers in the predicted motion sequence to avoid robustness issues; 3) designing a computationally efficient NeRF-based motion-to-video renderer to achieves fast training and real-time inference. With these settings, GeneFace++ becomes the first NeRF-based method that achieves stable and real-time talking face generation with generalized audio-lip synchronization. Extensive experiments show that our method outperforms state-of-the-art baselines in terms of subjective and objective evaluation. Video samples are available at https://genefaceplusplus.github.io .

翻译：生成与任意语音同步的说话人肖像，是数字人与元宇宙领域的关键问题。现代说话人脸生成方法需实现广义音频-唇形同步、高视频质量与高系统效率三大目标。近年来，神经辐射场（NeRF）因能用数分钟训练视频实现高保真、三维一致的说话人脸生成，成为该领域的主流渲染技术。然而，基于NeRF的方法仍面临挑战：1）唇形同步方面，难以生成长时域一致性高且音频-唇形精准的面部运动序列；2）视频质量方面，由于训练渲染器的数据有限，易受域外输入条件影响，偶尔产生低劣渲染结果；3）系统效率方面，原始NeRF缓慢的训练与推理速度严重阻碍其实际应用。本文提出的GeneFace++通过以下方案应对挑战：1）利用音高轮廓作为辅助特征，并在面部运动预测中引入时域损失函数；2）提出地标局部线性嵌入方法，对预测运动序列中的异常值进行正则化以避免鲁棒性问题；3）设计计算高效的NeRF运动-视频渲染器，实现快速训练与实时推理。基于这些设计，GeneFace++成为首个实现稳定、实时且具有广义音频-唇形同步的说话人脸生成的NeRF方法。大量实验表明，本方法在主观与客观评价中均优于前沿基线模型。视频示例见https://genefaceplusplus.github.io。