Talking face generation has been extensively investigated owing to its wide applicability. The two primary frameworks used for talking face generation comprise a text-driven framework, which generates synchronized speech and talking faces from text, and a speech-driven framework, which generates talking faces from speech. To integrate these frameworks, this paper proposes a unified facial landmark generator (UniFLG). The proposed system exploits end-to-end text-to-speech not only for synthesizing speech but also for extracting a series of latent representations that are common to text and speech, and feeds it to a landmark decoder to generate facial landmarks. We demonstrate that our system achieves higher naturalness in both speech synthesis and facial landmark generation compared to the state-of-the-art text-driven method. We further demonstrate that our system can generate facial landmarks from speech of speakers without facial video data or even speech data.

翻译：由于面部动画生成具有广泛的应用前景，该领域已得到大量研究。当前主流框架主要包括文本驱动和语音驱动两种范式：前者从文本生成同步语音与面部动画，后者则从语音生成面部动画。为整合这两种框架，本文提出统一面部标志点生成器（UniFLG）。该系统利用端到端文本转语音技术，不仅用于合成语音，还用于提取文本与语音共有的隐层表征序列，并将其输入至标志点解码器以生成面部标志点。实验表明，与现有最优文本驱动方法相比，本系统在语音合成与面部标志点生成两方面均实现了更高自然度。此外，本系统还能够对无面部视频数据甚至无语音数据说话者的语音生成面部标志点。