We introduce FaceTalk, a novel generative approach designed for synthesizing high-fidelity 3D motion sequences of talking human heads from input audio signal. To capture the expressive, detailed nature of human heads, including hair, ears, and finer-scale eye movements, we propose to couple speech signal with the latent space of neural parametric head models to create high-fidelity, temporally coherent motion sequences. We propose a new latent diffusion model for this task, operating in the expression space of neural parametric head models, to synthesize audio-driven realistic head sequences. In the absence of a dataset with corresponding NPHM expressions to audio, we optimize for these correspondences to produce a dataset of temporally-optimized NPHM expressions fit to audio-video recordings of people talking. To the best of our knowledge, this is the first work to propose a generative approach for realistic and high-quality motion synthesis of volumetric human heads, representing a significant advancement in the field of audio-driven 3D animation. Notably, our approach stands out in its ability to generate plausible motion sequences that can produce high-fidelity head animation coupled with the NPHM shape space. Our experimental results substantiate the effectiveness of FaceTalk, consistently achieving superior and visually natural motion, encompassing diverse facial expressions and styles, outperforming existing methods by 75% in perceptual user study evaluation.

翻译：我们提出FaceTalk，一种新颖的生成式方法，旨在从输入音频信号合成高保真度的人头说话3D运动序列。为捕捉人头富有表现力的细节特征（包括头发、耳朵及精细的眼部运动），我们提出将语音信号与神经参数化头部模型的潜在空间耦合，以生成高保真、时间连贯的运动序列。为此，我们设计了一种新的潜在扩散模型，该模型在神经参数化头部模型的表情空间中运行，以合成音频驱动的逼真头部序列。针对当前缺乏与NPHM表情对应的音频数据集，我们通过优化这些对应关系，构建了一个将时间优化后的NPHM表情拟合至人物说话音视频记录的数据集。据我们所知，这是首项提出用于三维体素人头逼真高质量运动合成的生成式方法，标志着音频驱动3D动画领域的重大突破。值得注意的是，我们的方法在生成合理运动序列方面表现突出，能够结合NPHM形状空间产生高保真头部动画。实验结果充分验证了FaceTalk的有效性，其在感知用户研究评估中以75%的优势持续优于现有方法，生成包含多样面部表情与风格的优异且视觉自然的运动。