The generation of stylistic 3D facial animations driven by speech poses a significant challenge as it requires learning a many-to-many mapping between speech, style, and the corresponding natural facial motion. However, existing methods either employ a deterministic model for speech-to-motion mapping or encode the style using a one-hot encoding scheme. Notably, the one-hot encoding approach fails to capture the complexity of the style and thus limits generalization ability. In this paper, we propose DiffPoseTalk, a generative framework based on the diffusion model combined with a style encoder that extracts style embeddings from short reference videos. During inference, we employ classifier-free guidance to guide the generation process based on the speech and style. We extend this to include the generation of head poses, thereby enhancing user perception. Additionally, we address the shortage of scanned 3D talking face data by training our model on reconstructed 3DMM parameters from a high-quality, in-the-wild audio-visual dataset. Our extensive experiments and user study demonstrate that our approach outperforms state-of-the-art methods. The code and dataset will be made publicly available.

翻译：语音驱动的风格化三维人脸动画生成是一项重大挑战，因为它需要学习语音、风格与相应自然面部运动之间的多对多映射。然而，现有方法要么采用确定性模型进行语音到运动的映射，要么使用独热编码方案对风格进行编码。值得注意的是，独热编码方法无法捕捉风格的复杂性，从而限制了泛化能力。本文提出DiffPoseTalk，这是一种基于扩散模型的生成框架，结合了从短视频参考中提取风格嵌入的风格编码器。在推理阶段，我们采用无分类器引导机制，根据语音和风格指导生成过程。我们将此方法扩展至头部姿态生成，从而提升用户感知。此外，我们通过使用高质量野外音视频数据集重建的三维形变模型参数训练模型，解决了扫描三维说话人脸数据短缺的问题。大量实验和用户研究表明，我们的方法优于现有最先进方法。代码和数据集将公开发布。