Speech-driven 3D facial animation is challenging due to the scarcity of large-scale visual-audio datasets despite extensive research. Most prior works, typically focused on learning regression models on a small dataset using the method of least squares, encounter difficulties generating diverse lip movements from speech and require substantial effort in refining the generated outputs. To address these issues, we propose a speech-driven 3D facial animation with a diffusion model (SAiD), a lightweight Transformer-based U-Net with a cross-modality alignment bias between audio and visual to enhance lip synchronization. Moreover, we introduce BlendVOCA, a benchmark dataset of pairs of speech audio and parameters of a blendshape facial model, to address the scarcity of public resources. Our experimental results demonstrate that the proposed approach achieves comparable or superior performance in lip synchronization to baselines, ensures more diverse lip movements, and streamlines the animation editing process.

翻译：语音驱动的三维面部动画因大规模视觉-音频数据集的稀缺性而面临挑战，尽管已有大量研究。以往多数工作通常聚焦于利用最小二乘法在小型数据集上训练回归模型，这导致难以从语音中生成多样化的唇部运动，且需要大量精力对生成结果进行精调。为解决这些问题，我们提出一种基于扩散模型的语音驱动三维面部动画方法（SAiD），该模型采用轻量级Transformer架构的U-Net，并引入音频与视觉间的跨模态对齐偏置以增强唇部同步性。此外，我们发布了BlendVOCA基准数据集，该数据集包含语音音频与混合变形面部模型参数对，旨在缓解公共资源匮乏的现状。实验结果表明，所提方法在唇部同步性能上与基线方法相当或更优，能确保更丰富的唇部运动多样性，并简化了动画编辑流程。