The automatic co-speech gesture generation draws much attention in computer animation. Previous works designed network structures on individual datasets, which resulted in a lack of data volume and generalizability across different motion capture standards. In addition, it is a challenging task due to the weak correlation between speech and gestures. To address these problems, we present UnifiedGesture, a novel diffusion model-based speech-driven gesture synthesis approach, trained on multiple gesture datasets with different skeletons. Specifically, we first present a retargeting network to learn latent homeomorphic graphs for different motion capture standards, unifying the representations of various gestures while extending the dataset. We then capture the correlation between speech and gestures based on a diffusion model architecture using cross-local attention and self-attention to generate better speech-matched and realistic gestures. To further align speech and gesture and increase diversity, we incorporate reinforcement learning on the discrete gesture units with a learned reward function. Extensive experiments show that UnifiedGesture outperforms recent approaches on speech-driven gesture generation in terms of CCA, FGD, and human-likeness. All code, pre-trained models, databases, and demos are available to the public at https://github.com/YoungSeng/UnifiedGesture.

翻译：自动生成伴随语音的手势在计算机动画领域备受关注。现有研究针对单个数据集设计网络结构，导致数据量不足且难以泛化至不同动作捕捉标准。此外，由于语音与手势间相关性较弱，该任务极具挑战性。为解决这些问题，我们提出UnifiedGesture——一种基于扩散模型的语音驱动手势合成新方法，该模型在包含多种骨架结构的手势数据集上训练。具体而言，我们首先构建重定向网络，学习不同动作捕捉标准下的潜在同胚图，在扩展数据集的同时统一多类手势表征；继而基于扩散模型架构，通过交叉局部注意力与自注意力机制捕捉语音与手势的关联，生成更贴合语音且逼真的手势。为进一步对齐语音与手势并增强多样性，我们引入强化学习机制，对离散手势单元采用学习到的奖励函数进行优化。大量实验表明，UnifiedGesture在CCA、FGD及类人属性指标上均优于当前语音驱动手势生成方法。所有代码、预训练模型、数据库及演示已公开于https://github.com/YoungSeng/UnifiedGesture。