Generating vivid and diverse 3D co-speech gestures is crucial for various applications in animating virtual avatars. While most existing methods can generate gestures from audio directly, they usually overlook that emotion is one of the key factors of authentic co-speech gesture generation. In this work, we propose EmotionGesture, a novel framework for synthesizing vivid and diverse emotional co-speech 3D gestures from audio. Considering emotion is often entangled with the rhythmic beat in speech audio, we first develop an Emotion-Beat Mining module (EBM) to extract the emotion and audio beat features as well as model their correlation via a transcript-based visual-rhythm alignment. Then, we propose an initial pose based Spatial-Temporal Prompter (STP) to generate future gestures from the given initial poses. STP effectively models the spatial-temporal correlations between the initial poses and the future gestures, thus producing the spatial-temporal coherent pose prompt. Once we obtain pose prompts, emotion, and audio beat features, we will generate 3D co-speech gestures through a transformer architecture. However, considering the poses of existing datasets often contain jittering effects, this would lead to generating unstable gestures. To address this issue, we propose an effective objective function, dubbed Motion-Smooth Loss. Specifically, we model motion offset to compensate for jittering ground-truth by forcing gestures to be smooth. Last, we present an emotion-conditioned VAE to sample emotion features, enabling us to generate diverse emotional results. Extensive experiments demonstrate that our framework outperforms the state-of-the-art, achieving vivid and diverse emotional co-speech 3D gestures. Our code and dataset will be released at the project page: https://xingqunqi-lab.github.io/Emotion-Gesture-Web/

翻译：生成生动且多样化的3D共语手势对于虚拟角色动画的各种应用至关重要。虽然现有方法大多能直接从音频生成手势，但它们通常忽略了情感是真实共语手势生成的关键因素之一。本文提出EmotionGesture，一种新颖框架，用于从音频合成生动且多样化的情感共语3D手势。考虑到情感常与语音音频中的节奏节拍交织，我们首先开发情感-节拍挖掘模块（EBM），提取情感与音频节拍特征，并通过基于文本的视觉-节奏对齐建模其相关性。随后，我们提出基于初始姿态的时空提示器（STP），从给定初始姿态生成未来手势。STP有效建模了初始姿态与未来手势之间的时空相关性，从而产生时空一致的姿态提示。获取姿态提示、情感及音频节拍特征后，我们通过Transformer架构生成3D共语手势。然而，现有数据集的姿态常包含抖动效应，这会导致生成不稳定的手势。为解决此问题，我们提出一种有效目标函数——运动平滑损失。具体而言，我们建模运动偏移，通过强制手势平滑来补偿抖动的真实值。最后，我们提出情感条件变分自编码器以采样情感特征，从而生成多样化的情感结果。大量实验表明，我们的框架优于现有方法，实现了生动且多样化的情感共语3D手势。代码与数据集将在项目页面发布：https://xingqunqi-lab.github.io/Emotion-Gesture-Web/