While increasing attention has been paid to co-speech gesture synthesis, most previous works neglect to investigate hand gestures with explicit and essential semantics. In this paper, we study co-speech gesture generation with an emphasis on specific hand gesture activation, which can deliver more instructional information than common body movements. To achieve this, we first build a high-quality dataset of 3D human body movements including a set of semantically explicit hand gestures that are commonly used by live streamers. Then we present a hybrid-modality gesture generation system GestureHYDRA built upon a hybrid-modality diffusion transformer architecture with novelly designed motion-style injective transformer layers, which enables advanced gesture modeling ability and versatile gesture operations. To guarantee these specific hand gestures can be activated, we introduce a cascaded retrieval-augmented generation strategy built upon a semantic gesture repository annotated for each subject and an adaptive audio-gesture synchronization mechanism, which substantially improves semantic gesture activation and production efficiency. Quantitative and qualitative experiments demonstrate that our proposed approach achieves superior performance over all the counterparts. The project page can be found at https://mumuwei.github.io/GestureHYDRA/.

翻译：尽管伴随语音手势合成日益受到关注，但以往多数研究未能深入探究具有明确且本质语义的手势。本文重点研究特定手势激活的伴随语音手势生成，此类手势比常见的身体动作能传递更具指导性的信息。为此，我们首先构建了一个包含直播主播常用语义明确手势的高质量三维人体运动数据集。随后，我们提出基于混合模态扩散Transformer架构的生成系统GestureHYDRA，该系统创新性地设计了运动风格注入式Transformer层，从而实现了高级手势建模能力与多样化手势操作。为确保这些特定手势能够被激活，我们引入了一种基于为每个主体标注的语义手势库与自适应音频-手势同步机制的级联检索增强生成策略，显著提升了语义手势的激活率与生成效率。定量与定性实验表明，我们提出的方法在所有对比方案中均取得了更优的性能。项目页面详见 https://mumuwei.github.io/GestureHYDRA/。