Generating holistic co-speech gestures that integrate full-body motion with facial expressions suffers from semantically incoherent coordination on body motion and spatially unstable meaningless movements due to existing part-decomposed or frame-level regression methods, We introduce 3DGesPolicy, a novel action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem through diffusion policy from robotics. By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns and ensures both spatially and semantically coherent movement trajectories that adhere to realistic motion manifolds. To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module that can deeply integrate and refine multi-modal signals, ensuring structured and fine-grained alignment between speech semantics, body motion, and facial expressions. Extensive quantitative and qualitative experiments on the BEAT2 dataset demonstrate the effectiveness of our 3DGesPolicy across other state-of-the-art methods in generating natural, expressive, and highly speech-aligned holistic gestures.

翻译：现有基于部件分解或帧级回归的方法在生成融合全身运动与面部表情的整体协同语音手势时，常导致身体动作语义协调性不足及空间不稳定的无意义运动。本文提出3DGesPolicy，一种基于动作的创新框架，通过机器人学中的扩散策略将整体手势生成重新定义为连续轨迹控制问题。通过将帧间变化建模为统一的整体动作，本方法能有效学习帧间整体手势运动模式，确保生成空间与语义协调且符合真实运动流形的运动轨迹。为弥合表达对齐的差距，我们进一步提出手势-音频-音素融合模块，能够深度融合并优化多模态信号，确保语音语义、身体运动与面部表情之间形成结构化、细粒度的对齐。在BEAT2数据集上进行的大量定量与定性实验表明，3DGesPolicy在生成自然、富有表现力且高度语音对齐的整体手势方面，优于其他现有先进方法。