Human communication seamlessly integrates speech and bodily motion, where hand gestures naturally complement vocal prosody to express intent, emotion, and emphasis. While recent text-to-speech (TTS) systems have begun incorporating multimodal cues such as facial expressions or lip movements, the role of hand gestures in shaping prosody remains largely underexplored. We propose a novel multimodal TTS framework, Gesture2Speech, that leverages visual gesture cues to modulate prosody in synthesized speech. Motivated by the observation that confident and expressive speakers coordinate gestures with vocal prosody, we introduce a multimodal Mixture-of-Experts (MoE) architecture that dynamically fuses linguistic content and gesture features within a dedicated style extraction module. The fused representation conditions an LLM-based speech decoder, enabling prosodic modulation that is temporally aligned with hand movements. We further design a gesture-speech alignment loss that explicitly models their temporal correspondence to ensure fine-grained synchrony between gestures and prosodic contours. Evaluations on the PATS dataset show that Gesture2Speech outperforms state-of-the-art baselines in both speech naturalness and gesture-speech synchrony. To the best of our knowledge, this is the first work to utilize hand gesture cues for prosody control in neural speech synthesis. Demo samples are available at https://research.sri-media-analysis.com/aaai26-beeu-gesture2speech/

翻译：人类交流无缝融合了语音与身体动作，其中手势自然补充了声音韵律以表达意图、情感和强调。尽管近期文本转语音（TTS）系统已开始融入如面部表情或唇部运动等多模态线索，但手势在韵律塑造中的作用仍远未得到充分探索。我们提出了一种新颖的多模态TTS框架Gesture2Speech，该框架利用视觉手势线索来调制合成语音中的韵律。受自信且富有表现力的说话者会协调手势与声音韵律这一观察的启发，我们引入了一种多模态混合专家（MoE）架构，该架构在专用的风格提取模块中动态融合语言内容与手势特征。融合后的表征条件作用于基于大语言模型的语音解码器，从而实现对与手部运动时间对齐的韵律调制。我们进一步设计了一种手势-语音对齐损失，显式建模其时间对应关系以确保手势与韵律轮廓的细粒度同步。在PATS数据集上的评估表明，Gesture2Speech在语音自然度与手势-语音同步性方面均优于现有基线方法。据我们所知，这是首次利用手势线索在神经语音合成中控制韵律的工作。演示样本请访问https://research.sri-media-analysis.com/aaai26-beeu-gesture2speech/