The automatic generation of stylized co-speech gestures has recently received increasing attention. Previous systems typically allow style control via predefined text labels or example motion clips, which are often not flexible enough to convey user intent accurately. In this work, we present GestureDiffuCLIP, a neural network framework for synthesizing realistic, stylized co-speech gestures with flexible style control. We leverage the power of the large-scale Contrastive-Language-Image-Pre-training (CLIP) model and present a novel CLIP-guided mechanism that extracts efficient style representations from multiple input modalities, such as a piece of text, an example motion clip, or a video. Our system learns a latent diffusion model to generate high-quality gestures and infuses the CLIP representations of style into the generator via an adaptive instance normalization (AdaIN) layer. We further devise a gesture-transcript alignment mechanism that ensures a semantically correct gesture generation based on contrastive learning. Our system can also be extended to allow fine-grained style control of individual body parts. We demonstrate an extensive set of examples showing the flexibility and generalizability of our model to a variety of style descriptions. In a user study, we show that our system outperforms the state-of-the-art approaches regarding human likeness, appropriateness, and style correctness.

翻译：具有风格化的共语手势自动生成近年来受到越来越多的关注。以往系统通常通过预定义文本标签或示例动作片段实现风格控制，但往往难以灵活且准确地传达用户意图。本文提出GestureDiffuCLIP——一个用于合成真实、风格化共语手势的神经网络框架，支持灵活的风格控制。我们利用大规模对比语言-图像预训练（CLIP）模型的力量，提出一种新颖的CLIP引导机制，从多种输入模态（如文本片段、示例动作片段或视频）中提取高效的风格表征。该系统学习了一个潜在扩散模型以生成高质量手势，并通过自适应实例归一化（AdaIN）层将CLIP风格表征注入生成器。我们进一步设计了一种基于对比学习的手势-文本对齐机制，确保生成语义正确的手势。该系统还可扩展至对身体各部位的细粒度风格控制。我们通过大量实例展示了模型对多样风格描述的灵活性与泛化能力。用户研究表明，我们的系统在人类相似度、动作适宜性和风格正确性方面均优于现有最优方法。