The automatic generation of stylized co-speech gestures has recently received increasing attention. Previous systems typically allow style control via predefined text labels or example motion clips, which are often not flexible enough to convey user intent accurately. In this work, we present GestureDiffuCLIP, a neural network framework for synthesizing realistic, stylized co-speech gestures with flexible style control. We leverage the power of the large-scale Contrastive-Language-Image-Pre-training (CLIP) model and present a novel CLIP-guided mechanism that extracts efficient style representations from multiple input modalities, such as a piece of text, an example motion clip, or a video. Our system learns a latent diffusion model to generate high-quality gestures and infuses the CLIP representations of style into the generator via an adaptive instance normalization (AdaIN) layer. We further devise a gesture-transcript alignment mechanism that ensures a semantically correct gesture generation based on contrastive learning. Our system can also be extended to allow fine-grained style control of individual body parts. We demonstrate an extensive set of examples showing the flexibility and generalizability of our model to a variety of style descriptions. In a user study, we show that our system outperforms the state-of-the-art approaches regarding human likeness, appropriateness, and style correctness.

翻译：自动生成具有风格化的共语手势近年来受到越来越多的关注。以往系统通常通过预定义文本标签或示例动作片段实现风格控制，但这类方法往往灵活性不足，难以准确传达用户意图。本文提出GestureDiffuCLIP——一种用于合成逼真、风格化共语手势且具备灵活风格控制的神经网络框架。我们利用大规模对比语言-图像预训练（CLIP）模型的能力，提出一种新颖的CLIP引导机制，能够从文本、示例动作片段或视频等多模态输入中提取高效风格表征。系统通过学习潜在扩散模型生成高质量手势，并通过自适应实例归一化（AdaIN）层将CLIP风格表征注入生成器。此外，我们设计了一种手势-文本对齐机制，基于对比学习确保生成语义正确的手势。该系统还可扩展至对个体身体部位的细粒度风格控制。我们通过大量实例展示了模型对多种风格描述的灵活性与泛化能力。用户研究表明，本系统在人类相似度、适当性和风格正确性方面均优于现有最优方法。