The generation of realistic and contextually relevant co-speech gestures is a challenging yet increasingly important task in the creation of multimodal artificial agents. Prior methods focused on learning a direct correspondence between co-speech gesture representations and produced motions, which created seemingly natural but often unconvincing gestures during human assessment. We present an approach to pre-train partial gesture sequences using a generative adversarial network with a quantization pipeline. The resulting codebook vectors serve as both input and output in our framework, forming the basis for the generation and reconstruction of gestures. By learning the mapping of a latent space representation as opposed to directly mapping it to a vector representation, this framework facilitates the generation of highly realistic and expressive gestures that closely replicate human movement and behavior, while simultaneously avoiding artifacts in the generation process. We evaluate our approach by comparing it with established methods for generating co-speech gestures as well as with existing datasets of human behavior. We also perform an ablation study to assess our findings. The results show that our approach outperforms the current state of the art by a clear margin and is partially indistinguishable from human gesturing. We make our data pipeline and the generation framework publicly available.

翻译：生成逼真且上下文相关的共语手势，是构建多模态人工智能体过程中一项兼具挑战性与重要性的任务。现有方法侧重于学习共语手势表征与生成动作之间的直接对应关系，虽然能产生看似自然的动作，但在人类评估中往往缺乏说服力。本文提出一种基于生成对抗网络与量化流水线的部分手势序列预训练方法。通过学习得到的码本向量作为框架中的输入与输出，构成手势生成与重建的基础。该框架通过学习隐空间表征的映射（而非直接映射至向量表征），能够生成高度逼真且富有表现力的手势，从而精确复现人类动作与行为，同时避免生成过程中的伪影问题。我们将所提方法与现有共语手势生成方法及人类行为数据集进行对比评估，并开展消融研究以验证结论。实验结果表明，本方法以显著优势超越当前最先进水平，且在部分场景下与人类手势难以区分。我们已将数据流水线与生成框架开源发布。