This paper addresses the problem of generating lifelike holistic co-speech motions for 3D avatars, focusing on two key aspects: variability and coordination. Variability allows the avatar to exhibit a wide range of motions even with similar speech content, while coordination ensures a harmonious alignment among facial expressions, hand gestures, and body poses. We aim to achieve both with ProbTalk, a unified probabilistic framework designed to jointly model facial, hand, and body movements in speech. ProbTalk builds on the variational autoencoder (VAE) architecture and incorporates three core designs. First, we introduce product quantization (PQ) to the VAE, which enriches the representation of complex holistic motion. Second, we devise a novel non-autoregressive model that embeds 2D positional encoding into the product-quantized representation, thereby preserving essential structure information of the PQ codes. Last, we employ a secondary stage to refine the preliminary prediction, further sharpening the high-frequency details. Coupling these three designs enables ProbTalk to generate natural and diverse holistic co-speech motions, outperforming several state-of-the-art methods in qualitative and quantitative evaluations, particularly in terms of realism. Our code and model will be released for research purposes at https://feifeifeiliu.github.io/probtalk/.

翻译：本文针对生成3D虚拟角色逼真的整体共语动作问题展开研究，聚焦两个关键方面：可变性与协调性。可变性使虚拟角色在相似语音内容下能展现多样化的动作，而协调性则确保面部表情、手势及身体姿态间的和谐对齐。我们通过ProbTalk这一统一概率框架实现上述目标，该框架专为联合建模语音驱动的面部、手部及身体运动而设计。ProbTalk基于变分自编码器（VAE）架构，并整合三大核心设计：首先，将乘积量化（PQ）引入VAE，以增强复杂整体动作的表示能力；其次，提出一种新颖的非自回归模型，通过将二维位置编码嵌入乘积量化表示中，保留PQ码的关键结构信息；最后，采用二级优化阶段对初步预测结果进行精炼，进一步强化高频细节。三大设计的协同作用使ProbTalk能够生成自然多样的整体共语动作，在定性与定量评估中多项指标超越现有最优方法，尤其在真实感表现上尤为突出。为促进研究，我们将开源相关代码与模型（https://feifeifeiliu.github.io/probtalk/）。