The recently emerging text-to-motion advances have spired numerous attempts for convenient and interactive human motion generation. Yet, existing methods are largely limited to generating body motions only without considering the rich two-hand motions, let alone handling various conditions like body dynamics or texts. To break the data bottleneck, we propose BOTH57M, a novel multi-modal dataset for two-hand motion generation. Our dataset includes accurate motion tracking for the human body and hands and provides pair-wised finger-level hand annotations and body descriptions. We further provide a strong baseline method, BOTH2Hands, for the novel task: generating vivid two-hand motions from both implicit body dynamics and explicit text prompts. We first warm up two parallel body-to-hand and text-to-hand diffusion models and then utilize the cross-attention transformer for motion blending. Extensive experiments and cross-validations demonstrate the effectiveness of our approach and dataset for generating convincing two-hand motions from the hybrid body-and-textual conditions. Our dataset and code will be disseminated to the community for future research.

翻译：近期兴起的文本到运动生成技术推动了便捷交互式人体运动生成的诸多尝试。然而，现有方法大多局限于仅生成身体运动，未考虑丰富双手运动的建模，更遑论处理身体动力学或文本等多种条件。为突破数据瓶颈，我们提出BOTH57M——一个面向双手运动生成的新型多模态数据集。该数据集包含人体与双手的精确运动追踪，并提供配对的指级手部标注与身体描述。我们进一步针对这一新任务提出强基线方法BOTH2Hands：从隐式身体动力学与显式文本提示中生成生动的双手运动。我们首先预热两个平行的身体到双手与文本到双手扩散模型，然后利用交叉注意力Transformer进行运动融合。大量实验与交叉验证表明，本方法与数据集可从混合的身体与文本条件中生成令人信服的双手运动。我们的数据集与代码将向研究社区开放，以促进未来研究。