Predicting natural and diverse 3D hand gestures from the upper body dynamics is a practical yet challenging task in virtual avatar creation. Previous works usually overlook the asymmetric motions between two hands and generate two hands in a holistic manner, leading to unnatural results. In this work, we introduce a novel bilateral hand disentanglement based two-stage 3D hand generation method to achieve natural and diverse 3D hand prediction from body dynamics. In the first stage, we intend to generate natural hand gestures by two hand-disentanglement branches. Considering the asymmetric gestures and motions of two hands, we introduce a Spatial-Residual Memory (SRM) module to model spatial interaction between the body and each hand by residual learning. To enhance the coordination of two hand motions wrt. body dynamics holistically, we then present a Temporal-Motion Memory (TMM) module. TMM can effectively model the temporal association between body dynamics and two hand motions. The second stage is built upon the insight that 3D hand predictions should be non-deterministic given the sequential body postures. Thus, we further diversify our 3D hand predictions based on the initial output from the stage one. Concretely, we propose a Prototypical-Memory Sampling Strategy (PSS) to generate the non-deterministic hand gestures by gradient-based Markov Chain Monte Carlo (MCMC) sampling. Extensive experiments demonstrate that our method outperforms the state-of-the-art models on the B2H dataset and our newly collected TED Hands dataset.

翻译：从上半身动力学预测自然且多样化的三维手势是虚拟化身创建中一项实际却具有挑战性的任务。以往的研究通常忽视双手之间的不对称运动，以整体方式生成双手，导致结果不自然。本文提出一种基于双侧手部解耦的两阶段三维手势生成方法，以实现从身体动力学中自然且多样化的三维手势预测。在第一阶段，我们通过两个手部解耦分支生成自然手势。考虑到双手的姿态与运动不对称，我们引入空间残差记忆模块，通过残差学习对躯体与每只手之间的空间交互进行建模。为增强双手运动相对于整体身体动力学的协调性，我们进一步提出时间运动记忆模块。该模块能够有效建模身体动力学与双手运动之间的时间关联。第二阶段基于以下洞见构建：根据连续的身体姿态，三维手势预测应具有非确定性。因此，我们在第一阶段初始输出的基础上进一步多样化三维手势预测。具体而言，我们提出原型记忆采样策略，通过基于梯度的马尔可夫链蒙特卡洛采样生成非确定性手势。大量实验表明，我们的方法在B2H数据集及新采集的TED Hands数据集上均优于现有最先进模型。