Estimating 3D human motion from an egocentric video sequence plays a critical role in human behavior understanding and has various applications in VR/AR. However, naively learning a mapping between egocentric videos and human motions is challenging, because the user's body is often unobserved by the front-facing camera placed on the head of the user. In addition, collecting large-scale, high-quality datasets with paired egocentric videos and 3D human motions requires accurate motion capture devices, which often limit the variety of scenes in the videos to lab-like environments. To eliminate the need for paired egocentric video and human motions, we propose a new method, Ego-Body Pose Estimation via Ego-Head Pose Estimation (EgoEgo), which decomposes the problem into two stages, connected by the head motion as an intermediate representation. EgoEgo first integrates SLAM and a learning approach to estimate accurate head motion. Subsequently, leveraging the estimated head pose as input, EgoEgo utilizes conditional diffusion to generate multiple plausible full-body motions. This disentanglement of head and body pose eliminates the need for training datasets with paired egocentric videos and 3D human motion, enabling us to leverage large-scale egocentric video datasets and motion capture datasets separately. Moreover, for systematic benchmarking, we develop a synthetic dataset, AMASS-Replica-Ego-Syn (ARES), with paired egocentric videos and human motion. On both ARES and real data, our EgoEgo model performs significantly better than the current state-of-the-art methods.

翻译：从第一人称视频序列中估计3D人体运动在人类行为理解中扮演着关键角色，并在VR/AR领域具有广泛应用。然而，直接学习第一人称视频与人体运动之间的映射关系极具挑战性，因为用户的身体往往未被置于头部的正面摄像头捕捉到。此外，收集包含配对的第一人称视频和3D人体运动的大规模高质量数据集需要精确的动作捕捉设备，这通常将视频场景限制在类似实验室的环境中。为消除对配对第一人称视频和人体运动数据的需求，我们提出了一种新方法——通过自头姿态估计进行自我身体姿态估计（EgoEgo），该方法将问题分解为两个阶段，并通过头部运动作为中间表征连接。EgoEgo首先整合SLAM与学习方法以估计精确的头部运动；随后，利用估计的头部姿态作为输入，EgoEgo采用条件扩散模型生成多个合理的全身运动。这种头部与身体姿态的解耦消除了对配对第一人称视频与3D人体运动训练数据集的需求，使我们能够分别利用大规模第一人称视频数据集和动作捕捉数据集。此外，为进行系统性基准测试，我们开发了合成数据集AMASS-Replica-Ego-Syn（ARES），该数据集包含配对的第一人称视频和人体运动。在ARES和真实数据上，我们的EgoEgo模型性能显著优于当前最先进方法。