In this work, we consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera. In contrast to expensive marker-based or multi-view systems, our lightweight setup is ideal for private users as it enables an affordable 3D motion capture that is easy to install and does not require expert knowledge. To deal with this challenging setting, we leverage recent advances in computer vision using large-scale pre-trained models for a variety of modalities, including 2D body joints, joint angles, normalized disparity maps, and human segmentation masks. Thus, we introduce the first non-linear optimization-based approach that jointly solves for the absolute 3D position of each human, their articulated pose, their individual shapes as well as the scale of the scene. In particular, we estimate the scene depth and person unique scale from normalized disparity predictions using the 2D body joints and joint angles. Given the per-frame scene depth, we reconstruct a point-cloud of the static scene in 3D space. Finally, given the per-frame 3D estimates of the humans and scene point-cloud, we perform a space-time coherent optimization over the video to ensure temporal, spatial and physical plausibility. We evaluate our method on established multi-person 3D human pose benchmarks where we consistently outperform previous methods and we qualitatively demonstrate that our method is robust to in-the-wild conditions including challenging scenes with people of different sizes.

翻译：在这项工作中，我们考虑从固定摄像头录制的一段单目RGB视频中，估算场景中多人的三维位置、体型及关节姿态的问题。与昂贵的标记点或多视图系统相比，我们的轻量级设置非常适合个人用户，因为它提供了一种经济实惠、易于安装且无需专业知识的3D运动捕捉方案。为应对这一挑战性环境，我们利用计算机视觉领域的最新进展，采用了大规模预训练模型处理多种模态信息，包括二维身体关节点、关节角度、归一化视差图以及人体分割掩码。因此，我们首次提出了一种基于非线性优化的方法，该方法能联合求解每个人的绝对三维位置、关节姿态、个体体型以及场景尺度。具体而言，我们利用二维身体关节点和关节角度，从归一化视差预测中估算场景深度和个体独特尺度。基于每帧的场景深度，我们在三维空间中重建静态场景的点云。最后，基于每帧的人体三维估计和场景点云，我们对视频进行时空一致性优化，以确保时间、空间和物理上的合理性。我们在公认的多人三维人体姿态基准上评估了该方法，结果持续优于以往方法，并通过定性实验证明该方法对野外条件（包括包含不同体型人物的复杂场景）具有鲁棒性。