We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources. Specifically, we propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations. The motion representations acquired in this way incorporate geometric, kinematic, and physical knowledge about human motion, which can be easily transferred to multiple downstream tasks. We implement the motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. Furthermore, our proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with a simple regression head (1-2 layers), which demonstrates the versatility of the learned motion representations. Code and models are available at https://motionbert.github.io/

翻译：我们提出了一种统一视角，通过从大规模、异构数据资源中学习人体运动表征，来解决多种以人为核心的视频任务。具体而言，我们设计了一个预训练阶段，在该阶段中，运动编码器被训练以从含噪的部分二维观测中恢复底层三维运动。通过这种方式获取的运动表征融合了关于人体运动的几何、运动学和物理知识，能够轻松迁移到多个下游任务。我们采用双流时空Transformer（DSTformer）神经网络实现运动编码器。该网络能够全面、自适应地捕捉骨骼关节间的长程时空关系，例如在从零开始训练时实现了迄今最低的三维姿态估计误差。此外，我们提出的框架仅需在预训练的运动编码器后附加简单的回归头（1-2层）进行微调，即可在所有三个下游任务上达到最优性能，这充分证明了所学习运动表征的通用性。代码和模型已开源：https://motionbert.github.io/