We present XFormer, a novel human mesh and motion capture method that achieves real-time performance on consumer CPUs given only monocular images as input. The proposed network architecture contains two branches: a keypoint branch that estimates 3D human mesh vertices given 2D keypoints, and an image branch that makes predictions directly from the RGB image features. At the core of our method is a cross-modal transformer block that allows information to flow across these two branches by modeling the attention between 2D keypoint coordinates and image spatial features. Our architecture is smartly designed, which enables us to train on various types of datasets including images with 2D/3D annotations, images with 3D pseudo labels, and motion capture datasets that do not have associated images. This effectively improves the accuracy and generalization ability of our system. Built on a lightweight backbone (MobileNetV3), our method runs blazing fast (over 30fps on a single CPU core) and still yields competitive accuracy. Furthermore, with an HRNet backbone, XFormer delivers state-of-the-art performance on Huamn3.6 and 3DPW datasets.

翻译：我们提出XFormer，一种仅以单目图像为输入即可在消费级CPU上实现实时性能的新型人体网格与动作捕捉方法。所提出的网络架构包含两个分支：关键点分支，根据二维关键点估计三维人体网格顶点；图像分支，直接从RGB图像特征进行预测。该方法的核心是跨模态变换器模块，通过建模二维关键点坐标与图像空间特征之间的注意力机制，实现两分支间的信息交互。我们巧妙设计的架构使其能够在多种类型数据集上训练，包括带二维/三维标注的图像、带三维伪标签的图像以及无关联图像的动作捕捉数据集。这有效提升了系统的精度与泛化能力。基于轻量级骨干网络（MobileNetV3），该方法运行极其迅速（单CPU核心超30fps），同时保持具有竞争力的精度。此外，采用HRNet骨干网络时，XFormer在Human3.6和3DPW数据集上均达到最先进性能。