3D interacting hand pose estimation from a single RGB image is a challenging task, due to serious self-occlusion and inter-occlusion towards hands, confusing similar appearance patterns between 2 hands, ill-posed joint position mapping from 2D to 3D, etc.. To address these, we propose to extend A2J-the state-of-the-art depth-based 3D single hand pose estimation method-to RGB domain under interacting hand condition. Our key idea is to equip A2J with strong local-global aware ability to well capture interacting hands' local fine details and global articulated clues among joints jointly. To this end, A2J is evolved under Transformer's non-local encoding-decoding framework to build A2J-Transformer. It holds 3 main advantages over A2J. First, self-attention across local anchor points is built to make them global spatial context aware to better capture joints' articulation clues for resisting occlusion. Secondly, each anchor point is regarded as learnable query with adaptive feature learning for facilitating pattern fitting capacity, instead of having the same local representation with the others. Last but not least, anchor point locates in 3D space instead of 2D as in A2J, to leverage 3D pose prediction. Experiments on challenging InterHand 2.6M demonstrate that, A2J-Transformer can achieve state-of-the-art model-free performance (3.38mm MPJPE advancement in 2-hand case) and can also be applied to depth domain with strong generalization.

翻译：从单张RGB图像进行三维交互手势姿态估计是一项极具挑战性的任务，其原因包括手部严重的自遮挡与互遮挡、双手间混淆的相似外观模式、以及从二维到三维的不适定关节点位置映射等。针对这些问题，我们提出将基于深度图的最先进三维单手势姿态估计方法A2J扩展到交互手势场景下的RGB领域。我们的核心思想是赋予A2J强大的局部-全局感知能力，使其能够同时有效捕捉交互手势的局部精细细节与关节点间的全局关节线索。为此，我们在Transformer的非局部编码-解码框架下对A2J进行演进，构建了A2J-Transformer。相较于A2J，该方法具有三大优势：首先，在局部锚点间建立自注意力机制，使其具备全局空间上下文感知能力，从而更好地捕捉关节点关节线索以抵抗遮挡；其次，每个锚点被视作可学习查询，通过自适应特征学习提升模式拟合能力，而非像A2J中所有锚点共享相同的局部表征；最后，锚点定位于三维空间而非二维空间，以充分利用三维姿态预测。在具有挑战性的InterHand 2.6M数据集上的实验表明，A2J-Transformer能够实现最先进的无模型性能（双手情况下MPJPE精度提升3.38mm），且可泛化应用于深度图领域。