Recent methods for arbitrary-skeleton motion capture from monocular video follow a factorized pipeline, where a Video-to-Pose network predicts joint positions and an analytical inverse-kinematics (IK) stage recovers joint rotations. While effective, this design is inherently limited, since joint positions do not fully determine rotations and leave degrees of freedom such as bone-axis twist ambiguous, and the non-differentiable IK stage prevents the system from adapting to noisy predictions or optimizing for the final animation objective. In this work, we present the first fully end-to-end framework in which both Video-to-Pose and Pose-to-Rotation are learnable and jointly optimized. We observe that the ambiguity in pose-to-rotation mapping arises from missing coordinate system information: the same joint positions can correspond to different rotations under different rest poses and local axis conventions. To resolve this, we introduce a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system. This formulation turns rotation prediction into a well-constrained conditional problem and enables effective learning. In addition, our model predicts joint positions directly from video without relying on mesh intermediates, improving both robustness and efficiency. Both stages share a skeleton-aware Global-Local Graph-guided Multi-Head Attention (GL-GMHA) module for joint-level local reasoning and global coordination. Experiments on Truebones Zoo and Objaverse show that our method reduces rotation error from ~17 degrees to ~10 degrees, and to 6.54 degrees on unseen skeletons, while achieving ~20x faster inference than mesh-based pipelines. Project page: https://animotionlab.github.io/MoCapAnythingV2/

翻译：近期针对单目视频的任意骨架运动捕捉方法采用分解式处理流程：由视频-姿态网络预测关节位置，再通过解析逆运动学（IK）阶段恢复关节旋转。该方案虽具有效性，但存在固有局限性——关节位置无法完全确定旋转参数，导致骨轴扭转等自由度存在模糊性，且不可微分的IK阶段使系统无法适应含噪声预测或针对最终动画目标进行优化。本文提出首个全端到端框架，其中视频-姿态与姿态-旋转两个阶段均可学习且联合优化。我们发现姿态-旋转映射的模糊性源于坐标系信息的缺失：相同关节位置在不同静止姿态及局部轴约定下可对应不同旋转值。为解决此问题，我们引入目标资产的参考姿态-旋转对，该数据对结合静止姿态不仅能锚定映射关系，更能定义底层旋转坐标系。该公式将旋转预测转化为约束明确的条件问题，从而支持有效学习。此外，我们的模型可直接从视频预测关节位置，无需网格中间表示，在提升鲁棒性的同时提高效率。两个阶段共享骨架感知的全局-局部图引导多头注意力（GL-GMHA）模块，实现关节级局部推理与全局协调。在Truebones Zoo与Objaverse数据集上的实验表明，本方法将旋转误差从约17度降至约10度，对未见骨架的误差为6.54度，同时推理速度比基于网格的流程快约20倍。项目主页：https://animotionlab.github.io/MoCapAnythingV2/