Recovering whole-body mesh by inferring the abstract pose and shape parameters from visual content can obtain 3D bodies with realistic structures. However, the inferring process is highly non-linear and suffers from image-mesh misalignment, resulting in inaccurate reconstruction. In contrast, 3D keypoint estimation methods utilize the volumetric representation to achieve pixel-level accuracy but may predict unrealistic body structures. To address these issues, this paper presents a novel hybrid inverse kinematics solution, HybrIK, that integrates the merits of 3D keypoint estimation and body mesh recovery in a unified framework. HybrIK directly transforms accurate 3D joints to body-part rotations via twist-and-swing decomposition. The swing rotations are analytically solved with 3D joints, while the twist rotations are derived from visual cues through neural networks. To capture comprehensive whole-body details, we further develop a holistic framework, HybrIK-X, which enhances HybrIK with articulated hands and an expressive face. HybrIK-X is fast and accurate by solving the whole-body pose with a one-stage model. Experiments demonstrate that HybrIK and HybrIK-X preserve both the accuracy of 3D joints and the realistic structure of the parametric human model, leading to pixel-aligned whole-body mesh recovery. The proposed method significantly surpasses the state-of-the-art methods on various benchmarks for body-only, hand-only, and whole-body scenarios. Code and results can be found at https://jeffli.site/HybrIK-X/

翻译：通过从视觉内容中推断抽象的姿势和形状参数来恢复全身网格，可以获得具有真实结构的三维人体。然而，推断过程高度非线性且存在图像-网格未对齐问题，导致重建不准确。相比之下，三维关键点估计方法利用体素表示实现像素级精度，但可能预测出不真实的肢体结构。为解决这些问题，本文提出了一种新颖的混合逆运动学解决方案HybrIK，将三维关键点估计与身体网格恢复的优势统一到同一框架中。HybrIK通过扭曲-摆动分解将精确的三维关节点直接转换为身体部位旋转。其中摆动旋转通过三维关节点解析求解，而扭曲旋转则通过神经网络从视觉线索中推导得出。为捕捉完整的全身细节，我们进一步开发了整体框架HybrIK-X，通过引入带关节的手部和富有表现力的面部增强HybrIK。HybrIK-X通过单阶段模型求解全身姿势，兼具快速与精确特性。实验表明，HybrIK与HybrIK-X既保留了三维关节点精度，又保持了参数化人体模型的真实结构，实现了像素级对齐的全身网格恢复。所提方法在纯身体、纯手部及全身场景的多个基准测试中显著超越现有最先进方法。代码与结果可通过https://jeffli.site/HybrIK-X/获取。