3D reconstruction of hand-object manipulations is important for emulating human actions. Most methods dealing with challenging object manipulation scenarios, focus on hands reconstruction in isolation, ignoring physical and kinematic constraints due to object contact. Some approaches produce more realistic results by jointly reconstructing 3D hand-object interactions. However, they focus on coarse pose estimation or rely upon known hand and object shapes. We propose the first approach for realistic 3D hand-object shape and pose reconstruction from a single depth map. Unlike previous work, our voxel-based reconstruction network regresses the vertex coordinates of a hand and an object and reconstructs more realistic interaction. Our pipeline additionally predicts voxelized hand-object shapes, having a one-to-one mapping to the input voxelized depth. Thereafter, we exploit the graph nature of the hand and object shapes, by utilizing the recent GraFormer network with positional embedding to reconstruct shapes from template meshes. In addition, we show the impact of adding another GraFormer component that refines the reconstructed shapes based on the hand-object interactions and its ability to reconstruct more accurate object shapes. We perform an extensive evaluation on the HO-3D and DexYCB datasets and show that our method outperforms existing approaches in hand reconstruction and produces plausible reconstructions for the objects

翻译：3D重建手物操控对于模拟人类动作至关重要。处理复杂物体操控场景的大多数方法仅孤立地重建手部，忽略了物体接触带来的物理与运动学约束。部分方法通过联合重建3D手物交互获得更真实结果，但侧重于粗略姿态估计或依赖已知的手部与物体形状。我们首次提出基于单深度图实现真实3D手物形状与姿态重建的方法。与以往工作不同，我们的体素化重建网络通过回归手部与物体的顶点坐标，重建出更真实的交互关系。该流程额外预测了与输入体素化深度一一对应的手物体素化形状。进而，我们利用手部与物体形状的图结构特性，采用集成位置嵌入的最新GraFormer网络从模板网格重建形状。此外，我们证明了添加另一个GraFormer组件——基于手物交互优化重建形状——对提升物体形状重建精度的影响。我们在HO-3D和DexYCB数据集上进行了广泛评估，结果表明我们的方法在手部重建方面优于现有方法，并能生成合理的物体重建结果。