As robotics progresses toward general manipulation, dexterous hands are becoming increasingly critical. However, proprioception in dexterous hands remains a bottleneck due to limitations in volume and generality. In this work, we present HandCept, the first visual-inertial proprioception framework designed to overcome the challenges of traditional joint angle estimation methods for dexterous hands. HandCept addresses the difficulty of achieving accurate and robust joint angle estimation in dynamic environments where both visual and inertial measurements are prone to noise and drift. It leverages a zero-shot learning approach using a wrist-mounted RGB-D camera and 9-axis IMUs, fused in real time via a latency-free Extended Kalman Filter (EKF). Our results show that HandCept achieves joint angle estimation errors generally between $2^{\circ}$ and $4^{\circ}$ without observable drift, outperforming visual-only and inertial-only methods. Furthermore, we validate the stability and uniformity of the IMU system, demonstrating that a common base frame across IMUs simplifies system calibration. To support sim-to-real transfer, we also open-source our high-fidelity rendering pipeline, which is essential for training without real-world ground truth. This work offers a robust, generalizable solution for proprioception in dexterous hands, with significant implications for robotic manipulation and human-robot interaction. https://github.com/huangjund/blenderYCB

翻译：随着机器人技术向通用操作发展，灵巧手正变得日益关键。然而，由于体积和通用性的限制，灵巧手的本体感知能力仍是技术瓶颈。本文提出HandCept，这是首个旨在克服灵巧手传统关节角度估计方法挑战的视觉-惯性本体感知框架。该框架解决了在视觉与惯性测量均易受噪声与漂移影响的动态环境中，实现精确鲁棒关节角度估计的难题。HandCept采用零样本学习方法，利用腕载RGB-D摄像头与九轴IMU，通过无延迟扩展卡尔曼滤波器实现实时融合。实验结果表明，HandCept的关节角度估计误差通常介于$2^{\circ}$至$4^{\circ}$之间且无显著漂移，性能优于纯视觉与纯惯性方法。此外，我们验证了IMU系统的稳定性与一致性，证实跨IMU通用基坐标系可简化系统标定。为支持仿真到现实的迁移，我们还开源了高保真渲染管线，这对于无真实地面真值数据的训练至关重要。本工作为灵巧手本体感知提供了鲁棒且通用的解决方案，对机器人操作与人机交互具有重要启示意义。https://github.com/huangjund/blenderYCB