Despite their potential, markerless hand tracking technologies are not yet applied in practice to the diagnosis or monitoring of the activity in inflammatory musculoskeletal diseases. One reason is that the focus of most methods lies in the reconstruction of coarse, plausible poses for gesture recognition or AR/VR applications, whereas in the clinical context, accurate, interpretable, and reliable results are required. Therefore, we propose ShaRPy, the first RGB-D Shape Reconstruction and hand Pose tracking system, which provides uncertainty estimates of the computed pose to guide clinical decision-making. Our method requires only a light-weight setup with a single consumer-level RGB-D camera yet it is able to distinguish similar poses with only small joint angle deviations. This is achieved by combining a data-driven dense correspondence predictor with traditional energy minimization, optimizing for both, pose and hand shape parameters. We evaluate ShaRPy on a keypoint detection benchmark and show qualitative results on recordings of a patient.

翻译：尽管无标记手部追踪技术潜力巨大，但尚未实际应用于炎症性肌肉骨骼疾病的活动性诊断或监测。原因之一是多数方法侧重于为手势识别或增强/虚拟现实应用重建粗粒度、合理的姿态，而临床场景需要精确、可解释且可靠的结果。为此，我们提出ShaRPy——首个基于RGB-D的形状重建与手部姿态追踪系统，该系统可提供计算姿态的不确定性估计以指导临床决策。我们的方法仅需单台消费级RGB-D摄像机构建轻量级设备，却能够区分仅存在微小关节角度偏差的相似姿态。这通过将数据驱动的密集对应预测器与传统能量最小化相结合，同时优化姿态与手部形状参数来实现。我们在关键点检测基准上评估了ShaRPy，并展示了患者记录数据的定性结果。