Despite their potential, markerless hand tracking technologies are not yet applied in practice to the diagnosis or monitoring of the activity in inflammatory musculoskeletal diseases. One reason is that the focus of most methods lies in the reconstruction of coarse, plausible poses, whereas in the clinical context, accurate, interpretable, and reliable results are required. Therefore, we propose ShaRPy, the first RGB-D Shape Reconstruction and hand Pose tracking system, which provides uncertainty estimates of the computed pose, e.g., when a finger is hidden or its estimate is inconsistent with the observations in the input, to guide clinical decision-making. Besides pose, ShaRPy approximates a personalized hand shape, promoting a more realistic and intuitive understanding of its digital twin. Our method requires only a light-weight setup with a single consumer-level RGB-D camera yet it is able to distinguish similar poses with only small joint angle deviations in a metrically accurate space. This is achieved by combining a data-driven dense correspondence predictor with traditional energy minimization. To bridge the gap between interactive visualization and biomedical simulation we leverage a parametric hand model in which we incorporate biomedical constraints and optimize for both, its pose and hand shape. We evaluate ShaRPy on a keypoint detection benchmark and show qualitative results of hand function assessments for activity monitoring of musculoskeletal diseases.

翻译：尽管无标记手势追踪技术潜力巨大，但尚未实际应用于炎症性肌肉骨骼疾病的活动诊断或监测。其原因在于：现有方法多侧重于粗略合理姿态的重建，而临床场景要求结果具有准确性、可解释性与可靠性。为此，我们提出ShaRPy——首个同时实现RGB-D形状重建与手势姿态追踪的系统。该系统能为计算出的姿态提供不确定性估计（例如当某根手指被遮挡或其估计值与输入观测不一致时），从而辅助临床决策。除姿态外，ShaRPy还能逼近个性化手部形状，促进对其数字孪生的更真实直观理解。我们的方法仅需单个消费级RGB-D摄像头的轻量化设备，即可在度量精度空间内区分仅存在微小关节角度偏差的相似姿态。这一效果通过将数据驱动的密集对应预测器与传统能量最小化方法相结合而实现。为弥合交互式可视化与生物医学模拟之间的鸿沟，我们采用参数化手部模型，在其中融入生物力学约束，并同时优化其姿态与手部形状。我们在关键点检测基准上评估ShaRPy，并展示了手功能评估用于肌肉骨骼疾病活动监测的定性结果。