Accurate real-time tracking of dexterous hand movements and interactions has numerous applications in human-computer interaction, metaverse, robotics, and tele-health. Capturing realistic hand movements is challenging because of the large number of articulations and degrees of freedom. Here, we report accurate and dynamic tracking of articulated hand and finger movements using stretchable, washable smart gloves with embedded helical sensor yarns and inertial measurement units. The sensor yarns have a high dynamic range, responding to low 0.005 % to high 155 % strains, and show stability during extensive use and washing cycles. We use multi-stage machine learning to report average joint angle estimation root mean square errors of 1.21 and 1.45 degrees for intra- and inter-subjects cross-validation, respectively, matching accuracy of costly motion capture cameras without occlusion or field of view limitations. We report a data augmentation technique that enhances robustness to noise and variations of sensors. We demonstrate accurate tracking of dexterous hand movements during object interactions, opening new avenues of applications including accurate typing on a mock paper keyboard, recognition of complex dynamic and static gestures adapted from American Sign Language and object identification.

翻译：对手部灵巧运动与交互的精确实时追踪在人机交互、元宇宙、机器人学和远程医疗领域具有广泛应用。由于手部关节众多且自由度复杂，捕捉真实的手部运动极具挑战性。本文报道了一种采用嵌入式螺旋传感纱线与惯性测量单元的可拉伸、可水洗智能手套，实现了对手指及手部关节运动的精确动态追踪。该传感纱线具备高动态范围，可响应从0.005%至155%的应变范围，并在长期使用与洗涤周期中保持稳定性。通过多阶段机器学习方法，我们在被试内与被试间交叉验证中分别实现了1.21度与1.45度的平均关节角度估计均方根误差，其精度可与昂贵的动作捕捉相机相媲美，且无遮挡或视野限制问题。我们提出了一种数据增强技术，有效提升了系统对传感器噪声与性能波动的鲁棒性。实验展示了在物体交互过程中对灵巧手部运动的精确追踪，为包括模拟纸质键盘精确打字、基于美国手语改编的复杂动态/静态手势识别以及物体辨识等应用开辟了新途径。