This paper presents HandFi, which constructs hand skeletons with practical WiFi devices. Unlike previous WiFi hand sensing systems that primarily employ predefined gestures for pattern matching, by constructing the hand skeleton, HandFi can enable a variety of downstream WiFi-based hand sensing applications in gaming, healthcare, and smart homes. Deriving the skeleton from WiFi signals is challenging, especially because the palm is a dominant reflector compared with fingers. HandFi develops a novel multi-task learning neural network with a series of customized loss functions to capture the low-level hand information from WiFi signals. During offline training, HandFi takes raw WiFi signals as input and uses the leap motion to provide supervision. During online use, only with commercial WiFi, HandFi is capable of producing 2D hand masks as well as 3D hand poses. We demonstrate that HandFi can serve as a foundation model to enable developers to build various applications such as finger tracking and sign language recognition, and outperform existing WiFi-based solutions. Artifacts can be found: https://github.com/SIJIEJI/HandFi

翻译：本文提出HandFi系统，该系统利用现有商用WiFi设备构建手部骨架。与以往主要采用预定义手势进行模式匹配的WiFi手部感知系统不同，通过构建手部骨架，HandFi可在游戏、医疗健康和智能家居领域实现多种基于WiFi的下游手部感知应用。从WiFi信号中提取手部骨架极具挑战性，尤其是手掌作为主反射体会显著干扰手指信号。HandFi开发了一种创新的多任务学习神经网络，配备系列定制化损失函数，用于从WiFi信号中捕获底层手部信息。在离线训练阶段，HandFi以原始WiFi信号为输入，借助Leap Motion提供监督信号；在线使用时，仅凭商用WiFi设备即可生成二维手部掩膜与三维手部姿态。实验证明，HandFi可作为基础模型支持开发者构建手指追踪、手语识别等多种应用，且性能优于现有基于WiFi的解决方案。相关代码请见：https://github.com/SIJIEJI/HandFi