Increasingly popular home assistants are widely utilized as the central controller for smart home devices. However, current designs heavily rely on voice interfaces with accessibility and usability issues; some latest ones are equipped with additional cameras and displays, which are costly and raise privacy concerns. These concerns jointly motivate Beyond-Voice, a novel deep-learning-driven acoustic sensing system that allows commodity home assistant devices to track and reconstruct hand poses continuously. It transforms the home assistant into an active sonar system using its existing onboard microphones and speakers. We feed a high-resolution range profile to the deep learning model that can analyze the motions of multiple body parts and predict the 3D positions of 21 finger joints, bringing the granularity for acoustic hand tracking to the next level. It operates across different environments and users without the need for personalized training data. A user study with 11 participants in 3 different environments shows that Beyond-Voice can track joints with an average mean absolute error of 16.47mm without any training data provided by the testing subject.

翻译：日益普及的家居助手被广泛用作智能家居设备的中央控制器。然而，当前的设计严重依赖语音接口，存在可访问性和可用性问题；部分最新产品配备了额外的摄像头和显示屏，这不仅成本高昂，还引发了隐私担忧。这些问题共同推动了Beyond-Voice这一新型深度学习驱动声学感知系统的诞生，该系统使商用家居助手设备能够持续追踪和重建手部姿态。它利用设备现有的内置麦克风和扬声器，将家居助手转变为主动声呐系统。我们将高分辨率距离剖面输入深度学习模型，该模型能够分析多个身体部位的运动，并预测21个手指关节的三维位置，从而将声学手部追踪的精细度提升至新水平。该系统无需个性化训练数据，即可在不同环境和用户间运行。针对3种不同环境中11名参与者的用户研究表明，Beyond-Voice能够在测试对象未提供任何训练数据的情况下，以平均绝对误差16.47毫米追踪关节。