This paper introduces Helios, the first extremely low-power, real-time, event-based hand gesture recognition system designed for all-day on smart eyewear. As augmented reality (AR) evolves, current smart glasses like the Meta Ray-Bans prioritize visual and wearable comfort at the expense of functionality. Existing human-machine interfaces (HMIs) in these devices, such as capacitive touch and voice controls, present limitations in ergonomics, privacy and power consumption. Helios addresses these challenges by leveraging natural hand interactions for a more intuitive and comfortable user experience. Our system utilizes a extremely low-power and compact 3mmx4mm/20mW event camera to perform natural hand-based gesture recognition for always-on smart eyewear. The camera's output is processed by a convolutional neural network (CNN) running on a NXP Nano UltraLite compute platform, consuming less than 350mW. Helios can recognize seven classes of gestures, including subtle microgestures like swipes and pinches, with 91% accuracy. We also demonstrate real-time performance across 20 users at a remarkably low latency of 60ms. Our user testing results align with the positive feedback we received during our recent successful demo at AWE-USA-2024.

翻译：本文介绍了Helios，这是首个专为全天候智能眼镜设计的极低功耗、实时、事件驱动的手势识别系统。随着增强现实（AR）技术的发展，当前如Meta Ray-Bans等智能眼镜优先考虑视觉与佩戴舒适性，却牺牲了功能性。这些设备中现有的人机交互界面（HMI），如电容触控和语音控制，在人体工学、隐私保护和功耗方面存在局限。Helios通过利用自然的手部交互来解决这些挑战，为用户提供更直观、舒适的体验。我们的系统采用一款极低功耗、紧凑型（3mm×4mm/20mW）的事件相机，为全天候智能眼镜实现基于自然手部动作的手势识别。相机输出由在NXP Nano UltraLite计算平台上运行的卷积神经网络（CNN）处理，整机功耗低于350mW。Helios能够识别七类手势，包括滑动、捏合等精细微手势，准确率达到91%。我们还在20名用户中验证了其实时性能，延迟低至60ms。用户测试结果与我们近期在AWE-USA-2024上成功演示所获得的积极反馈一致。