Smart eyewear enables unobtrusive, context-aware interaction through multimodal sensors and on-device intelligence, but is severely limited by power, memory, and compute constraints in a compact form factor. Open-hardware platforms supporting event-based vision and embedded ML at this scale are rare. This work introduces an open-source smart glasses platform for rapid prototyping of novel sensors and algorithms. Its modular design uses a flexible FPC interposer to support both event-based and frame-based cameras without full PCB redesign. A hardware-software co-designed power management system combines a configurable PMIC with event-driven wake-up via an nRF5340 coordinator, keeping the GAP9 RISC-V SoC powered down between inferences. The prototype achieves up to 11.8 hours of continuous on-device ML from a 200 mAh battery. As a demonstration, an egocentric hand gesture recognition pipeline was evaluated on the LynX dataset using polarity-separated event histograms from a Prophesee GENX320 camera. R(2+1)D achieved the best cross-subject accuracy of 83.94\% (macro F1 = 0.781) under leave-two-subjects-out validation, with 33.9 ms end-to-end latency on the GAP9. Temporal augmentation and removal of ambiguous classes provided the largest gains (+8.9 pp). All hardware designs, firmware, and models are released open source.
翻译:摘要:智能眼镜通过多模态传感器和端侧智能实现了无干扰的上下文感知交互,但其紧凑形态下的功耗、内存和计算资源严重受限。支持事件视觉与嵌入式机器学习的大规模开源硬件平台极为罕见。本文提出一种用于新型传感器与算法快速原型开发的开源智能眼镜平台。其模块化设计采用柔性FPC基板实现事件相机与帧相机的兼容切换,无需重新设计PCB板。通过硬件-软件协同设计的电源管理系统,将可配置PMIC与nRF5340协处理器的事件驱动唤醒机制相结合,使GAP9 RISC-V SoC在推理间隔期间保持断电状态。样机在200 mAh电池供电下可实现长达11.8小时的连续端侧机器学习。作为验证,基于LynX数据集构建了以自我为中心的手势识别流程,采用Prophesee GENX320相机提取极性分离事件直方图。在留二受试者交叉验证中,R(2+1)D模型取得了83.94%的最佳跨受试者准确率(宏F1=0.781),在GAP9上的端到端推理延迟为33.9毫秒。时间增强与模糊类别剔除贡献了最大性能提升(+8.9个百分点)。所有硬件设计、固件与模型均已开源发布。