Multimodal hand gesture recognition (HGR) systems can achieve higher recognition accuracy. However, acquiring multimodal gesture recognition data typically requires users to wear additional sensors, thereby increasing hardware costs. This paper proposes a novel generative approach to improve Surface Electromyography (sEMG)-based HGR accuracy via virtual Inertial Measurement Unit (IMU) signals. Specifically, we trained a deep generative model based on the intrinsic correlation between forearm sEMG signals and forearm IMU signals to generate virtual forearm IMU signals from the input forearm sEMG signals at first. Subsequently, the sEMG signals and virtual IMU signals were fed into a multimodal Convolutional Neural Network (CNN) model for gesture recognition. To evaluate the performance of the proposed approach, we conducted experiments on 6 databases, including 5 publicly available databases and our collected database comprising 28 subjects performing 38 gestures, containing both sEMG and IMU data. The results show that our proposed approach outperforms the sEMG-based unimodal HGR method (with increases of 2.15%-13.10%). It demonstrates that incorporating virtual IMU signals, generated by deep generative models, can significantly enhance the accuracy of sEMG-based HGR. The proposed approach represents a successful attempt to transition from unimodal HGR to multimodal HGR without additional sensor hardware.

翻译：多模态手势识别系统能够实现更高的识别精度。然而，获取多模态手势识别数据通常需要用户佩戴额外的传感器，从而增加硬件成本。本文提出了一种新颖的生成式方法，通过虚拟惯性测量单元信号提高基于表面肌电信号的手势识别精度。具体而言，我们基于前臂表面肌电信号与前臂惯性测量单元信号之间的内在相关性，首先训练了一个深度生成模型，用于从输入的前臂表面肌电信号生成虚拟的前臂惯性测量单元信号。随后，将表面肌电信号与虚拟惯性测量单元信号输入到多模态卷积神经网络模型中进行手势识别。为评估所提方法的性能，我们在6个数据库上进行了实验，包括5个公开数据库以及我们自行收集的包含28名受试者执行38种手势的数据库（同时包含表面肌电信号与惯性测量单元数据）。结果表明，我们所提出的方法优于基于表面肌电信号的单模态手势识别方法（提升幅度为2.15%-13.10%），证明通过深度生成模型生成的虚拟惯性测量单元信号能够显著提高基于表面肌电信号的手势识别精度。该方法是无需额外传感器硬件便从单模态手势识别成功过渡到多模态手势识别的一次尝试。