Wearable sensor-based Human Action Recognition (HAR) has made significant strides in recent times. However, the accuracy performance of wearable sensor-based HAR is currently still lagging behind that of visual modalities-based systems, such as RGB video and depth data. Although diverse input modalities can provide complementary cues and improve the accuracy performance of HAR, wearable devices can only capture limited kinds of non-visual time series input, such as accelerometers and gyroscopes. This limitation hinders the deployment of multimodal simultaneously using visual and non-visual modality data in parallel on current wearable devices. To address this issue, we propose a novel Physical-aware Cross-modal Adversarial (PCA) framework that utilizes only time-series accelerometer data from four inertial sensors for the wearable sensor-based HAR problem. Specifically, we propose an effective IMU2SKELETON network to produce corresponding synthetic skeleton joints from accelerometer data. Subsequently, we imposed additional constraints on the synthetic skeleton data from a physical perspective, as accelerometer data can be regarded as the second derivative of the skeleton sequence coordinates. After that, the original accelerometer as well as the constrained skeleton sequence were fused together to make the final classification. In this way, when individuals wear wearable devices, the devices can not only capture accelerometer data, but can also generate synthetic skeleton sequences for real-time wearable sensor-based HAR applications that need to be conducted anytime and anywhere. To demonstrate the effectiveness of our proposed PCA framework, we conduct extensive experiments on Berkeley-MHAD, UTD-MHAD, and MMAct datasets. The results confirm that the proposed PCA approach has competitive performance compared to the previous methods on the mono sensor-based HAR classification problem.

翻译：基于可穿戴传感器的人体动作识别（HAR）近期取得了显著进展。然而，当前基于可穿戴传感器的HAR准确率仍落后于基于视觉模态的系统（如RGB视频和深度数据）。尽管多种输入模态可提供互补线索并提升HAR的准确率，但可穿戴设备只能捕获有限类型的非视觉时间序列输入（如加速度计和陀螺仪）。这一限制阻碍了在当前可穿戴设备上并行使用视觉与非视觉模态数据的多模态部署。为解决此问题，我们提出了一种新颖的物理感知跨模态对抗（PCA）框架，仅利用来自四个惯性传感器的加速度计时间序列数据来解决基于可穿戴传感器的HAR问题。具体而言，我们提出了一种有效的IMU2SKELETON网络，从加速度计数据生成对应的合成骨架关节点。随后，我们从物理角度对合成骨架数据施加额外约束——加速度计数据可被视为骨架序列坐标的二阶导数。之后，原始加速度计数据与受约束的骨架序列被融合以进行最终分类。通过这种方式，当用户佩戴可穿戴设备时，设备不仅能捕获加速度计数据，还能实时生成合成骨架序列，适用于需要随时随地进行基于可穿戴传感器的HAR应用。为了验证所提出PCA框架的有效性，我们在Berkeley-MHAD、UTD-MHAD和MMAct数据集上进行了广泛实验。结果表明，在基于单传感器的HAR分类问题上，所提出的PCA方法相比先前方法具有竞争力。