Human Activity Recognition (HAR) systems have been extensively studied by the vision and ubiquitous computing communities due to their practical applications in daily life, such as smart homes, surveillance, and health monitoring. Typically, this process is supervised in nature and the development of such systems requires access to large quantities of annotated data. However, the higher costs and challenges associated with obtaining good quality annotations have rendered the application of self-supervised methods an attractive option and contrastive learning comprises one such method. However, a major component of successful contrastive learning is the selection of good positive and negative samples. Although positive samples are directly obtainable, sampling good negative samples remain a challenge. As human activities can be recorded by several modalities like camera and IMU sensors, we propose a hard negative sampling method for multimodal HAR with a hard negative sampling loss for skeleton and IMU data pairs. We exploit hard negatives that have different labels from the anchor but are projected nearby in the latent space using an adjustable concentration parameter. Through extensive experiments on two benchmark datasets: UTD-MHAD and MMAct, we demonstrate the robustness of our approach forlearning strong feature representation for HAR tasks, and on the limited data setting. We further show that our model outperforms all other state-of-the-art methods for UTD-MHAD dataset, and self-supervised methods for MMAct: Cross session, even when uni-modal data are used during downstream activity recognition.

翻译：人体活动识别（HAR）系统因在日常生活中具有实际应用价值（如智能家居、监控和健康监测）而受到视觉与普适计算领域的广泛研究。通常，该过程本质上是监督式的，这类系统的开发需要获取大量标注数据。然而，获取高质量标注的高昂成本和挑战使得自监督方法成为具有吸引力的选择，而对比学习正是这类方法之一。然而，成功对比学习的关键组成部分是正负样本的良好选择。尽管正样本可直接获取，但采样优质负样本仍是一大挑战。由于人体活动可通过多种模态（如摄像头和IMU传感器）记录，我们针对骨骼与IMU数据对提出了一种基于硬负样本采样损失的 multimodal HAR 硬负样本采样方法。我们利用与锚点样本标签不同、但在潜在空间中利用可调节浓度参数被投影至邻近位置的硬负样本。通过在UTD-MHAD和MMAct两个基准数据集上的广泛实验，我们证明了该方法在HAR任务中学习强特征表示的鲁棒性，尤其在数据有限的情况下。我们进一步表明，即使在下游活动识别中仅使用单模态数据，我们的模型在UTD-MHAD数据集上优于所有其他最先进方法，并在MMAct数据集跨会话设置中优于自监督方法。