Deep neural networks (DNNs) have been applied in many computer vision tasks and achieved state-of-the-art (SOTA) performance. However, misclassification will occur when DNNs predict adversarial examples which are created by adding human-imperceptible adversarial noise to natural examples. This limits the application of DNN in security-critical fields. In order to enhance the robustness of models, previous research has primarily focused on the unimodal domain, such as image recognition and video understanding. Although multi-modal learning has achieved advanced performance in various tasks, such as action recognition, research on the robustness of RGB-skeleton action recognition models is scarce. In this paper, we systematically investigate how to improve the robustness of RGB-skeleton action recognition models. We initially conducted empirical analysis on the robustness of different modalities and observed that the skeleton modality is more robust than the RGB modality. Motivated by this observation, we propose the \formatword{A}ttention-based \formatword{M}odality \formatword{R}eweighter (\formatword{AMR}), which utilizes an attention layer to re-weight the two modalities, enabling the model to learn more robust features. Our AMR is plug-and-play, allowing easy integration with multimodal models. To demonstrate the effectiveness of AMR, we conducted extensive experiments on various datasets. For example, compared to the SOTA methods, AMR exhibits a 43.77\% improvement against PGD20 attacks on the NTU-RGB+D 60 dataset. Furthermore, it effectively balances the differences in robustness between different modalities.

翻译：深度神经网络（DNNs）已在众多计算机视觉任务中得到应用并取得了最先进的性能。然而，当DNNs预测那些通过在自然样本上添加人眼难以察觉的对抗性噪声而生成的对抗样本时，会发生误分类。这限制了DNN在安全关键领域的应用。为增强模型的鲁棒性，先前的研究主要集中在单模态领域，如图像识别和视频理解。尽管多模态学习在动作识别等多种任务中已取得先进性能，但针对RGB-骨架动作识别模型鲁棒性的研究却十分匮乏。本文系统性地研究了如何提升RGB-骨架动作识别模型的鲁棒性。我们首先对不同模态的鲁棒性进行了实证分析，发现骨架模态比RGB模态具有更强的鲁棒性。受此观察启发，我们提出了基于注意力的模态重加权器（Attention-based Modality Reweighter, AMR），它利用注意力层对两种模态进行重新加权，使模型能够学习到更具鲁棒性的特征。我们的AMR模块即插即用，可轻松集成到多模态模型中。为验证AMR的有效性，我们在多个数据集上进行了大量实验。例如，相较于最先进方法，AMR在NTU-RGB+D 60数据集上针对PGD20攻击的鲁棒性提升了43.77%。此外，它还能有效平衡不同模态间鲁棒性的差异。