Recently, video recognition is emerging with the help of multi-modal learning, which focuses on integrating distinct modalities to improve the performance or robustness of the model. Although various multi-modal learning methods have been proposed and offer remarkable recognition results, almost all of these methods rely on high-quality manual annotations and assume that modalities among multi-modal data provide semantically relevant information. Unfortunately, the widely used video datasets are usually coarse-annotated or collected from the Internet. Thus, it inevitably contains a portion of noisy labels and noisy correspondence. To address this challenge, we use the audio-visual action recognition task as a proxy and propose a noise-tolerant learning framework to find anti-interference model parameters against both noisy labels and noisy correspondence. Specifically, our method consists of two phases that aim to rectify noise by the inherent correlation between modalities. First, a noise-tolerant contrastive training phase is performed to make the model immune to the possible noisy-labeled data. To alleviate the influence of noisy correspondence, we propose a cross-modal noise estimation component to adjust the consistency between different modalities. As the noisy correspondence existed at the instance level, we further propose a category-level contrastive loss to reduce its interference. Second, in the hybrid-supervised training phase, we calculate the distance metric among features to obtain corrected labels, which are used as complementary supervision to guide the training. Extensive experiments on a wide range of noisy levels demonstrate that our method significantly improves the robustness of the action recognition model and surpasses the baselines by a clear margin.

翻译：近年来，视频识别在跨模态学习的助力下蓬勃发展，该领域着重于整合不同模态以提升模型的性能或鲁棒性。尽管已有多种跨模态学习方法被提出并取得了显著识别效果，但几乎所有方法都依赖于高质量的人工标注，且假设跨模态数据间包含语义相关的信息。然而，广泛使用的视频数据集通常标注粗糙或源自网络，不可避免地包含部分噪声标签和噪声对应关系。为解决这一挑战，我们以音频-视觉动作识别任务为代理，提出了一种面向噪声的学习框架，以寻找能够同时抵御噪声标签和噪声对应关系的抗干扰模型参数。具体而言，我们的方法包含两个阶段，旨在通过模态间的内在关联性纠正噪声。首先，执行面向噪声的对比训练阶段，使模型对可能的噪声标注数据免疫。为缓解噪声对应关系的影响，我们提出跨模态噪声估计组件来调整不同模态间的一致性。由于噪声对应关系存在于实例层面，我们进一步提出类别级对比损失以降低其干扰。其次，在混合监督训练阶段，我们计算特征间的距离度量以获得修正标签，并将其作为补充监督信号引导训练。在多种噪声水平上的大量实验表明，我们的方法显著提升了动作识别模型的鲁棒性，并以明显优势超越了基线方法。