As one of the most fundamental techniques in multimodal learning, cross-modal matching aims to project various sensory modalities into a shared feature space. To achieve this, massive and correctly aligned data pairs are required for model training. However, unlike unimodal datasets, multimodal datasets are extremely harder to collect and annotate precisely. As an alternative, the co-occurred data pairs (e.g., image-text pairs) collected from the Internet have been widely exploited in the area. Unfortunately, the cheaply collected dataset unavoidably contains many mismatched data pairs, which have been proven to be harmful to the model's performance. To address this, we propose a general framework called BiCro (Bidirectional Cross-modal similarity consistency), which can be easily integrated into existing cross-modal matching models and improve their robustness against noisy data. Specifically, BiCro aims to estimate soft labels for noisy data pairs to reflect their true correspondence degree. The basic idea of BiCro is motivated by that -- taking image-text matching as an example -- similar images should have similar textual descriptions and vice versa. Then the consistency of these two similarities can be recast as the estimated soft labels to train the matching model. The experiments on three popular cross-modal matching datasets demonstrate that our method significantly improves the noise-robustness of various matching models, and surpass the state-of-the-art by a clear margin.

翻译：作为多模态学习中最基础的技术之一，跨模态匹配旨在将多种感官模态投影到共享特征空间。为此，模型训练需要大量且准确对齐的数据对。然而，与单模态数据集不同，多模态数据集的精确收集和标注极为困难。作为一种替代方案，从互联网收集的共现数据对（如图像-文本对）已在该领域得到广泛应用。但令人遗憾的是，这种低成本收集的数据集不可避免地包含大量错误匹配的数据对，这些数据已被证明会损害模型性能。为解决这一问题，我们提出一个名为BiCro（双向跨模态相似性一致性）的通用框架，该框架可轻松集成至现有跨模态匹配模型，提升其对噪声数据的鲁棒性。具体而言，BiCro致力于为噪声数据对估计软标签，以反映其真实对应程度。该框架的基本思想源于——以图像-文本匹配为例——相似图像应具有相似的文本描述，反之亦然。进而，这两种相似性的一致性可重构为训练匹配模型的估计软标签。在三个主流跨模态匹配数据集上的实验表明，我们的方法显著提升了多种匹配模型的抗噪鲁棒性，并以明显优势超越了当前最优方法。