Cross-modal retrieval methods are the preferred tool to search databases for the text that best matches a query image and vice versa. However, image-text retrieval models commonly learn to memorize spurious correlations in the training data, such as frequent object co-occurrence, instead of looking at the actual underlying reasons for the prediction in the image. For image-text retrieval, this manifests in retrieved sentences that mention objects that are not present in the query image. In this work, we introduce ODmAP@k, an object decorrelation metric that measures a model's robustness to spurious correlations in the training data. We use automatic image and text manipulations to control the presence of such object correlations in designated test data. Additionally, our data synthesis technique is used to tackle model biases due to spurious correlations of semantically unrelated objects in the training data. We apply our proposed pipeline, which involves the finetuning of image-text retrieval frameworks on carefully designed synthetic data, to three state-of-the-art models for image-text retrieval. This results in significant improvements for all three models, both in terms of the standard retrieval performance and in terms of our object decorrelation metric. The code is available at https://github.com/ExplainableML/Spurious_CM_Retrieval.

翻译：跨模态检索方法是搜索数据库以找到与查询图像最匹配文本的首选工具，反之亦然。然而，图像-文本检索模型通常学会记忆训练数据中的虚假相关性（例如频繁的物体共现），而非关注图像中预测的实际根本原因。在图像-文本检索中，这表现为检索到的句子提到了查询图像中不存在的物体。本文引入ODmAP@k，一种物体去相关度量，用于衡量模型对训练数据中虚假相关性的鲁棒性。我们利用自动图像和文本操作来控制指定测试数据中此类物体相关性的存在。此外，我们的数据合成技术用于解决训练数据中语义无关物体的虚假相关性导致的模型偏差。我们将所提出的流程（涉及在精心设计的合成数据上微调图像-文本检索框架）应用于三种最先进的图像-文本检索模型。结果在所有三个模型上均取得显著改进，既体现在标准检索性能上，也体现在我们的物体去相关度量上。代码见https://github.com/ExplainableML/Spurious_CM_Retrieval。