Most approaches to cross-modal retrieval (CMR) focus either on object-centric datasets, meaning that each document depicts or describes a single object, or on scene-centric datasets, meaning that each image depicts or describes a complex scene that involves multiple objects and relations between them. We posit that a robust CMR model should generalize well across both dataset types. Despite recent advances in CMR, the reproducibility of the results and their generalizability across different dataset types has not been studied before. We address this gap and focus on the reproducibility of the state-of-the-art CMR results when evaluated on object-centric and scene-centric datasets. We select two state-of-the-art CMR models with different architectures: (i) CLIP; and (ii) X-VLM. Additionally, we select two scene-centric datasets, and three object-centric datasets, and determine the relative performance of the selected models on these datasets. We focus on reproducibility, replicability, and generalizability of the outcomes of previously published CMR experiments. We discover that the experiments are not fully reproducible and replicable. Besides, the relative performance results partially generalize across object-centric and scene-centric datasets. On top of that, the scores obtained on object-centric datasets are much lower than the scores obtained on scene-centric datasets. For reproducibility and transparency we make our source code and the trained models publicly available.

翻译：大多数跨模态检索（CMR）方法要么聚焦于以对象为中心的数据集（即每份文档描述或描绘单一对象），要么聚焦于以场景为中心的数据集（即每张图像描述或描绘包含多个对象及其关系的复杂场景）。我们认为，一个稳健的CMR模型应在两类数据集上均具有良好的泛化能力。尽管近年来CMR领域取得了显著进展，但研究结果的跨数据集类型可重复性与泛化性此前尚未被探讨。本文针对这一空白，聚焦于评估当前最优CMR方法在以对象为中心和以场景为中心的数据集上的可重复性。我们选取了两种架构不同的最新CMR模型：（i）CLIP；以及（ii）X-VLM。此外，我们选用两个以场景为中心的数据集和三个以对象为中心的数据集，测定所选模型在这些数据集上的相对性能。我们重点关注先前已发表CMR实验结果的可用性、可复现性及泛化能力。研究发现，这些实验无法被完全复现与复刻。此外，相对性能结果仅在部分程度上泛化于两类数据集之间。值得注意的是，以对象为中心的数据集上的得分显著低于以场景为中心的数据集。为促进可重复性与透明度，我们将源代码与训练后的模型公开提供。