In the recent years, the dual-encoder vision-language models (\eg CLIP) have achieved remarkable text-to-image retrieval performance. However, we discover that these models usually results in very different retrievals for a pair of paraphrased queries. Such behavior might render the retrieval system less predictable and lead to user frustration. In this work, we consider the task of paraphrased text-to-image retrieval where a model aims to return similar results given a pair of paraphrased queries. To start with, we collect a dataset of paraphrased image descriptions to facilitate quantitative evaluation for this task. We then hypothesize that the undesired behavior of existing dual-encoder model is due to their text towers which are trained on image-sentence pairs and lack the ability to capture the semantic similarity between paraphrased queries. To improve on this, we investigate multiple strategies for training a dual-encoder model starting from a language model pretrained on a large text corpus. Compared to public dual-encoder models such as CLIP and OpenCLIP, the model trained with our best adaptation strategy achieves a significantly higher ranking similarity for paraphrased queries while maintaining similar zero-shot classification and retrieval accuracy.

翻译：近年来，双编码器视觉-语言模型（如CLIP）在图文检索任务中取得了显著成效。然而我们发现，这类模型对一对释义查询往往会返回差异极大的检索结果。这种表现可能会降低检索系统的可预测性，并引发用户挫败感。本研究聚焦于释义文本到图像检索任务，即要求模型对一对释义查询返回相似结果。首先，我们构建了一个包含释义图像描述的数据集，以便对该任务进行量化评估。进而我们假设：现有双编码器模型的异常表现源于其文本编码器——这些编码器基于图像-句子对训练，缺乏捕捉释义查询间语义相似性的能力。为解决该问题，我们探索了多种训练策略，从预训练于大型文本语料库的语言模型出发，构建双编码器模型。相较于CLIP、OpenCLIP等公开双编码器模型，采用最优适配策略训练的模型在保持相近零样本分类与检索精度的前提下，对释义查询的排序相似性获得了显著提升。