Composed image retrieval (CIR) is the task of retrieving specific images by using a query that involves both a reference image and a relative caption. Most existing CIR models adopt the late-fusion strategy to combine visual and language features. Besides, several approaches have also been suggested to generate a pseudo-word token from the reference image, which is further integrated into the relative caption for CIR. However, these pseudo-word-based prompting methods have limitations when target image encompasses complex changes on reference image, e.g., object removal and attribute modification. In this work, we demonstrate that learning an appropriate sentence-level prompt for the relative caption (SPRC) is sufficient for achieving effective composed image retrieval. Instead of relying on pseudo-word-based prompts, we propose to leverage pretrained V-L models, e.g., BLIP-2, to generate sentence-level prompts. By concatenating the learned sentence-level prompt with the relative caption, one can readily use existing text-based image retrieval models to enhance CIR performance. Furthermore, we introduce both image-text contrastive loss and text prompt alignment loss to enforce the learning of suitable sentence-level prompts. Experiments show that our proposed method performs favorably against the state-of-the-art CIR methods on the Fashion-IQ and CIRR datasets. The source code and pretrained model are publicly available at https://github.com/chunmeifeng/SPRC

翻译：组合图像检索（CIR）是一项通过包含参考图像和相对描述文本的查询来检索特定图像的任务。现有大多数CIR模型采用晚期融合策略来结合视觉与语言特征。此外，也有若干方法提出从参考图像生成伪词标记，并将其进一步整合到相对描述文本中用于CIR。然而，当目标图像在参考图像上涉及复杂变化（例如物体移除和属性修改）时，这些基于伪词提示的方法存在局限性。本研究表明，学习针对相对描述文本的适当句子级提示（SPRC）足以实现有效的组合图像检索。我们不再依赖基于伪词提示的方法，而是提出利用预训练视觉-语言模型（如BLIP-2）来生成句子级提示。通过将学习到的句子级提示与相对描述文本拼接，即可直接使用现有的基于文本的图像检索模型来提升CIR性能。此外，我们引入了图像-文本对比损失和文本提示对齐损失，以强制学习合适的句子级提示。实验表明，所提方法在Fashion-IQ和CIRR数据集上相较于最先进的CIR方法表现优异。源代码和预训练模型已在https://github.com/chunmeifeng/SPRC公开提供。