The task of open-vocabulary object-centric image retrieval involves the retrieval of images containing a specified object of interest, delineated by an open-set text query. As working on large image datasets becomes standard, solving this task efficiently has gained significant practical importance. Applications include targeted performance analysis of retrieved images using ad-hoc queries and hard example mining during training. Recent advancements in contrastive-based open vocabulary systems have yielded remarkable breakthroughs, facilitating large-scale open vocabulary image retrieval. However, these approaches use a single global embedding per image, thereby constraining the system's ability to retrieve images containing relatively small object instances. Alternatively, incorporating local embeddings from detection pipelines faces scalability challenges, making it unsuitable for retrieval from large databases. In this work, we present a simple yet effective approach to object-centric open-vocabulary image retrieval. Our approach aggregates dense embeddings extracted from CLIP into a compact representation, essentially combining the scalability of image retrieval pipelines with the object identification capabilities of dense detection methods. We show the effectiveness of our scheme to the task by achieving significantly better results than global feature approaches on three datasets, increasing accuracy by up to 15 mAP points. We further integrate our scheme into a large scale retrieval framework and demonstrate our method's advantages in terms of scalability and interpretability.

翻译：开放词汇的面向对象图像检索任务旨在检索包含指定感兴趣对象的图像，该对象由开放集文本查询定义。随着处理大规模图像数据集成为常态，高效解决这一任务具有重要的实际意义，其应用包括使用临时查询对检索图像进行定向性能分析以及训练中的困难样本挖掘。基于对比学习的开放词汇系统的最新进展取得了显著突破，推动了大规模开放词汇图像检索的发展。然而，这类方法对每张图像仅使用单一全局嵌入，从而限制了系统检索包含较小对象实例的图像的能力。另一方面，引入检测流程中的局部嵌入面临可扩展性挑战，使其不适合从大型数据库中进行检索。本文提出了一种简单而有效的面向对象的开放词汇图像检索方法。该方法将CLIP提取的密集嵌入聚合为紧凑表示，实质上结合了图像检索流程的可扩展性与密集检测方法的目标识别能力。我们在三个数据集上的实验表明，该方法相比全局特征方法取得了显著更优的结果，准确率提升高达15个mAP点。此外，我们将该方法集成到大规模检索框架中，并展示了其在可扩展性和可解释性方面的优势。