Image and text retrieval is one of the foundational tasks in the vision and language domain with multiple real-world applications. State-of-the-art approaches, e.g. CLIP, ALIGN, represent images and texts as dense embeddings and calculate the similarity in the dense embedding space as the matching score. On the other hand, sparse semantic features like bag-of-words models are more interpretable, but believed to suffer from inferior accuracy than dense representations. In this work, we show that it is possible to build a sparse semantic representation that is as powerful as, or even better than, dense presentations. We extend the CLIP model and build a sparse text and image representation (STAIR), where the image and text are mapped to a sparse token space. Each token in the space is a (sub-)word in the vocabulary, which is not only interpretable but also easy to integrate with existing information retrieval systems. STAIR model significantly outperforms a CLIP model with +$4.9\%$ and +$4.3\%$ absolute Recall@1 improvement on COCO-5k text$\rightarrow$image and image$\rightarrow$text retrieval respectively. It also achieved better performance on both of ImageNet zero-shot and linear probing compared to CLIP.

翻译：图像与文本检索是视觉与语言领域的基础任务之一，具有广泛的实际应用。当前最先进的方法（如CLIP、ALIGN）将图像和文本表示为密集嵌入，并通过密集嵌入空间中的相似度计算匹配分数。另一方面，词袋模型等稀疏语义特征更具可解释性，但通常被认为在准确率上不及密集表示。本研究证明，构建与密集表示性能相当甚至更优的稀疏语义表示是可行的。我们扩展了CLIP模型，构建了稀疏文本与图像表示（STAIR），将图像和文本映射至稀疏词元空间。该空间中的每个词元均为词汇表中的（子）词，不仅具备可解释性，且易于与现有信息检索系统集成。在COCO-5k数据集的文本→图像和图像→文本检索任务中，STAIR模型相较CLIP分别实现了Recall@1指标绝对提升4.9%和4.3%；在ImageNet零样本与线性探测任务中，其性能亦优于CLIP。