Image-text retrieval (ITR) is a task to retrieve the relevant images/texts, given the query from another modality. The conventional dense retrieval paradigm relies on encoding images and texts into dense representations using dual-stream encoders, however, it faces challenges with low retrieval speed in large-scale retrieval scenarios. In this work, we propose the lexicon-weighting paradigm, where sparse representations in vocabulary space are learned for images and texts to take advantage of the bag-of-words models and efficient inverted indexes, resulting in significantly reduced retrieval latency. A crucial gap arises from the continuous nature of image data, and the requirement for a sparse vocabulary space representation. To bridge this gap, we introduce a novel pre-training framework, Lexicon-Bottlenecked Language-Image Pre-Training (LexLIP), that learns importance-aware lexicon representations. This framework features lexicon-bottlenecked modules between the dual-stream encoders and weakened text decoders, allowing for constructing continuous bag-of-words bottlenecks to learn lexicon-importance distributions. Upon pre-training with same-scale data, our LexLIP achieves state-of-the-art performance on two benchmark ITR datasets, MSCOCO and Flickr30k. Furthermore, in large-scale retrieval scenarios, LexLIP outperforms CLIP with a 5.5 ~ 221.3X faster retrieval speed and 13.2 ~ 48.8X less index storage memory.

翻译：图文检索（ITR）是一项根据另一模态的查询，检索相关图像或文本的任务。传统密集检索范式依赖双流编码器将图像和文本编码为密集表征，但在大规模检索场景中面临检索速度慢的挑战。本文提出词汇权重化范式，通过为图像和文本学习词汇空间中的稀疏表征，利用词袋模型和高效倒排索引的优势，显著降低检索延迟。图像数据的连续性特质与稀疏词汇空间表征需求之间存在关键差距。为弥合这一差距，我们引入新型预训练框架——基于词汇瓶颈的语言-图像预训练（LexLIP），该框架可学习重要性感知的词汇表征。其特色在于双流编码器与弱化文本解码器之间嵌入词汇瓶颈模块，通过构建连续词袋瓶颈来学习词汇重要性分布。在相同规模数据预训练后，LexLIP在MSCOCO和Flickr30k两个基准ITR数据集上达到了最优性能。此外，在大规模检索场景中，LexLIP相比CLIP实现了5.5至221.3倍的检索速度提升，并节省13.2至48.8倍的索引存储内存。