Vector embeddings from pre-trained language models form a core component in Neural Information Retrieval systems across a multitude of knowledge extraction tasks. The paradigm of late interaction, introduced in ColBERT, demonstrates high accuracy along with runtime efficiency. However, the current formulation fails to take into account the attention weights of query and document terms, which intuitively capture the "importance" of similarities between them, that might lead to a better understanding of relevance between the queries and documents. This work proposes ColBERT-Att, to explicitly integrate attention mechanism into the late interaction framework for enhanced retrieval performance. Empirical evaluation of ColBERT-Att depicts improvements in recall accuracy on MS-MARCO as well as on a wide range of BEIR and LoTTE benchmark datasets.

翻译：摘要：来自预训练语言模型的向量嵌入构成了神经信息检索系统在众多知识抽取任务中的核心组件。ColBERT中引入的延迟交互范式在运行时效率的同时展现了高准确性。然而，当前公式未能考虑查询和文档词项的注意力权重，这些权重直观地捕捉了它们之间相似性的“重要性”，这可能有助于更好地理解查询与文档之间的相关性。本文提出ColBERT-Att，旨在显式地将注意力机制整合到延迟交互框架中，以提升检索性能。对ColBERT-Att的实证评估表明，其在MS-MARCO以及广泛的BEIR和LoTTE基准数据集上均能提升召回准确性。