Retrieval augmented generation (RAG) has been widely adopted to help Large Language Models (LLMs) to process tasks involving long documents. However, existing retrieval models are not designed for long document retrieval and fail to address several key challenges of long document retrieval, including context-awareness, causal dependence, and scope of retrieval. In this paper, we proposed AttentionRetriever, a novel long document retrieval model that leverages attention mechanism and entity-based retrieval to build context-aware embeddings for long document and determine the scope of retrieval. With extensive experiments, we found AttentionRetriever is able to outperform existing retrieval models on long document retrieval datasets by a large margin while remaining as efficient as dense retrieval models.

翻译：检索增强生成（RAG）已被广泛采用，以协助大型语言模型（LLM）处理涉及长文档的任务。然而，现有检索模型并非为长文档检索而设计，未能解决长文档检索中的若干关键挑战，包括上下文感知、因果依赖以及检索范围。本文提出AttentionRetriever，一种新颖的长文档检索模型，它利用注意力机制与基于实体的检索，为长文档构建上下文感知的嵌入表示，并确定检索范围。通过大量实验，我们发现AttentionRetriever在长文档检索数据集上能够显著超越现有检索模型，同时保持与稠密检索模型相当的效率。