Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences.

翻译：大型语言模型（LLMs）即使在专门训练用于处理长输入上下文时，仍难以捕捉位于输入中间位置的相关信息。这一现象被称为"迷失在中间"问题。本研究作出三项贡献。首先，我们致力于探究导致该现象的因素。通过分析，我们建立了"迷失在中间"现象与LLMs内在注意力偏差之间的联系：LLMs呈现出U形注意力偏差，即输入序列开头和结尾的标记会获得更高关注度，而与其相关性无关。其次，我们通过校准机制"在中间发现"来缓解这种位置偏差，使模型能够根据上下文的相关性准确分配注意力，即使这些信息位于序列中部。第三，我们证明"在中间发现"机制不仅能在长上下文中更有效地定位相关信息，还能最终提升各类任务的检索增强生成（RAG）性能，较现有方法最高可提升15个百分点。这些发现为理解LLM注意力偏差及其潜在影响开辟了新的研究方向。