Recent advancements in Large Language Models (LLMs) underscore the necessity of Retrieval Augmented Generation (RAG) to leverage external information. However, LLMs are sensitive to the position of relevant information within contexts and tend to generate incorrect responses when such information is placed in the middle, known as `Lost in the Middle' phenomenon. In this paper, we introduce a framework that generates consistent outputs for decoder-only models, irrespective of the input context order. Experimental results for three open domain question answering tasks demonstrate position invariance, where the model is not sensitive to input context order, and superior robustness to irrelevent passages compared to prevailing approaches for RAG pipelines.

翻译：近年来，大型语言模型（LLMs）的发展突显了检索增强生成（RAG）在利用外部信息方面的必要性。然而，LLMs对上下文中的相关信息位置非常敏感，当相关信息位于中间位置时，模型倾向于生成错误答案，这种现象被称为“迷失于中间”。本文提出了一种框架，能够为仅解码器模型生成一致的输出，而不受输入上下文顺序的影响。在三个开放域问答任务上的实验结果表明，该框架具有位置不变性（即模型对输入上下文顺序不敏感），并且与当前主流的RAG流程方法相比，对无关段落具有更强的鲁棒性。