While large language models (LLMs) are equipped with longer text input capabilities than before, they are struggling to seek correct information in long contexts. The "lost in the middle" problem challenges most LLMs, referring to the dramatic decline in accuracy when correct information is located in the middle. To overcome this crucial issue, this paper proposes to enhance the information searching and reflection ability of LLMs in long contexts via specially designed tasks called Attention Strengthening Multi-doc QA (ASM QA). Following these tasks, our model excels in focusing more precisely on the desired information. Experimental results show substantial improvement in Multi-doc QA and other benchmarks, superior to state-of-the-art models by 13.7% absolute gain in shuffled settings, by 21.5% in passage retrieval task. We release our model, Ziya-Reader to promote related research in the community.

翻译：尽管大型语言模型（LLMs）具备了比以往更长的文本输入能力，但其在长上下文中检索正确信息时仍面临困难。“迷失于中段”问题困扰着大多数LLMs，即当正确信息位于文本中部时，模型准确率会出现显著下降。为克服这一关键问题，本文提出通过专门设计的“注意力增强多文档问答”任务，增强LLMs在长上下文中的信息检索与反思能力。基于这些任务，我们的模型能够更精准地聚焦目标信息。实验结果表明，该模型在多文档问答及其他基准测试中取得显著提升：在乱序设置下以13.7%的绝对优势超越现有最优模型，在段落检索任务中领先21.5%。我们开源了Ziya-Reader模型以推动相关领域研究。