Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually catching up. However, existing benchmarks employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-context applications. To bridge this gap, we propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Unlike typical document QA, in Loong's test cases, each document is relevant to the final answer, ignoring any document will lead to the failure of the answer. Furthermore, Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Extensive experiments indicate that existing long-context language models still exhibit considerable potential for enhancement. Retrieval augmented generation (RAG) achieves poor performance, demonstrating that Loong can reliably assess the model's long-context modeling capabilities.

翻译：长上下文建模能力已引起广泛关注，催生了具备超长上下文窗口的大语言模型（LLMs）。与此同时，用于评估长上下文LLMs的基准测试体系也在逐步完善。然而，现有基准测试普遍采用无关噪声文本人为扩展测试用例长度，这与长上下文应用的实际场景存在偏差。为弥补这一差距，我们提出一种新型长上下文基准测试——Loong，其通过扩展多文档问答（QA）任务实现与现实场景的对齐。与典型文档QA不同，在Loong的测试用例中，每份文档均与最终答案相关，忽略任意文档都将导致答案错误。此外，Loong引入涵盖多种上下文长度的四类任务：焦点定位、对比分析、聚类归纳和推理链构建，以促进对长上下文理解能力更真实、更全面的评估。大量实验表明，现有长上下文语言模型仍具有显著的提升空间。检索增强生成（RAG）方法表现欠佳，这证明Loong能够可靠评估模型的长上下文建模能力。