Recent reports suggest that LLMs can handle increasingly long contexts. However, many existing benchmarks for context understanding embed substantial query-irrelevant content, which shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information. Under this setting, we view that current benchmarks can overestimate true context-understanding ability of LLMs. In particular, we demonstrate that when the context consists entirely of query-relevant text, even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens. To evaluate this capability more rigorously, we introduce NeedleChain, a benchmark designed to test whether models can faithfully incorporate all given evidence. NeedleChain includes three variants that differ in the required order of comprehension, along with a parallel benchmark based on the needle-in-a-haystack(NIAH) paradigm. By comparing these variants, NeedleChain enables a more comprehensive assessment of context understanding. We further propose a training-free strategy that encourages models to reflect all available information, ROPE contraction, highlighting the importance of full-context integration and pointing to new directions for improving reliable reasoning over context.

翻译：近期研究表明，大语言模型能够处理日益增长的上下文长度。然而，现有许多上下文理解基准测试中嵌入了大量与查询无关的内容，这使得评估重点转向检索相关片段，而非完全整合所有提供的信息。在此设定下，我们认为当前基准测试可能高估了大语言模型真实的上下文理解能力。具体而言，我们证明当上下文完全由与查询相关的文本构成时，即使是GPT-4o等先进模型也难以可靠地整合短至200个词元的输入。为更严格地评估此能力，我们提出了NeedleChain——一个专门测试模型能否忠实整合所有给定证据的基准测试。NeedleChain包含三种变体，其区别在于所需的理解顺序，同时提供了一个基于“大海捞针”范式的并行基准。通过比较这些变体，NeedleChain能够对上下文理解进行更全面的评估。我们进一步提出了一种无需训练的策略——ROPE收缩，以鼓励模型反映所有可用信息，这凸显了全上下文整合的重要性，并为提升基于上下文的可靠推理指出了新的研究方向。