Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystack (NIAH) test, which involves retrieving a "needle" (relevant information) from a "haystack" (long irrelevant context). Extensions of this approach include increasing distractors, fact chaining, and in-context reasoning. However, in these benchmarks, models can exploit existing literal matches between the needle and haystack to simplify the task. To address this, we introduce NoLiMa, a benchmark extending NIAH with a carefully designed needle set, where questions and needles have minimal lexical overlap, requiring models to infer latent associations to locate the needle within the haystack. We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information. We publicly release the dataset and evaluation code at https://github.com/adobe-research/NoLiMa.

翻译：近期的大型语言模型（LLMs）已支持128K至1M令牌的长上下文处理能力。评估此类能力的主流方法是“大海捞针”（NIAH）测试，即从“干草堆”（长段无关上下文）中检索“针”（相关信息）。该方法的扩展包括增加干扰项、事实链式推理及上下文推理等。然而，在这些基准测试中，模型可能利用针与干草堆之间已有的字面匹配来简化任务。为此，我们提出NoLiMa基准，通过精心设计的针集合对NIAH进行扩展：该集合中问题与针的词汇重叠度极低，要求模型通过推断潜在关联在干草堆中定位目标针。我们对12款宣称支持至少128K上下文的主流LLMs进行评估。这些模型在短上下文（<1K）中表现良好，但随着上下文长度增加，性能显著下降。例如在32K长度下，10款模型的性能降至其强劲短长度基线的50%以下。即使是表现最优异的GPT-4o，其准确率也从接近完美的基线值99.3%下降至69.7%。分析表明，当缺乏字面匹配时，注意力机制在长上下文中面临更大挑战，导致相关信息检索困难加剧。我们已公开数据集与评估代码：https://github.com/adobe-research/NoLiMa。