Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despite recent strides in making LLMs process contexts with more than 100K tokens, there is currently a lack of a standardized benchmark to evaluate this long-context capability. Existing public benchmarks typically focus on contexts around 10K tokens, limiting the assessment and comparison of LLMs in processing longer contexts. In this paper, we propose $\infty$Bench, the first LLM benchmark featuring an average data length surpassing 100K tokens. $\infty$Bench comprises synthetic and realistic tasks spanning diverse domains, presented in both English and Chinese. The tasks in $\infty$Bench are designed to require well understanding of long dependencies in contexts, and make simply retrieving a limited number of passages from contexts not sufficient for these tasks. In our experiments, based on $\infty$Bench, we evaluate the state-of-the-art proprietary and open-source LLMs tailored for processing long contexts. The results indicate that existing long context LLMs still require significant advancements to effectively process 100K+ context. We further present three intriguing analyses regarding the behavior of LLMs processing long context.

翻译：处理和理解长上下文对于大型语言模型（LLMs）的许多实际应用至关重要，例如文档理解和智能体构建。尽管近期在使LLMs能够处理超过十万词上下文的方面取得了进展，但目前缺乏一个标准化的基准来评估这种长上下文能力。现有的公开基准通常聚焦于约一万词左右的上下文，限制了在更长上下文情境下对LLMs的评估与比较。本文提出$\infty$Bench，这是首个平均数据长度超过十万词的LLM基准。$\infty$Bench包含跨多个领域的合成任务与真实任务，并以中英文两种语言呈现。$\infty$Bench中的任务设计需充分理解上下文中的长距离依赖关系，仅从上下文中简单检索有限段落不足以完成这些任务。基于$\infty$Bench，我们对当前最先进的专有和开源长上下文LLMs进行了实验评估。结果表明，现有的长上下文LLMs仍需显著改进才能有效处理十万词以上的上下文。我们进一步提供了三项关于LLMs处理长上下文行为的有趣分析。