We introduce SciTrek, a novel question-answering benchmark designed to evaluate long-context reasoning capabilities of large language models (LLMs) using scientific articles. Current long-context benchmarks often focus on simple information retrieval tasks, or employ artificial contexts. SciTrek addresses these limitations by creating benchmark questions that require information aggregation and synthesis across multiple full-text scientific articles. The questions and their ground-truth answers are automatically generated by formulating them as SQL queries over a database constructed from article metadata (i.e., titles, authors, and references). These SQL queries provide explicit, verifiable reasoning processes that enable fine-grained error analysis on model answers, and the data construction scales to contexts of up to 1M tokens with minimal supervision. Experiments on open-weight and proprietary LLMs show that SciTrek poses significant challenges as the context length increases, with supervised fine-tuning and reinforcement learning offering only limited gains. Our analysis reveals systematic shortcomings of frontier LLMs' ability to effectively perform numerical operations and accurately locate information in long contexts.

翻译：我们提出了SciTrek，一种新颖的问答基准，旨在利用科学文献评估大语言模型（LLMs）的长上下文推理能力。当前的长上下文基准通常侧重于简单的信息检索任务，或采用人工构造的上下文。SciTrek通过创建需要跨多篇全文科学文献进行信息聚合与综合的基准问题，解决了这些局限性。问题及其标准答案通过将其表述为对基于文献元数据（即标题、作者和参考文献）构建的数据库的SQL查询而自动生成。这些SQL查询提供了明确、可验证的推理过程，使得能够对模型答案进行细粒度错误分析，并且数据构建可扩展到高达100万个标记的上下文，仅需极少的监督。在开源权重和专有LLMs上的实验表明，随着上下文长度的增加，SciTrek带来了显著的挑战，监督微调和强化学习仅能带来有限的性能提升。我们的分析揭示了前沿LLMs在有效执行数值运算和准确定位长上下文信息方面存在系统性不足。