We propose RecaLLM, a set of reasoning language models post-trained to make effective use of long-context information. In-context retrieval, which identifies relevant evidence from context, and reasoning are deeply intertwined: retrieval supports reasoning, while reasoning often determines what must be retrieved. However, their interaction remains largely underexplored. In preliminary experiments on several open-source LLMs, we observe that in-context retrieval performance substantially degrades even after a short reasoning span, revealing a key bottleneck for test-time scaling that we refer to as lost-in-thought: reasoning steps that improve performance also make subsequent in-context retrieval more challenging. To address this limitation, RecaLLM interleaves reasoning with explicit in-context retrieval, alternating between reasoning and retrieving context information needed to solve intermediate subproblems. We introduce a negligible-overhead constrained decoding mechanism that enables verbatim copying of evidence spans, improving the grounding of subsequent generation. Trained on diverse lexical and semantic retrieval tasks, RecaLLM achieves strong performance on two long-context benchmarks, RULER and HELMET, significantly outperforming baselines. Notably, we observe consistent gains at context windows of up to 128K tokens using training samples of at most 10K tokens, far shorter than those used by existing long-context approaches, highlighting a promising path toward improving long-context performance without expensive long-context training data.

翻译：我们提出RecaLLM，这是一组经过后训练的推理语言模型，旨在有效利用长上下文信息。上下文检索（即从上下文中识别相关证据）与推理深度交织：检索支撑推理，而推理往往决定需要检索什么。然而，两者间的相互作用仍未得到充分探索。在多个开源大语言模型的初步实验中，我们观察到即便经过简短的推理过程，上下文检索性能也会大幅下降，这揭示了测试时扩展中的关键瓶颈——我们称之为“思想迷失”：提升性能的推理步骤同时使后续上下文检索更加困难。为解决这一局限，RecaLLM将推理与显式上下文检索交替进行，即在推理与检索解决中间子问题所需上下文信息之间切换。我们引入了一种开销极小的约束解码机制，可实现证据片段的逐字复制，从而增强后续生成过程的接地性。RecaLLM通过多种词汇与语义检索任务训练，在RULER和HELMET两个长上下文基准测试中取得了强劲性能，显著优于基线模型。值得注意的是，我们观察到，在使用最多仅10K标记的训练样本时，RecaLLM在高达128K标记的上下文窗口中均能保持一致的性能提升——这远短于现有长上下文方法使用的样本长度，凸显出一条无需昂贵长上下文训练数据即可提升长上下文性能的可行路径。