In this paper, we propose active recap learning (ARL), a framework for enhancing large language model (LLM) in understanding long contexts. ARL enables models to revisit and summarize earlier content through targeted sequence construction during contined pretraining and retrospective summarization at inference. First, we identify key tokens in prepared long context based on loss gaps between long and short forward contexts and find most revant preceding paragraphs, then summarize them using an LLM. Second, ARL equips models with the ability to autonomously generate and utilize these retrospective summaries during inference, thereby establishing a recursive memory mechanism across paragraphs. Experimental results show substantial gains, with ARL achieving a 26.8% improvement on RULER and a 9.44% improvement on LongBench. Overall, ARL offers a simple yet effective continued pretraining-based approach to strengthen long-context understanding, advancing scalable memory augmentation in LLM

翻译：本文提出主动回顾学习框架，旨在增强大语言模型的长上下文理解能力。该框架通过在持续预训练阶段构建定向序列，以及在推理阶段执行回溯式摘要，使模型能够重访并总结先前的文本内容。首先，我们基于长/短前向上下文损失差异识别长文本中的关键标记，定位最相关的历史段落，并利用大语言模型生成摘要。其次，该框架使模型在推理时能自主生成并运用此类回溯摘要，从而建立跨段落的递归记忆机制。实验结果表明该方法带来显著性能提升：在RULER基准上实现26.8%的改进，在LongBench基准上获得9.44%的提升。总体而言，该框架提供了一种简洁有效的持续预训练方案，不仅强化了长上下文理解能力，更推动了大语言模型可扩展记忆增强机制的发展。