Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of NLP tasks, but they remain fundamentally stateless, constrained by limited context windows that hinder long-horizon reasoning. Recent efforts to address this limitation often augment LLMs with an external memory bank, yet most existing pipelines are static and heuristic-driven, lacking any learned mechanism for deciding what to store, update, or retrieve. We present Memory-R1, a reinforcement learning (RL) framework that equips LLMs with the ability to actively manage and utilize external memory through two specialized agents: a Memory Manager that learns to perform structured memory operations, including adding, updating, deleting, or taking no operation on memory entries; and an Answer Agent that selects the most relevant entries and reasons over them to produce an answer. Both agents are fine-tuned with outcome-driven RL (PPO and GRPO), enabling adaptive memory management and utilization with minimal supervision. With as few as 152 question-answer pairs and a corresponding temporal memory bank for training, Memory-R1 outperforms the strongest existing baseline and demonstrates strong generalization across diverse question types and LLM backbones. Beyond presenting an effective approach, this work provides insights into how RL can unlock more agentic, memory-aware behavior in LLMs, pointing toward richer, more persistent reasoning systems.

翻译：大型语言模型（LLMs）已在广泛的自然语言处理任务中展现出卓越能力，但其本质上仍是无状态的，受限于有限的上下文窗口，阻碍了长程推理。近期为解决这一局限性，常通过外部记忆库增强LLMs，然而现有流程多为静态且依赖启发式规则，缺乏决定存储、更新或检索内容的学习机制。本文提出Memory-R1，一种强化学习（RL）框架，通过两个专用代理赋予LLMs主动管理与利用外部记忆的能力：记忆管理器学习执行结构化记忆操作，包括对记忆条目进行添加、更新、删除或无操作；答案代理则选择最相关的条目并基于其进行推理以生成答案。两个代理均通过结果驱动的强化学习（PPO与GRPO）进行微调，实现以最小监督完成自适应记忆管理与利用。仅需152个问答对及对应时序记忆库进行训练，Memory-R1即超越现有最强基线，并在多样化问题类型与LLM骨干模型中展现出强大泛化能力。除提出有效方法外，本研究揭示了强化学习如何解锁LLMs更具自主性、记忆感知的行为，为构建更丰富、更持久的推理系统指明方向。