Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of NLP tasks, but they remain fundamentally stateless, constrained by limited context windows that hinder long-horizon reasoning. Recent efforts to address this limitation often augment LLMs with an external memory bank, yet most existing pipelines are static and heuristic-driven, lacking a learned mechanism for deciding what to store, update, or retrieve. We present Memory-R1, a reinforcement learning (RL) framework that equips LLMs with the ability to actively manage and utilize external memory through two specialized agents: a Memory Manager that learns structured operations, including ADD, UPDATE, DELETE, and NOOP; and an Answer Agent that pre-selects and reasons over relevant entries. Both agents are fine-tuned with outcome-driven RL (PPO and GRPO), enabling adaptive memory management with minimal supervision. With only 152 training QA pairs, Memory-R1 outperforms strong baselines and generalizes across diverse question types, three benchmarks (LoCoMo, MSC, LongMemEval), and multiple model scales (3B-14B).

翻译：大型语言模型（LLMs）已在广泛的自然语言处理任务中展现出卓越能力，但其本质上仍是无状态的，受限于有限的上下文窗口，阻碍了长程推理。近期为解决此限制的研究通常为LLMs配备外部记忆库，然而现有流程大多为静态且启发式驱动，缺乏决定存储、更新或检索内容的学习机制。本文提出Memory-R1——一种强化学习（RL）框架，通过两个专用代理使LLMs具备主动管理和利用外部记忆的能力：学习结构化操作（包括ADD、UPDATE、DELETE和NOOP）的记忆管理器，以及预筛选相关条目并进行推理的应答代理。两个代理均通过结果驱动的强化学习（PPO和GRPO）进行微调，实现以最小监督完成自适应记忆管理。仅使用152个训练问答对，Memory-R1即在多样化问题类型、三个基准测试（LoCoMo、MSC、LongMemEval）及多种模型规模（3B-14B）上超越现有强基线并展现出良好泛化能力。