Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address this, we propose a framework, Language Models Augmented with Long-Term Memory (LongMem), which enables LLMs to memorize long history. We design a novel decoupled network architecture with the original backbone LLM frozen as a memory encoder and an adaptive residual side-network as a memory retriever and reader. Such a decoupled memory design can easily cache and update long-term past contexts for memory retrieval without suffering from memory staleness. Enhanced with memory-augmented adaptation training, LongMem can thus memorize long past context and use long-term memory for language modeling. The proposed memory retrieval module can handle unlimited-length context in its memory bank to benefit various downstream tasks. Typically, LongMem can enlarge the long-form memory to 65k tokens and thus cache many-shot extra demonstration examples as long-form memory for in-context learning. Experiments show that our method outperforms strong long-context models on ChapterBreak, a challenging long-context modeling benchmark, and achieves remarkable improvements on memory-augmented in-context learning over LLMs. The results demonstrate that the proposed method is effective in helping language models to memorize and utilize long-form contents. Our code is open-sourced at https://aka.ms/LongMem.

翻译：现有的大语言模型（LLMs）受限于输入长度限制，只能处理固定大小的输入，无法充分利用过去输入中丰富的长程上下文信息。为解决该问题，我们提出一种名为"长时记忆增强语言模型"（LongMem）的框架，使LLMs能够记忆长程历史信息。我们设计了一种新颖的解耦网络架构：将原始骨干LLM冻结作为记忆编码器，并构建自适应残差侧网络作为记忆检索器与阅读器。这种解耦式记忆设计可轻松缓存和更新长程历史上下文进行记忆检索，且不会出现记忆陈旧问题。通过记忆增强适配训练，LongMem能够记忆长程历史上下文并将其用于语言建模。所提出的记忆检索模块可在其记忆库中处理无限长度的上下文，从而赋能各类下游任务。典型情况下，LongMem可将长程记忆扩展至65k词元，从而缓存大量少样本演示示例作为长程记忆以支持上下文学习。实验表明，我们的方法在长程上下文建模挑战基准ChapterBreak上优于强基线长上下文模型，并在LLMs的记忆增强上下文学习任务中取得显著提升。研究结果证明，该方法能有效帮助语言模型记忆和利用长程内容。我们的代码已在 https://aka.ms/LongMem 开源。