Language models deployed in real-world systems often require post-hoc updates to incorporate new or corrected knowledge. However, editing such models efficiently and reliably-without retraining or forgetting previous information-remains a major challenge. Existing methods for lifelong model editing either compromise generalization, interfere with past edits, or fail to scale to long editing sequences. We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, i.e., a dedicated parameter module, while preserving the core capabilities of the pre-trained model. By sparsifying input activations through sample-dependent masks, MEMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits. At inference, it identifies relevant edits by comparing the sparse activation patterns of new queries to those stored during editing. This enables generalization to rephrased queries by activating only the relevant knowledge while suppressing unnecessary memory activation for unrelated prompts. Experiments on question answering, hallucination correction, and out-of-distribution generalization benchmarks for LLaMA-3 and Mistral backbones demonstrate that MEMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.

翻译：现实系统中部署的语言模型通常需要进行事后更新以纳入新知识或修正知识。然而，如何高效可靠地编辑此类模型——无需重新训练且不遗忘先前信息——仍是重大挑战。现有终身模型编辑方法往往在泛化性、历史编辑干扰或长序列编辑可扩展性方面存在局限。本文提出MEMOIR，一种创新的可扩展框架，通过残差记忆（即专用参数模块）注入知识，同时保持预训练模型的核心能力。该方法通过样本依赖掩码对输入激活进行稀疏化处理，将每次编辑限制在记忆参数的特定子集内，从而最小化编辑间干扰。在推理阶段，通过对比新查询的稀疏激活模式与编辑阶段存储的模式来识别相关编辑。该机制仅激活相关知识即可实现改写查询的泛化，同时抑制不相关提示对记忆的不必要激活。基于LLaMA-3和Mistral架构在问答、幻觉修正和分布外泛化基准上的实验表明，MEMOIR在可靠性、泛化性和局部性指标上均达到最先进性能，可扩展至数千次连续编辑且遗忘效应最小。