Model editing aims to correct outdated or erroneous knowledge in large language models (LLMs) without the need for costly retraining. Lifelong model editing is the most challenging task that caters to the continuous editing requirements of LLMs. Prior works primarily focus on single or batch editing; nevertheless, these methods fall short in lifelong editing scenarios due to catastrophic knowledge forgetting and the degradation of model performance. Although retrieval-based methods alleviate these issues, they are impeded by slow and cumbersome processes of integrating the retrieved knowledge into the model. In this work, we introduce RECIPE, a RetriEval-augmented ContInuous Prompt lEarning method, to boost editing efficacy and inference efficiency in lifelong learning. RECIPE first converts knowledge statements into short and informative continuous prompts, prefixed to the LLM's input query embedding, to efficiently refine the response grounded on the knowledge. It further integrates the Knowledge Sentinel (KS) that acts as an intermediary to calculate a dynamic threshold, determining whether the retrieval repository contains relevant knowledge. Our retriever and prompt encoder are jointly trained to achieve editing properties, i.e., reliability, generality, and locality. In our experiments, RECIPE is assessed extensively across multiple LLMs and editing datasets, where it achieves superior editing performance. RECIPE also demonstrates its capability to maintain the overall performance of LLMs alongside showcasing fast editing and inference speed.

翻译：模型编辑旨在纠正大型语言模型（LLM）中的过时或错误知识，而无需昂贵的重新训练。终身模型编辑是最具挑战性的任务，需满足LLM持续编辑的需求。先前的工作主要关注单次或批量编辑；然而，这些方法在终身编辑场景中存在知识灾难性遗忘和模型性能退化的问题。尽管基于检索的方法缓解了这些问题，但将检索到的知识整合到模型中的过程缓慢且繁琐。本文提出RECIPE——一种检索增强的连续提示学习方法，以提升终身学习中的编辑效果和推理效率。RECIPE首先将知识陈述转换为简短且信息丰富的连续提示，并将其作为前缀附加到LLM输入查询嵌入中，从而高效地基于知识优化响应。该方法进一步集成了知识哨兵（KS），作为中介计算动态阈值，以判断检索库中是否包含相关知识。我们联合训练检索器和提示编码器，以实现编辑的可信性、泛化性和局部性属性。实验表明，RECIPE在多个LLM和编辑数据集上取得了优异的编辑性能，同时能够保持LLM的整体性能，并展现出快速的编辑和推理速度。