Model editing aims to correct outdated or erroneous knowledge in large language models (LLMs) without the need for costly retraining. Lifelong model editing is the most challenging task that caters to the continuous editing requirements of LLMs. Prior works primarily focus on single or batch editing; nevertheless, these methods fall short in lifelong editing scenarios due to catastrophic knowledge forgetting and the degradation of model performance. Although retrieval-based methods alleviate these issues, they are impeded by slow and cumbersome processes of integrating the retrieved knowledge into the model. In this work, we introduce RECIPE, a RetriEval-augmented ContInuous Prompt lEarning method, to boost editing efficacy and inference efficiency in lifelong learning. RECIPE first converts knowledge statements into short and informative continuous prompts, prefixed to the LLM's input query embedding, to efficiently refine the response grounded on the knowledge. It further integrates the Knowledge Sentinel (KS) that acts as an intermediary to calculate a dynamic threshold, determining whether the retrieval repository contains relevant knowledge. Our retriever and prompt encoder are jointly trained to achieve editing properties, i.e., reliability, generality, and locality. In our experiments, RECIPE is assessed extensively across multiple LLMs and editing datasets, where it achieves superior editing performance. RECIPE also demonstrates its capability to maintain the overall performance of LLMs alongside showcasing fast editing and inference speed.

翻译：模型编辑旨在无需昂贵的重新训练即可纠正大语言模型（LLMs）中的过时或错误知识。终身模型编辑是满足LLMs持续编辑需求的最具挑战性任务。先前工作主要关注单次或批量编辑；然而，这些方法在终身编辑场景中因灾难性知识遗忘和模型性能退化而存在不足。尽管基于检索的方法缓解了这些问题，但其将检索知识整合到模型中的过程缓慢且笨拙。本文提出RECIPE（检索增强连续提示学习方法），旨在提升终身学习中的编辑效能与推理效率。RECIPE首先将知识陈述转化为简短且信息丰富的连续提示，附加到LLM的输入查询嵌入前，以高效地基于该知识优化响应。它进一步整合知识哨兵（KS）作为中间件，计算动态阈值以判断检索库是否包含相关知识。我们的检索器与提示编码器经过联合训练以实现编辑属性（即可靠性、通用性和局部性）。实验表明，RECIPE在多种LLM和编辑数据集上均取得优越的编辑性能，同时保持LLM整体性能，并展现出快速的编辑与推理速度。