Model editing aims to correct outdated or erroneous knowledge in large language models (LLMs) without the need for costly retraining. Lifelong model editing is the most challenging task that caters to the continuous editing requirements of LLMs. Prior works primarily focus on single or batch editing; nevertheless, these methods fall short in lifelong editing scenarios due to catastrophic knowledge forgetting and the degradation of model performance. Although retrieval-based methods alleviate these issues, they are impeded by slow and cumbersome processes of integrating the retrieved knowledge into the model. In this work, we introduce RECIPE, a RetriEval-augmented ContInuous Prompt lEarning method, to boost editing efficacy and inference efficiency in lifelong learning. RECIPE first converts knowledge statements into short and informative continuous prompts, prefixed to the LLM's input query embedding, to efficiently refine the response grounded on the knowledge. It further integrates the Knowledge Sentinel (KS) that acts as an intermediary to calculate a dynamic threshold, determining whether the retrieval repository contains relevant knowledge. Our retriever and prompt encoder are jointly trained to achieve editing properties, i.e., reliability, generality, and locality. In our experiments, RECIPE is assessed extensively across multiple LLMs and editing datasets, where it achieves superior editing performance. RECIPE also demonstrates its capability to maintain the overall performance of LLMs alongside showcasing fast editing and inference speed.

翻译：模型编辑旨在无需昂贵重新训练的情况下，修正大语言模型（LLMs）中过时或错误的知识。终身模型编辑是最具挑战性的任务，旨在满足LLMs的持续编辑需求。先前的研究主要集中于单次或批量编辑；然而，这些方法在终身编辑场景中因灾难性知识遗忘和模型性能退化而存在不足。尽管基于检索的方法缓解了这些问题，但其将检索到的知识整合到模型中的过程缓慢且繁琐，阻碍了应用。在本工作中，我们提出了RECIPE（一种检索增强的连续提示学习方法），以提升终身学习中的编辑效能与推理效率。RECIPE首先将知识陈述转化为简短且信息丰富的连续提示，并将其作为前缀添加到LLM输入查询的嵌入表示中，从而基于知识高效地优化响应。该方法进一步集成了知识哨兵（KS），其作为中介计算动态阈值，以判断检索知识库中是否包含相关知识。我们的检索器与提示编码器经过联合训练，以实现编辑特性，即可靠性、泛化性与局部性。在实验中，RECIPE在多个LLM和编辑数据集上进行了广泛评估，并取得了卓越的编辑性能。RECIPE还证明了其在保持LLM整体性能的同时，具备快速编辑与推理速度的能力。