Large language models (LLMs) have shown great success in various Natural Language Processing (NLP) tasks, whist they still need updates after deployment to fix errors or keep pace with the changing knowledge in the world. Researchers formulate such problem as Model Editing and have developed various editors focusing on different axes of editing properties. However, current editors can hardly support all properties and rely on heavy computational resources. In this paper, we propose a plug-in Model Editing method based on neuron-indexed dynamic LoRA (MELO), which alters the behavior of language models by dynamically activating certain LoRA blocks according to the index built in an inner vector database. Our method satisfies various editing properties with high efficiency and can be easily integrated into multiple LLM backbones. Experimental results show that our proposed MELO achieves state-of-the-art editing performance on three sequential editing tasks (document classification, question answering and hallucination correction), while requires the least trainable parameters and computational cost.

翻译：大语言模型在各种自然语言处理任务中取得了巨大成功，但它们在部署后仍需进行更新，以修正错误或跟上世界知识的动态变化。研究者将此类问题定义为模型编辑，并开发了多种聚焦于不同编辑属性的编辑器。然而，现有编辑器难以同时支持所有属性，且依赖大量计算资源。本文提出了一种基于神经元索引的动态LoRA插件式模型编辑方法（MELO），该方法通过根据内部向量数据库中构建的索引动态激活特定LoRA模块来改变语言模型的行为。我们的方法在满足多种编辑属性的同时保持高效性，并可轻松集成到多种大语言模型主干中。实验结果表明，所提出的MELO在三个序列编辑任务（文档分类、问答与幻觉修正）上取得了最先进的编辑性能，同时所需的可训练参数和计算成本最少。