Large language models (LLMs) can make predictions using parametric knowledge--knowledge encoded in the model weights--or contextual knowledge--knowledge presented in the context. In many scenarios, a desirable behavior is that LLMs give precedence to contextual knowledge when it conflicts with the parametric knowledge, and fall back to using their parametric knowledge when the context is irrelevant. This enables updating and correcting the model's knowledge by in-context editing instead of retraining. Previous works have shown that LLMs are inclined to ignore contextual knowledge and fail to reliably fall back to parametric knowledge when presented with irrelevant context. In this work, we discover that, with proper prompting methods, instruction-finetuned LLMs can be highly controllable by contextual knowledge and robust to irrelevant context. Utilizing this feature, we propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. To better evaluate the robustness of model editors, we collect a new dataset, that contains irrelevant questions that are more challenging than the ones in existing datasets. Empirical results show that our method outperforms current state-of-the-art methods by a large margin. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs (and vice versa). The source code can be found at https://github.com/thunlp/EREN.

翻译：大型语言模型（LLMs）可通过参数化知识（编码在模型权重中的知识）或上下文知识（在上下文中呈现的知识）进行预测。在许多场景下，理想的行为是：当上下文知识与参数化知识冲突时，LLMs优先采用上下文知识；而当上下文无关时，则回退到参数化知识。这使得通过上下文编辑而非重新训练来更新和纠正模型知识成为可能。先前研究表明，LLMs倾向于忽略上下文知识，且在呈现无关上下文时无法可靠地回退到参数化知识。本工作中我们发现，通过恰当的提示方法，指令微调的LLMs能够高度受控于上下文知识，并对无关上下文具有鲁棒性。利用这一特性，我们提出EREN（通过阅读笔记编辑模型）方法，以提升LLMs编辑的可扩展性和鲁棒性。为更好评估模型编辑器的鲁棒性，我们构建了一个新数据集，其中包含比现有数据集更具挑战性的无关问题。实验结果表明，我们的方法大幅超越了当前最先进方法。与现有技术不同，该方法可整合多次编辑的知识，并能正确回应语法相似但语义无关的输入（反之亦然）。源代码见https://github.com/thunlp/EREN。