The information stored in large language models (LLMs) falls out of date quickly, and retraining from scratch is often not an option. This has recently given rise to a range of techniques for injecting new facts through updating model weights. Current evaluation paradigms are extremely limited, mainly validating the recall of edited facts, but changing one fact should cause rippling changes to the model's related beliefs. If we edit the UK Prime Minister to now be Rishi Sunak, then we should get a different answer to Who is married to the British Prime Minister? In this work, we present a benchmark, MQuAKE (Multi-hop Question Answering for Knowledge Editing), comprising multi-hop questions that assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. While we find that current knowledge-editing approaches can recall edited facts accurately, they fail catastrophically on the constructed multi-hop questions. We thus propose a simple memory-based approach, MeLLo, which stores all edited facts externally while prompting the language model iteratively to generate answers that are consistent with the edited facts. While MQuAKE remains challenging, we show that MeLLo scales well with LLMs (up to 175B) and outperforms previous model editors by a large margin.

翻译：摘要：大型语言模型（LLM）中存储的信息会迅速过时，而从头重新训练往往不可行。这催生了一系列通过更新模型权重来注入新事实的技术。当前的评估范式极为有限，主要验证对已编辑事实的召回能力，但修改一个事实应引发模型相关信念的连锁变化。若将英国首相编辑为里希·苏纳克，那么对“谁与英国首相结婚”这一问题的答案也应随之改变。本文提出基准MQuAKE（面向知识编辑的多跳问答），包含多跳问题，用于评估编辑后的模型能否正确回答那些答案应随已编辑事实的必然结果而改变的问题。我们发现，现有知识编辑方法虽能准确召回已编辑事实，但在构造的多跳问题上表现灾难性失败。为此，我们提出一种简单的基于记忆的方法MeLLo，将所有已编辑事实存储于外部，同时迭代提示语言模型生成与已编辑事实一致的答案。尽管MQuAKE仍具挑战性，但实验表明MeLLo在大型语言模型（参数规模达175B）上具有良好的可扩展性，且性能大幅优于先前的模型编辑方法。