Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. However, a primary limitation lies in the significant computational demands during training, arising from their extensive parameterization. This challenge is further intensified by the dynamic nature of the world, necessitating frequent updates to LLMs to correct outdated information or integrate new knowledge, thereby ensuring their continued relevance. Note that many applications demand continual model adjustments post-training to address deficiencies or undesirable behaviors. There is an increasing interest in efficient, lightweight methods for on-the-fly model modifications. To this end, recent years have seen a burgeoning in the techniques of knowledge editing for LLMs, which aim to efficiently modify LLMs' behaviors within specific domains while preserving overall performance across various inputs. In this paper, we first define the knowledge editing problem and then provide a comprehensive review of cutting-edge approaches. Drawing inspiration from educational and cognitive research theories, we propose a unified categorization criterion that classifies knowledge editing methods into three groups: resorting to external knowledge, merging knowledge into the model, and editing intrinsic knowledge. Furthermore, we introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches. Additionally, we provide an in-depth analysis of knowledge location, which can give a deeper understanding of the knowledge structures inherent within LLMs. Finally, we discuss several potential applications of knowledge editing, outlining its broad and impactful implications.

翻译：大语言模型（LLMs）在理解和生成接近人类交流的文本方面展现出非凡能力。然而，其主要局限性在于训练过程中因参数规模庞大而产生的显著计算需求。现实世界的动态变化进一步加剧了这一挑战，导致需要频繁更新LLMs以修正过时信息或整合新知识，从而确保其持续相关性。值得注意的是，许多应用场景要求在训练后持续调整模型以解决缺陷或不良行为。学界对高效轻量化的在线模型修正方法日益关注。为此，近年来针对LLMs的知识编辑技术蓬勃发展，该技术旨在保持模型整体性能的同时，高效修正特定领域的行为。本文首先定义知识编辑问题，随后系统综述前沿方法。受教育和认知研究理论启发，我们提出统一分类标准，将知识编辑方法划分为三类：借助外部知识、将知识融入模型、编辑内在知识。此外，我们引入新基准KnowEdit，对代表性知识编辑方法进行全面的实证评估。同时深入分析知识定位机制，以加深对LLMs内在知识结构的理解。最后讨论知识编辑的潜在应用前景，阐述其广泛而深远的影响。