Large language models (LLMs) have recently transformed both the academic and industrial landscapes due to their remarkable capacity to understand, analyze, and generate texts based on their vast knowledge and reasoning ability. Nevertheless, one major drawback of LLMs is their substantial computational cost for pre-training due to their unprecedented amounts of parameters. The disadvantage is exacerbated when new knowledge frequently needs to be introduced into the pre-trained model. Therefore, it is imperative to develop effective and efficient techniques to update pre-trained LLMs. Traditional methods encode new knowledge in pre-trained LLMs through direct fine-tuning. However, naively re-training LLMs can be computationally intensive and risks degenerating valuable pre-trained knowledge irrelevant to the update in the model. Recently, Knowledge-based Model Editing (KME) has attracted increasing attention, which aims to precisely modify the LLMs to incorporate specific knowledge, without negatively influencing other irrelevant knowledge. In this survey, we aim to provide a comprehensive and in-depth overview of recent advances in the field of KME. We first introduce a general formulation of KME to encompass different KME strategies. Afterward, we provide an innovative taxonomy of KME techniques based on how the new knowledge is introduced into pre-trained LLMs, and investigate existing KME strategies while analyzing key insights, advantages, and limitations of methods from each category. Moreover, representative metrics, datasets, and applications of KME are introduced accordingly. Finally, we provide an in-depth analysis regarding the practicality and remaining challenges of KME and suggest promising research directions for further advancement in this field.

翻译：大语言模型（LLMs）凭借其基于海量知识与推理能力的理解、分析和生成文本的卓越能力，近期同时改变了学术与工业领域。然而，LLMs的一个主要缺陷在于其前所未有的参数量导致预训练的高昂计算成本。当需要频繁向预训练模型注入新知识时，这一缺陷更加凸显。因此，开发高效且有效的技术来更新预训练LLMs势在必行。传统方法通过直接微调将新知识编码至预训练LLMs，但朴素地重新训练LLMs不仅计算密集，还可能危及模型中与更新无关的宝贵预训练知识。近年来，基于知识的模型编辑（KME）日益受到关注，其目标是在不影响其他无关知识的前提下，精确修改LLMs以融入特定知识。本文旨在系统全面地综述KME领域的最新进展。我们首先给出统一形式的KME定义以涵盖不同策略，继而提出基于新知识注入预训练LLMs方式的创新性KME技术分类体系，深入探究现有KME策略，分析各类方法的核心理念、优势与局限性。之后，系统介绍KME的代表性评估指标、数据集及应用场景。最后，深入剖析KME的实用性与现存挑战，并指出推动该领域进一步发展的潜在研究方向。