Large language models (LLMs) acquire knowledge during pre-training, but over time, this knowledge may become incorrect or outdated, necessitating updates after training. Knowledge editing techniques address this issue without the need for costly re-training. However, most existing methods are designed for single edits, and as the number of edits increases, they often cause a decline in the model's overall performance, posing significant challenges for sequential editing. To overcome this, we propose Orthogonal Subspace Editing, O-Edit. This algorithm orthogonalizes the direction of each knowledge update, minimizing interference between successive updates and reducing the impact of new updates on unrelated knowledge. Our approach does not require replaying previously edited data and processes each edit knowledge on time. It can perform thousands of edits on mainstream LLMs, achieving an average performance improvement that is 4.2 times better than existing methods while effectively preserving the model's performance on downstream tasks, all with minimal additional parameter overhead.

翻译：大型语言模型（LLMs）在预训练过程中习得知识，但随着时间的推移，这些知识可能变得不正确或过时，需要在训练后进行更新。知识编辑技术旨在解决这一问题，而无需进行成本高昂的重新训练。然而，现有方法大多针对单次编辑设计，随着编辑次数的增加，它们往往导致模型整体性能下降，这给序列化编辑带来了重大挑战。为克服此问题，我们提出了正交子空间编辑方法——O-Edit。该算法将每次知识更新的方向正交化，以最小化连续更新之间的干扰，并减少新更新对无关知识的影响。我们的方法无需重放先前编辑过的数据，并能对每次编辑知识进行实时处理。该方法可在主流LLMs上执行数千次编辑，其平均性能提升比现有方法高出4.2倍，同时能有效保持模型在下游任务上的性能，且仅需极少的额外参数开销。