In scenarios where language models must incorporate new information efficiently without extensive retraining, traditional fine-tuning methods are prone to overfitting, degraded generalization, and unnatural language generation. To address these limitations, we introduce Consistent In-Context Editing (ICE), a novel approach leveraging the model's in-context learning capability to optimize toward a contextual distribution rather than a one-hot target. ICE introduces a simple yet effective optimization framework for the model to internalize new knowledge by aligning its output distributions with and without additional context. This method enhances the robustness and effectiveness of gradient-based tuning methods, preventing overfitting and preserving the model's integrity. We analyze ICE across four critical aspects of knowledge editing: accuracy, locality, generalization, and linguistic quality, demonstrating its advantages. Experimental results confirm the effectiveness of ICE and demonstrate its potential for continual editing, ensuring that the integrity of the model is preserved while updating information.

翻译：在语言模型需要高效融入新信息而无需大量重新训练的场景中，传统的微调方法容易出现过拟合、泛化能力下降以及不自然的语言生成问题。为应对这些局限，我们提出了一致性上下文编辑（ICE），这是一种新颖的方法，利用模型的上下文学习能力，使其优化朝向上下文分布而非单一目标。ICE引入了一个简单而有效的优化框架，使模型通过对齐其带有额外上下文与不带有额外上下文的输出分布，从而内化新知识。该方法增强了基于梯度的调优方法的鲁棒性和有效性，防止过拟合并保持模型的完整性。我们从知识编辑的四个关键方面——准确性、局部性、泛化性和语言质量——对ICE进行了分析，展示了其优势。实验结果证实了ICE的有效性，并证明了其在持续编辑中的潜力，确保在更新信息的同时保持模型的完整性。