Catastrophic Forgetting (CF) means models forgetting previously acquired knowledge when learning new data. It compromises the effectiveness of large language models (LLMs) during fine-tuning, yet the underlying causes have not been thoroughly investigated. This paper takes the first step to reveal the direct link between the flatness of the model loss landscape and the extent of CF in the field of LLMs. Based on this, we introduce the sharpness-aware minimization to mitigate CF by flattening the loss landscape. Experiments on three widely-used fine-tuning datasets, spanning different model scales, demonstrate the effectiveness of our method in alleviating CF. Analyses show that we nicely complement the existing anti-forgetting strategies, further enhancing the resistance of LLMs to CF.

翻译：灾难性遗忘（Catastrophic Forgetting, CF）指模型在学习新数据时遗忘先前获得的知识的现象。该问题削弱了大型语言模型（LLMs）在微调过程中的有效性，然而其根本原因尚未得到深入探究。本文首次揭示了LLM领域中模型损失景观平坦度与CF程度之间的直接联系。基于此，我们引入锐度感知最小化方法，通过平坦化损失景观来缓解CF。在三个广泛使用的微调数据集上（涵盖不同模型规模）的实验结果表明，我们的方法能有效缓解CF。分析显示，该方法与现有抗遗忘策略具有良好的互补性，进一步增强了LLMs对CF的抵抗能力。