As the number of large language models (LLMs) released to the public grows, there is a pressing need to understand the safety implications associated with these models learning from third-party custom finetuning data. We explore the behavior of LLMs finetuned on noisy custom data containing unsafe content, represented by datasets that contain biases, toxicity, and harmfulness, finding that while aligned LLMs can readily learn this unsafe content, they also tend to forget it more significantly than other examples when subsequently finetuned on safer content. Drawing inspiration from the discrepancies in forgetting, we introduce the "ForgetFilter" algorithm, which filters unsafe data based on how strong the model's forgetting signal is for that data. We demonstrate that the ForgetFilter algorithm ensures safety in customized finetuning without compromising downstream task performance, unlike sequential safety finetuning. ForgetFilter outperforms alternative strategies like replay and moral self-correction in curbing LLMs' ability to assimilate unsafe content during custom finetuning, e.g. 75% lower than not applying any safety measures and 62% lower than using self-correction in toxicity score.

翻译：随着向公众发布的大语言模型（LLM）数量不断增加，理解这些模型从第三方自定义微调数据中学习所伴随的安全影响变得至关重要。我们探索了LLM在包含不安全内容的噪声自定义数据（表现为包含偏见、毒性和有害性的数据集）上进行微调时的行为，发现虽然经过对齐的LLM能够轻易学习这些不安全内容，但在后续对更安全内容进行微调时，它们对这些内容的遗忘程度往往显著高于其他示例。受遗忘差异的启发，我们提出了“遗忘过滤器”（ForgetFilter）算法，该算法根据模型对特定数据的遗忘信号强度来过滤不安全数据。我们证明，与顺序安全微调相比，ForgetFilter算法在保证自定义微调安全性的同时，不会损害下游任务性能。在抑制LLM在自定义微调期间吸收不安全内容的能力方面，ForgetFilter优于重放和道德自我修正等替代策略，例如毒性评分比不采取任何安全措施低75%，比使用自我修正低62%。