LLMs have been found to memorize training textual sequences and regurgitate verbatim said sequences during text generation time. This fact is known to be the cause of privacy and related (e.g., copyright) problems. Unlearning in LLMs then takes the form of devising new algorithms that will properly deal with these side-effects of memorized data, while not hurting the model's utility. We offer a fresh perspective towards this goal, namely, that each textual sequence to be forgotten should be treated differently when being unlearned based on its degree of memorization within the LLM. We contribute a new metric for measuring unlearning quality, an adversarial attack showing that SOTA algorithms lacking this perspective fail for privacy, and two new unlearning methods based on Gradient Ascent and Task Arithmetic, respectively. A comprehensive performance evaluation across an extensive suite of NLP tasks then mapped the solution space, identifying the best solutions under different scales in model capacities and forget set sizes and quantified the gains of the new approaches.

翻译：大型语言模型被发现会记忆训练文本序列，并在生成文本时逐字复述这些序列。这一事实已知是导致隐私及相关（例如版权）问题的根源。对大型语言模型而言，遗忘则表现为设计新算法以妥善处理这些记忆数据的副作用，同时不损害模型的效用。我们为此目标提供了一种全新视角，即：在遗忘过程中，应根据每个待遗忘文本序列在语言模型中的记忆程度，对其区别对待。我们贡献了一种衡量遗忘质量的新指标，一次证明缺乏这一视角的现有最优算法在隐私保护方面失效的对抗攻击，以及两种分别基于梯度上升与任务算术的新遗忘方法。通过对涵盖广泛自然语言处理任务的全面性能评估，我们绘制了解决方案空间，识别出在不同模型容量与遗忘集规模尺度下的最优方案，并量化了新方法的性能增益。