Although language models (LMs) demonstrate exceptional capabilities on various tasks, they are potentially vulnerable to extraction attacks, which represent a significant privacy risk. To mitigate the privacy concerns of LMs, machine unlearning has emerged as an important research area, which is utilized to induce the LM to selectively forget about some of its training data. While completely retraining the model will guarantee successful unlearning and privacy assurance, it is impractical for LMs, as it would be time-consuming and resource-intensive. Prior works efficiently unlearn the target token sequences, but upon subsequent iterations, the LM displays significant degradation in performance. In this work, we propose Privacy Protection via Optimal Parameters (POP), a novel unlearning method that effectively forgets the target token sequences from the pretrained LM by applying optimal gradient updates to the parameters. Inspired by the gradient derivation of complete retraining, we approximate the optimal training objective that successfully unlearns the target sequence while retaining the knowledge from the rest of the training data. Experimental results demonstrate that POP exhibits remarkable retention performance post-unlearning across 9 classification and 4 dialogue benchmarks, outperforming the state-of-the-art by a large margin. Furthermore, we introduce Remnant Memorization Accuracy that quantifies privacy risks based on token likelihood and validate its effectiveness through both qualitative and quantitative analyses.

翻译：尽管语言模型在各种任务上展现出卓越能力，但其可能面临提取攻击的威胁，这构成了显著的隐私风险。为缓解语言模型的隐私隐患，机器遗忘已成为重要研究方向，其目标在于引导语言模型选择性地遗忘部分训练数据。虽然完全重新训练模型能够确保成功遗忘并保障隐私，但对语言模型而言这种方法并不现实，因其耗时且资源密集。现有研究虽能高效遗忘目标词元序列，但在后续迭代中语言模型会出现显著的性能退化。本研究提出基于最优参数的隐私保护方法（POP），这是一种通过向参数施加最优梯度更新，从而有效使预训练语言模型遗忘目标词元序列的新型遗忘方法。受完整重训练梯度推导的启发，我们逼近了能够成功遗忘目标序列同时保留其余训练数据知识的最优训练目标。实验结果表明，POP在9个分类基准和4个对话基准测试中展现出遗忘后卓越的知识保持性能，大幅超越现有最优方法。此外，我们提出了基于词元似然度量化隐私风险的残余记忆精度指标，并通过定性与定量分析验证了其有效性。