Large language model unlearning aims to remove harmful information that LLMs have learnt to prevent their use for malicious purposes. LLMU and RMU have been proposed as two methods for LLM unlearning, achieving impressive results on unlearning benchmarks. We study in detail the efficacy of these methods by evaluating their impact on general model capabilities on the WMDP benchmark as well as a biology benchmark we create. Our experiments show that RMU generally leads to better preservation of model capabilities, for similar or better unlearning. We further test the robustness of these methods and find that doing 5-shot prompting or rephrasing the question in simple ways can lead to an over ten-fold increase in accuracy on unlearning benchmarks. Finally, we show that training on unrelated data can almost completely recover pre-unlearning performance, demonstrating that these methods fail at truly unlearning. The code is available at: https://github.com/JaiDoshi/Knowledge-Erasure.

翻译：大型语言模型遗忘旨在移除LLM已学习的有害信息，以防止其被用于恶意目的。LLMU和RMU作为两种LLM遗忘方法被提出，在遗忘基准测试中取得了显著成果。我们通过评估这些方法在WMDP基准测试以及我们创建的生物学基准测试中对模型通用能力的影响，详细研究了这些方法的有效性。实验表明，在实现相似或更优遗忘效果的同时，RMU通常能更好地保持模型能力。我们进一步测试了这些方法的鲁棒性，发现通过5样本提示或简单改写问题，可使遗忘基准测试的准确率提升超过十倍。最后，我们证明在无关数据上进行训练几乎可以完全恢复遗忘前的性能，这表明这些方法未能实现真正的遗忘。代码发布于：https://github.com/JaiDoshi/Knowledge-Erasure。