Large language models show impressive abilities in memorizing world knowledge, which leads to concerns regarding memorization of private information, toxic or sensitive knowledge, and copyrighted content. We introduce the problem of Large Scale Knowledge Washing, focusing on unlearning an extensive amount of factual knowledge. Previous unlearning methods usually define the reverse loss and update the model via backpropagation, which may affect the model's fluency and reasoning ability or even destroy the model due to extensive training with the reverse loss. Existing works introduce additional data from downstream tasks to prevent the model from losing capabilities, which requires downstream task awareness. Controlling the tradeoff of unlearning and maintaining existing capabilities is also challenging. To this end, we propose LAW (Large Scale Washing) to update the MLP layers in decoder-only large language models to perform knowledge washing, as inspired by model editing methods and based on the hypothesis that knowledge and reasoning are disentanglable. We derive a new objective with the knowledge to be unlearned to update the weights of certain MLP layers. Experimental results demonstrate the effectiveness of LAW in forgetting target knowledge while maintaining reasoning ability. The code will be open-sourced at https://github.com/wangyu-ustc/LargeScaleWashing.

翻译：大型语言模型在记忆世界知识方面展现出令人印象深刻的能力，这引发了对其记忆私人信息、有害或敏感知识以及受版权保护内容的担忧。我们提出了大规模知识清洗问题，重点关注遗忘大量事实性知识。以往的遗忘方法通常定义逆向损失并通过反向传播更新模型，这可能因大量逆向损失训练而影响模型的流畅性和推理能力，甚至破坏模型。现有工作引入下游任务的额外数据以防止模型能力丧失，但这需要下游任务感知。控制遗忘与保持现有能力之间的权衡也具有挑战性。为此，受模型编辑方法启发并基于知识与推理可分离的假设，我们提出LAW（大规模清洗）方法，通过更新仅解码器大型语言模型中的MLP层来实现知识清洗。我们利用待遗忘知识推导出新的目标函数来更新特定MLP层的权重。实验结果证明了LAW在遗忘目标知识的同时保持推理能力的有效性。代码将在https://github.com/wangyu-ustc/LargeScaleWashing开源。