The rapid advancement of diffusion models (DMs) has not only transformed various real-world industries but has also introduced negative societal concerns, including the generation of harmful content, copyright disputes, and the rise of stereotypes and biases. To mitigate these issues, machine unlearning (MU) has emerged as a potential solution, demonstrating its ability to remove undesired generative capabilities of DMs in various applications. However, by examining existing MU evaluation methods, we uncover several key challenges that can result in incomplete, inaccurate, or biased evaluations for MU in DMs. To address them, we enhance the evaluation metrics for MU, including the introduction of an often-overlooked retainability measurement for DMs post-unlearning. Additionally, we introduce UnlearnCanvas, a comprehensive high-resolution stylized image dataset that facilitates us to evaluate the unlearning of artistic painting styles in conjunction with associated image objects. We show that this dataset plays a pivotal role in establishing a standardized and automated evaluation framework for MU techniques on DMs, featuring 7 quantitative metrics to address various aspects of unlearning effectiveness. Through extensive experiments, we benchmark 5 state-of-the-art MU methods, revealing novel insights into their pros and cons, and the underlying unlearning mechanisms. Furthermore, we demonstrate the potential of UnlearnCanvas to benchmark other generative modeling tasks, such as style transfer. The UnlearnCanvas dataset, benchmark, and the codes to reproduce all the results in this work can be found at https://github.com/OPTML-Group/UnlearnCanvas.

翻译：扩散模型（DMs）的快速发展不仅改变了多个现实行业，也引发了有害内容生成、版权争议以及刻板印象与偏见加剧等负面社会问题。为缓解这些问题，机器遗忘（MU）作为一种潜在解决方案应运而生，并在多项应用中展现出移除扩散模型不良生成能力的作用。然而，通过审视现有MU评估方法，我们发现在扩散模型MU评估中存在若干关键挑战，可能导致评估结果不完整、不准确或存在偏差。为此，我们改进了MU的评估指标，包括引入扩散模型遗忘后常被忽略的保持能力度量方法。此外，我们提出了UnlearnCanvas——一个高分辨率风格化图像综合数据集，可用于评估与图像对象关联的艺术绘画风格的遗忘效果。研究表明，该数据集在建立标准化、自动化的扩散模型MU技术评估框架中起到关键作用，该框架包含7项定量指标，从多个维度衡量遗忘有效性。通过大量实验，我们对5种最先进的MU方法进行基准测试，揭示了其优缺点及底层遗忘机制的新见解。进一步地，我们展示了UnlearnCanvas在风格迁移等其他生成建模任务基准测试中的潜力。UnlearnCanvas数据集、基准测试以及本工作所有结果的可复现代码均可在https://github.com/OPTML-Group/UnlearnCanvas获取。