The rapid advancement of diffusion models (DMs) has not only transformed various real-world industries but has also introduced negative societal concerns, including the generation of harmful content, copyright disputes, and the rise of stereotypes and biases. To mitigate these issues, machine unlearning (MU) has emerged as a potential solution, demonstrating its ability to remove undesired generative capabilities of DMs in various applications. However, by examining existing MU evaluation methods, we uncover several key challenges that can result in incomplete, inaccurate, or biased evaluations for MU in DMs. To address them, we enhance the evaluation metrics for MU, including the introduction of an often-overlooked retainability measurement for DMs post-unlearning. Additionally, we introduce UnlearnCanvas, a comprehensive high-resolution stylized image dataset that facilitates us to evaluate the unlearning of artistic painting styles in conjunction with associated image objects. We show that this dataset plays a pivotal role in establishing a standardized and automated evaluation framework for MU techniques on DMs, featuring 7 quantitative metrics to address various aspects of unlearning effectiveness. Through extensive experiments, we benchmark 5 state-of-the-art MU methods, revealing novel insights into their pros and cons, and the underlying unlearning mechanisms. Furthermore, we demonstrate the potential of UnlearnCanvas to benchmark other generative modeling tasks, such as style transfer. The UnlearnCanvas dataset, benchmark, and the codes to reproduce all the results in this work can be found at https://github.com/OPTML-Group/UnlearnCanvas.

翻译：扩散模型（DMs）的飞速发展不仅变革了多个实际行业，也引发了有害内容生成、版权争议以及刻板印象与偏见加剧等社会负面问题。为缓解这些问题，机器遗忘（MU）作为一种潜在解决方案应运而生，其在移除DM不良生成能力方面展现了显著效果。然而，通过剖析现有MU评估方法，我们发现了若干关键挑战，这些挑战可能导致DM中MU评估的片面性、不准确性或偏差。为此，我们改进了MU评估指标，包括引入DM遗忘后常被忽视的保留能力度量。此外，我们提出了UnlearnCanvas——一个高分辨率风格化图像综合数据集，用于评估与关联图像对象相结合的艺术绘画风格遗忘效果。研究表明，该数据集在构建DM上MU技术的标准化、自动化评估框架中发挥关键作用，包含7项量化指标以全面衡量遗忘效果。通过大量实验，我们对5种前沿MU方法进行了基准测试，揭示了其优势、不足及底层遗忘机制。进一步地，我们展示了UnlearnCanvas在风格迁移等其他生成建模任务基准测试中的潜力。本工作中使用的UnlearnCanvas数据集、基准测试及复现所有结果的代码已开源至https://github.com/OPTML-Group/UnlearnCanvas。