Diffusion models have achieved remarkable success in Text-to-Image generation tasks, leading to the development of many commercial models. However, recent studies have reported that diffusion models often generate replicated images in train data when triggered by specific prompts, potentially raising social issues ranging from copyright to privacy concerns. To sidestep the memorization, there have been recent studies for developing memorization mitigation methods for diffusion models. Nevertheless, the lack of benchmarks impedes the assessment of the true effectiveness of these methods. In this work, we present MemBench, the first benchmark for evaluating image memorization mitigation methods. Our benchmark includes a large number of memorized image trigger prompts in Stable Diffusion, the most popularly used model nowadays. Furthermore, in contrast to the prior work evaluating mitigation performance only on trigger prompts, we present metrics evaluating on both trigger prompts and general prompts, so that we can see whether mitigation methods address the memorization issue while maintaining performance for general prompts. This is an important development considering the practical applications which previous works have overlooked. Through evaluation on MemBench, we verify that the performance of existing image memorization mitigation methods is still insufficient for application to diffusion models.

翻译：扩散模型在文本到图像生成任务中取得了显著成功，促成了许多商业模型的开发。然而，近期研究表明，扩散模型在受到特定提示触发时，常会生成训练数据中的复制图像，这可能引发从版权到隐私等一系列社会问题。为规避记忆化问题，近期已有研究致力于开发扩散模型的记忆化缓解方法。然而，由于缺乏基准测试，这些方法的实际效果难以评估。本研究提出MemBench——首个用于评估图像记忆化缓解方法的基准测试集。该基准包含大量针对当前最常用模型Stable Diffusion的记忆化图像触发提示。此外，与先前仅针对触发提示评估缓解性能的研究不同，我们提出了同时评估触发提示和通用提示的指标，从而能够判断缓解方法在解决记忆化问题的同时，是否保持了对通用提示的生成性能。考虑到实际应用中先前研究忽视的这一维度，该进展具有重要意义。通过在MemBench上的评估，我们验证了现有图像记忆化缓解方法的性能仍不足以应用于扩散模型。