Recent text-to-image diffusion models have shown surprising performance in generating high-quality images. However, concerns have arisen regarding the unauthorized data usage during the training or fine-tuning process. One example is when a model trainer collects a set of images created by a particular artist and attempts to train a model capable of generating similar images without obtaining permission and giving credit to the artist. To address this issue, we propose a method for detecting such unauthorized data usage by planting the injected memorization into the text-to-image diffusion models trained on the protected dataset. Specifically, we modify the protected images by adding unique contents on these images using stealthy image warping functions that are nearly imperceptible to humans but can be captured and memorized by diffusion models. By analyzing whether the model has memorized the injected content (i.e., whether the generated images are processed by the injected post-processing function), we can detect models that had illegally utilized the unauthorized data. Experiments on Stable Diffusion and VQ Diffusion with different model training or fine-tuning methods (i.e, LoRA, DreamBooth, and standard training) demonstrate the effectiveness of our proposed method in detecting unauthorized data usages. Code: https://github.com/ZhentingWang/DIAGNOSIS.

翻译：近年来，文本到图像扩散模型在生成高质量图像方面展现出令人惊讶的性能。然而，训练或微调过程中的未授权数据使用问题日益引发关注。例如，模型训练者收集某位艺术家创作的图像集，未经许可且未注明出处，便尝试训练能够生成类似图像的模型。为解决这一问题，我们提出了一种通过向基于受保护数据集训练的文本到图像扩散模型中植入可控记忆痕迹来检测未授权数据使用的方法。具体而言，我们利用对人类几乎不可见但可被扩散模型捕获并记忆的隐蔽图像扭曲函数，向受保护图像中添加独特内容。通过分析模型是否记忆了注入内容（即生成图像是否经过注入后处理函数的处理），即可检测非法使用未授权数据的模型。基于Stable Diffusion和VQ Diffusion，结合不同模型训练或微调方法（如LoRA、DreamBooth和标准训练）的实验，验证了所提方法在检测未授权数据使用方面的有效性。代码：https://github.com/ZhentingWang/DIAGNOSIS。