Text-to-image generative models, especially those based on latent diffusion models (LDMs), have demonstrated outstanding ability in generating high-quality and high-resolution images from textual prompts. With this advancement, various fine-tuning methods have been developed to personalize text-to-image models for specific applications such as artistic style adaptation and human face transfer. However, such advancements have raised copyright concerns, especially when the data are used for personalization without authorization. For example, a malicious user can employ fine-tuning techniques to replicate the style of an artist without consent. In light of this concern, we propose FT-Shield, a watermarking solution tailored for the fine-tuning of text-to-image diffusion models. FT-Shield addresses copyright protection challenges by designing new watermark generation and detection strategies. In particular, it introduces an innovative algorithm for watermark generation. It ensures the seamless transfer of watermarks from training images to generated outputs, facilitating the identification of copyrighted material use. To tackle the variability in fine-tuning methods and their impact on watermark detection, FT-Shield integrates a Mixture of Experts (MoE) approach for watermark detection. Comprehensive experiments validate the effectiveness of our proposed FT-Shield.

翻译：文本到图像生成模型，尤其是基于潜在扩散模型（LDMs）的模型，在根据文本提示生成高质量、高分辨率图像方面展现出卓越能力。随着这一进展，各种微调方法被开发出来，用于个性化定制文本到图像模型以满足特定应用需求，例如艺术风格迁移和人脸转换。然而，此类进展引发了版权担忧，尤其是在数据未经授权被用于个性化定制时。例如，恶意用户可能利用微调技术，在未经艺术家同意的情况下复制其风格。为此，我们提出FT-Shield——一种专为文本到图像扩散模型微调场景设计的水印解决方案。FT-Shield通过设计新颖的水印生成与检测策略，应对版权保护挑战。具体而言，它引入了一种创新的水印生成算法，确保水印从训练图像无缝迁移至生成输出，从而便于识别受版权保护材料的使用。为应对微调方法多样性及其对水印检测的影响，FT-Shield集成了混合专家（MoE）方法用于水印检测。综合实验验证了我们提出的FT-Shield的有效性。