While diffusion models demonstrate a remarkable capability for generating high-quality images, their tendency to `replicate' training data raises privacy concerns. Although recent research suggests that this replication may stem from the insufficient generalization of training data captions and duplication of training images, effective mitigation strategies remain elusive. To address this gap, our paper first introduces a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. Subsequently, we leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models. Our empirical results demonstrate that our proposed methods can significantly reduce replication by 43.5% compared to the original diffusion model while maintaining the diversity and quality of generations. Code is available at https://github.com/HowardLi0816/dual-fusion-diffusion.

翻译：摘要：尽管扩散模型在生成高质量图像方面展现出卓越能力，但其倾向于“复制”训练数据的特性引发了隐私担忧。近期研究表明，这种复制可能源于训练数据字幕泛化不足与训练图像重复，然而有效的缓解策略仍难以实现。为填补这一空白，本文首先引入衡量字幕泛化能力的泛化性评分，并利用大语言模型对训练字幕进行泛化处理。随后，我们基于泛化后的字幕提出了一种新颖的双融合增强方法，以缓解扩散模型的复制问题。实验结果表明，与原始扩散模型相比，所提方法在保持生成结果多样性及质量的同时，复制现象显著降低43.5%。代码开源地址：https://github.com/HowardLi0816/dual-fusion-diffusion。