Due to their capacity to generate novel and high-quality samples, diffusion models have attracted significant research interest in recent years. Notably, the typical training objective of diffusion models, i.e., denoising score matching, has a closed-form optimal solution that can only generate training data replicating samples. This indicates that a memorization behavior is theoretically expected, which contradicts the common generalization ability of state-of-the-art diffusion models, and thus calls for a deeper understanding. Looking into this, we first observe that memorization behaviors tend to occur on smaller-sized datasets, which motivates our definition of effective model memorization (EMM), a metric measuring the maximum size of training data at which a learned diffusion model approximates its theoretical optimum. Then, we quantify the impact of the influential factors on these memorization behaviors in terms of EMM, focusing primarily on data distribution, model configuration, and training procedure. Besides comprehensive empirical results identifying the influential factors, we surprisingly find that conditioning training data on uninformative random labels can significantly trigger the memorization in diffusion models. Our study holds practical significance for diffusion model users and offers clues to theoretical research in deep generative models. Code is available at https://github.com/sail-sg/DiffMemorize.

翻译：由于其生成新颖且高质量样本的能力，扩散模型近年来引起了广泛的研究兴趣。值得注意的是，扩散模型的典型训练目标，即去噪分数匹配，具有一个封闭形式的最优解，该解只能生成复制训练数据的样本。这表明记忆化行为在理论上是预期的，这与现有最先进扩散模型的泛化能力相矛盾，因此需要更深入的理解。针对这一点，我们首先观察到记忆化行为往往发生在较小规模的数据集上，这促使我们定义了有效模型记忆化（EMM）这一指标，用于衡量学习到的扩散模型逼近其理论最优时训练数据的最大规模。随后，我们从EMM角度量化了影响因素对记忆化行为的作用，主要关注数据分布、模型配置和训练过程。除了识别影响因素的全面实证结果外，我们意外发现，在训练数据上附加无信息随机标签会显著触发扩散模型中的记忆化。我们的研究对扩散模型用户具有实际意义，并为深度生成模型的理论研究提供了线索。代码可在 https://github.com/sail-sg/DiffMemorize 获取。