Deep generative models have made tremendous progress in modeling complex data, often exhibiting generation quality that surpasses a typical human's ability to discern the authenticity of samples. Undeniably, a key driver of this success is enabled by the massive amounts of web-scale data consumed by these models. Due to these models' striking performance and ease of availability, the web will inevitably be increasingly populated with synthetic content. Such a fact directly implies that future iterations of generative models must contend with the reality that their training is curated from both clean data and artificially generated data from past models. In this paper, we develop a framework to rigorously study the impact of training generative models on mixed datasets (of real and synthetic data) on their stability. We first prove the stability of iterative training under the condition that the initial generative models approximate the data distribution well enough and the proportion of clean training data (w.r.t. synthetic data) is large enough. We empirically validate our theory on both synthetic and natural images by iteratively training normalizing flows and state-of-the-art diffusion models on CIFAR10 and FFHQ.

翻译：深度生成模型在建模复杂数据方面取得了巨大进展，通常能生成超越普通人辨别样本真实性的高质量内容。不可否认，这一成功的关键驱动力来自于这些模型所消耗的海量网络规模数据。由于这些模型卓越的性能和易获取性，网络将不可避免地充斥着越来越多的人工合成内容。这一事实直接意味着未来迭代的生成模型必须面对一个现实：它们的训练数据既包含真实数据，也包含来自先前模型的人工生成数据。本文提出一个框架，严谨地研究训练生成模型时使用混合数据集（包含真实与合成数据）对其稳定性的影响。我们首先证明：在初始生成模型对数据分布有足够好的近似，且真实训练数据相对于合成数据的比例足够大时，迭代训练具有稳定性。我们通过使用归一化流和当前最先进的扩散模型，在CIFAR10和FFHQ数据集上进行迭代训练，从理论和实验两方面验证了这一结论。