The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops proposed that such loops would lead to a phenomenon termed model collapse, under which performance progressively degrades with each model-data feedback iteration until fitted models become useless. However, those studies largely assumed that new data replace old data over time, where an arguably more realistic assumption is that data accumulate over time. In this paper, we ask: what effect does accumulating data have on model collapse? We empirically study this question by pretraining sequences of language models on text corpora. We confirm that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse, then demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse; these results hold across a range of model sizes, architectures, and hyperparameters. We obtain similar results for deep generative models on other types of real data: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work in which a sequence of linear models are fit to the previous models' outputs. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse no longer occurs.

翻译：生成式模型的普及，加之基于海量网络数据的预训练，引发了一个紧迫问题：当这些模型以其自身生成的输出进行训练时会发生什么？近期针对模型-数据反馈循环的研究表明，此类循环将导致一种名为“模型崩塌”的现象——随着每次模型-数据反馈迭代，性能逐步退化，直至拟合模型完全失效。然而，这些研究大多假设新数据会随时间替代旧数据，而更具现实意义的假设或许是数据会随时间累积。本文提出疑问：数据累积对模型崩塌有何影响？我们通过在文本语料库上预训练一系列语言模型进行实证研究。我们确认，若用每代模型的合成数据替换原始真实数据，确实倾向于导致模型崩塌；随后证明，若将连续数代的合成数据与原始真实数据共同累积，则可避免模型崩塌——这些结论在不同模型规模、架构和超参数下均成立。针对其他类型真实数据的深度生成模型（用于分子构象生成的扩散模型和用于图像生成的变分自编码器），我们获得了相似结果。为理解数据累积为何能避免模型崩塌，我们采用先前研究提出的解析可处理框架，该框架中线性模型序列需拟合前序模型的输出。先前研究借助该框架证明：若数据被替换，测试误差会随模型拟合迭代次数增加而上升；我们扩展该论证，证明若数据改为累积，测试误差存在独立于迭代次数的有限上界，即模型崩塌不再发生。