Diffusion models are powerful generative models that achieve state-of-the-art performance in image synthesis. However, training them demands substantial amounts of data and computational resources. Continual learning would allow for incrementally learning new tasks and accumulating knowledge, thus enabling the reuse of trained models for further learning. One potentially suitable continual learning approach is generative replay, where a copy of a generative model trained on previous tasks produces synthetic data that are interleaved with data from the current task. However, standard generative replay applied to diffusion models results in a catastrophic loss in denoising capabilities. In this paper, we propose generative distillation, an approach that distils the entire reverse process of a diffusion model. We demonstrate that our approach substantially improves the continual learning performance of generative replay with only a modest increase in the computational costs.

翻译：扩散模型是强大的生成模型，在图像合成中达到了最先进的性能。然而，训练这些模型需要大量的数据和计算资源。持续学习能够逐步学习新任务并积累知识，从而使得经过训练的模型可以复用于进一步的学习。生成式重放是一种潜在适用的持续学习方法，该方法会复制一个在先前任务上训练好的生成模型，用其生成的合成数据与当前任务的数据交替使用。然而，将标准生成式重放应用于扩散模型会导致去噪能力的灾难性损失。本文提出生成式蒸馏方法，将扩散模型的整个反向过程进行蒸馏。实验表明，该方法在仅适度增加计算成本的情况下，显著提升了生成式重放中持续学习的性能。