Diffusion models are powerful generative models that achieve state-of-the-art performance in tasks such as image synthesis. However, training them demands substantial amounts of data and computational resources. Continual learning would allow for incrementally learning new tasks and accumulating knowledge, thus reusing already trained models would be possible. One potentially suitable approach is generative replay, where a copy of a generative model trained on previous tasks produces synthetic data that are interleaved with data from the current task. However, standard generative replay applied to diffusion models results in a catastrophic loss in denoising capabilities. In this paper, we propose generative distillation, an approach that distils the entire reverse process of a diffusion model. We demonstrate that our approach significantly improves the continual learning performance of generative replay with only a moderate increase in the computational costs.

翻译：扩散模型是强大的生成模型，在图像合成等任务中达到了最先进的性能。然而，训练它们需要大量的数据和计算资源。持续学习能够逐步学习新任务并积累知识，从而使得复用已训练的模型成为可能。一种潜在适用的方法是生成式重放，其中先前任务训练的生成模型的副本会产生合成数据，这些数据与当前任务的数据穿插使用。然而，将标准生成式重放应用于扩散模型会导致去噪能力的灾难性损失。在本文中，我们提出生成式蒸馏，一种蒸馏扩散模型整个逆过程的方法。我们证明，我们的方法显著提升了生成式重放的持续学习性能，而计算成本仅适度增加。