We address a challenging lifelong few-shot image generation task for the first time. In this situation, a generative model learns a sequence of tasks using only a few samples per task. Consequently, the learned model encounters both catastrophic forgetting and overfitting problems at a time. Existing studies on lifelong GANs have proposed modulation-based methods to prevent catastrophic forgetting. However, they require considerable additional parameters and cannot generate high-fidelity and diverse images from limited data. On the other hand, the existing few-shot GANs suffer from severe catastrophic forgetting when learning multiple tasks. To alleviate these issues, we propose a framework called Lifelong Few-Shot GAN (LFS-GAN) that can generate high-quality and diverse images in lifelong few-shot image generation task. Our proposed framework learns each task using an efficient task-specific modulator - Learnable Factorized Tensor (LeFT). LeFT is rank-constrained and has a rich representation ability due to its unique reconstruction technique. Furthermore, we propose a novel mode seeking loss to improve the diversity of our model in low-data circumstances. Extensive experiments demonstrate that the proposed LFS-GAN can generate high-fidelity and diverse images without any forgetting and mode collapse in various domains, achieving state-of-the-art in lifelong few-shot image generation task. Surprisingly, we find that our LFS-GAN even outperforms the existing few-shot GANs in the few-shot image generation task. The code is available at Github.

翻译：摘要： 我们首次应对了一项具有挑战性的终身小样本图像生成任务。在此情境下，生成模型仅利用每个任务中的少量样本学习一系列任务。因此，所学习到的模型同时面临灾难性遗忘和过拟合问题。现有关于终身GAN的研究提出了基于调制的方法来防止灾难性遗忘。然而，这些方法需要大量额外参数，且无法从有限数据中生成高保真度和多样化的图像。另一方面，现有小样本GAN在学习多个任务时会出现严重的灾难性遗忘。为缓解这些问题，我们提出了一种名为终身小样本GAN（LFS-GAN）的框架，该框架能够在终身小样本图像生成任务中生成高质量且多样化的图像。我们提出的框架使用一种高效的任务特定调制器——可分解因子张量（LeFT）来学习每个任务。LeFT具有秩约束，并因其独特的重构技术而拥有丰富的表征能力。此外，我们提出了一种新颖的模式搜索损失函数，以在低数据情况下提升模型的多样性。大量实验表明，所提出的LFS-GAN能够在各种领域中生成高保真度和多样化的图像，且无任何遗忘和模式崩溃现象，在终身小样本图像生成任务中达到了最先进水平。令人惊讶的是，我们发现我们的LFS-GAN在小样本图像生成任务中甚至优于现有小样本GAN。代码已在Github上开源。