We address a challenging lifelong few-shot image generation task for the first time. In this situation, a generative model learns a sequence of tasks using only a few samples per task. Consequently, the learned model encounters both catastrophic forgetting and overfitting problems at a time. Existing studies on lifelong GANs have proposed modulation-based methods to prevent catastrophic forgetting. However, they require considerable additional parameters and cannot generate high-fidelity and diverse images from limited data. On the other hand, the existing few-shot GANs suffer from severe catastrophic forgetting when learning multiple tasks. To alleviate these issues, we propose a framework called Lifelong Few-Shot GAN (LFS-GAN) that can generate high-quality and diverse images in lifelong few-shot image generation task. Our proposed framework learns each task using an efficient task-specific modulator - Learnable Factorized Tensor (LeFT). LeFT is rank-constrained and has a rich representation ability due to its unique reconstruction technique. Furthermore, we propose a novel mode seeking loss to improve the diversity of our model in low-data circumstances. Extensive experiments demonstrate that the proposed LFS-GAN can generate high-fidelity and diverse images without any forgetting and mode collapse in various domains, achieving state-of-the-art in lifelong few-shot image generation task. Surprisingly, we find that our LFS-GAN even outperforms the existing few-shot GANs in the few-shot image generation task. The code is available at Github.

翻译：我们首次解决了一项具有挑战性的终身少样本图像生成任务。在此情境下，生成模型仅需利用每个任务中少量样本即可学习一系列任务。因此，该学习模型同时面临灾难性遗忘和过拟合问题。现有关于终身生成对抗网络的研究提出了基于调制的方法来防止灾难性遗忘，但这些方法需要大量额外参数，且无法从有限数据中生成高保真度和多样性的图像。另一方面，现有少样本生成对抗网络在学习多个任务时会出现严重的灾难性遗忘问题。为缓解这些问题，我们提出了一种名为终身少样本生成对抗网络（LFS-GAN）的框架，该框架可在终身少样本图像生成任务中生成高质量且多样化的图像。我们提出的框架利用了一种高效的任务特定调制器——可分解因子张量（LeFT）来学习每个任务。LeFT具有秩约束特性，并凭借其独特的重建技术展现了丰富的表示能力。此外，我们提出了一种新颖的模式搜索损失，以提高模型在低数据情况下的多样性。大量实验表明，所提出的LFS-GAN能够在不同领域生成高保真度和多样性的图像，且无遗忘和模式崩溃问题，在终身少样本图像生成任务中达到了先进水平。令人惊讶的是，我们发现LFS-GAN在少样本图像生成任务中的表现甚至优于现有的少样本生成对抗网络。代码已发布在Github上。