Recent work has demonstrated a remarkable ability to customize text-to-image diffusion models to multiple, fine-grained concepts in a sequential (i.e., continual) manner while only providing a few example images for each concept. This setting is known as continual diffusion. Here, we ask the question: Can we scale these methods to longer concept sequences without forgetting? Although prior work mitigates the forgetting of previously learned concepts, we show that its capacity to learn new tasks reaches saturation over longer sequences. We address this challenge by introducing a novel method, STack-And-Mask INcremental Adapters (STAMINA), which is composed of low-ranked attention-masked adapters and customized MLP tokens. STAMINA is designed to enhance the robust fine-tuning properties of LoRA for sequential concept learning via learnable hard-attention masks parameterized with low rank MLPs, enabling precise, scalable learning via sparse adaptation. Notably, all introduced trainable parameters can be folded back into the model after training, inducing no additional inference parameter costs. We show that STAMINA outperforms the prior SOTA for the setting of text-to-image continual customization on a 50-concept benchmark composed of landmarks and human faces, with no stored replay data. Additionally, we extended our method to the setting of continual learning for image classification, demonstrating that our gains also translate to state-of-the-art performance in this standard benchmark.

翻译：近期研究表明，文本到图像扩散模型能够以顺序（即持续）方式对多个细粒度概念进行定制化，且每个概念仅需少量示例图像。这种设置被称为持续扩散。本文提出以下问题：能否在不遗忘的前提下，将这些方法扩展到更长的概念序列？尽管先前方法缓解了已学概念的遗忘问题，但实验表明，其学习新任务的能力在长序列中会达到饱和。为攻克这一挑战，我们提出了一种新方法——堆叠掩码增量适配器（STAMINA），该方法由低秩注意力掩码适配器和定制化MLP标记组成。STAMINA通过可学习的硬注意力掩码（以低秩MLP参数化）增强LoRA的鲁棒微调特性，从而实现基于稀疏适配的精确可扩展学习。值得注意的是，所有可训练参数在训练后均可折叠回模型，不引入额外推理参数成本。实验证明，在由地标和人脸组成的50概念基准测试中，STAMINA在文本到图像持续定制任务上超越了先前最先进方法，且无需存储重放数据。此外，我们将该方法扩展到图像分类的持续学习场景，验证了其在标准基准测试中同样达到了最优性能。