Recent works demonstrate a remarkable ability to customize text-to-image diffusion models while only providing a few example images. What happens if you try to customize such models using multiple, fine-grained concepts in a sequential (i.e., continual) manner? In our work, we show that recent state-of-the-art customization of text-to-image models suffer from catastrophic forgetting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high quality images of past, similar concepts degrade. To circumvent this forgetting, we propose a new method, C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the popular Stable Diffusion model. Furthermore, we use customization prompts which do not include the word of the customized object (i.e., "person" for a human face dataset) and are initialized as completely random embeddings. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. We show that C-LoRA not only outperforms several baselines for our proposed setting of text-to-image continual customization, which we refer to as Continual Diffusion, but that we achieve a new state-of-the-art in the well-established rehearsal-free continual learning setting for image classification. The high achieving performance of C-LoRA in two separate domains positions it as a compelling solution for a wide range of applications, and we believe it has significant potential for practical impact. Project page: https://jamessealesmith.github.io/continual-diffusion/

翻译：近期研究展示了仅通过提供少量示例图像即可定制文本到图像扩散模型的显著能力。然而，若试图以顺序（即持续）方式对多个细粒度概念进行此类模型定制，将会出现何种情况？在本研究中，我们发现当前最先进的文本到图像模型定制方法在面对新概念顺序出现时，会遭受灾难性遗忘。具体而言，当添加新概念时，模型生成先前相似概念的高质量图像的能力会退化。为规避这一遗忘问题，我们提出了一种新方法——C-LoRA，该方法在流行模型Stable Diffusion的交叉注意力层中引入了持续自正则化的低秩适配。此外，我们使用了不包含定制对象词汇（例如，针对人脸数据集使用"person"）的定制提示，并将其初始化为完全随机的嵌入。重要的是，我们的方法仅引入边际性的额外参数成本，且无需存储用户数据进行重放。实验表明，C-LoRA不仅在我们提出的文本到图像持续定制设定（称为持续扩散）中优于多种基线方法，还在无需重放的持续学习图像分类设定中取得了新的最优结果。C-LoRA在两个不同领域中的优异表现使其成为广泛应用的引人注目的解决方案，我们相信它具有显著的实践应用潜力。项目页面：https://jamessealesmith.github.io/continual-diffusion/