Recent works demonstrate a remarkable ability to customize text-to-image diffusion models while only providing a few example images. What happens if you try to customize such models using multiple, fine-grained concepts in a sequential (i.e., continual) manner? In our work, we show that recent state-of-the-art customization of text-to-image models suffer from catastrophic forgetting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high quality images of past, similar concepts degrade. To circumvent this forgetting, we propose a new method, C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the popular Stable Diffusion model. Furthermore, we use customization prompts which do not include the word of the customized object (i.e., "person" for a human face dataset) and are initialized as completely random embeddings. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. We show that C-LoRA not only outperforms several baselines for our proposed setting of text-to-image continual customization, which we refer to as Continual Diffusion, but that we achieve a new state-of-the-art in the well-established rehearsal-free continual learning setting for image classification. The high achieving performance of C-LoRA in two separate domains positions it as a compelling solution for a wide range of applications, and we believe it has significant potential for practical impact.

翻译：近期研究展示了仅需少量示例图像即可定制文本到图像扩散模型的显著能力。若以顺序（即持续）方式使用多个细粒度概念进行定制，结果会如何？本研究表明，当新概念顺序加入时，当前最先进的文本到图像模型定制方法会出现灾难性遗忘。具体而言，添加新概念会降低生成先前相似概念高质量图像的能力。为解决这一遗忘问题，我们提出新方法C-LoRA，该方法在广泛使用的Stable Diffusion模型的交叉注意力层中采用持续自正则化低秩适应。此外，我们使用的定制提示词不包含定制对象的词汇（例如对人脸数据集使用"person"），并初始化为完全随机嵌入。重要的是，我们的方法仅引入边际额外参数成本，且无需存储用户数据进行回放。实验证明，C-LoRA不仅在我们提出的"持续扩散"文本到图像持续定制设置中优于多个基线方法，还在图像分类领域经典的无重放持续学习设定中达到新最优水平。C-LoRA在两个独立领域的卓越性能使其成为广泛应用的极具潜力解决方案，我们相信其具有重要的实际应用价值。