Pre-trained large text-to-image (T2I) models with an appropriate text prompt has attracted growing interests in customized images generation field. However, catastrophic forgetting issue make it hard to continually synthesize new user-provided styles while retaining the satisfying results amongst learned styles. In this paper, we propose MuseumMaker, a method that enables the synthesis of images by following a set of customized styles in a never-end manner, and gradually accumulate these creative artistic works as a Museum. When facing with a new customization style, we develop a style distillation loss module to extract and learn the styles of the training data for new image generation. It can minimize the learning biases caused by content of new training images, and address the catastrophic overfitting issue induced by few-shot images. To deal with catastrophic forgetting amongst past learned styles, we devise a dual regularization for shared-LoRA module to optimize the direction of model update, which could regularize the diffusion model from both weight and feature aspects, respectively. Meanwhile, to further preserve historical knowledge from past styles and address the limited representability of LoRA, we consider a task-wise token learning module where a unique token embedding is learned to denote a new style. As any new user-provided style come, our MuseumMaker can capture the nuances of the new styles while maintaining the details of learned styles. Experimental results on diverse style datasets validate the effectiveness of our proposed MuseumMaker method, showcasing its robustness and versatility across various scenarios.

翻译：预训练的大规模文本到图像（T2I）模型结合恰当的文本提示，在定制图像生成领域引起了越来越多的关注。然而，灾难性遗忘问题使得在持续合成用户提供的新风格时，难以保留已学习风格中令人满意的结果。本文提出博物馆制造者（MuseumMaker）方法，该方法能以一种无止境的方式遵循一组定制风格合成图像，并逐步将这些创意艺术作品积累成一座“博物馆”。面对新的定制风格时，我们设计了一个风格蒸馏损失模块，用于提取和学习训练数据中的风格以生成新图像。它能最小化新训练图像内容引起的学习偏差，并解决由少样本图像引发的灾难性过拟合问题。为处理已学习风格间的灾难性遗忘，我们为共享LoRA模块设计了一种双重正则化机制，从权重和特征两个层面优化模型更新方向，从而分别对扩散模型进行正则化。同时，为进一步保留历史风格知识并解决LoRA表示能力有限的问题，我们考虑了一种任务级令牌学习模块，通过学习独特的令牌嵌入来表征新风格。随着任何用户提供的新风格出现，MuseumMaker能在保持已学习风格细节的同时捕捉新风格的细微差别。在多样化风格数据集上的实验结果验证了我们提出的MuseumMaker方法的有效性，展示了其在各种场景下的鲁棒性和多功能性。