We introduce Compartmentalized Diffusion Models (CDM), a method to train different diffusion models (or prompts) on distinct data sources and arbitrarily compose them at inference time. The individual models can be trained in isolation, at different times, and on different distributions and domains and can be later composed to achieve performance comparable to a paragon model trained on all data simultaneously. Furthermore, each model only contains information about the subset of the data it was exposed to during training, enabling several forms of training data protection. In particular, CDMs enable perfect selective forgetting and continual learning for large-scale diffusion models, allow serving customized models based on the user's access rights. Empirically the quality (FID) of the class-conditional CDMs (8-splits) is within 10% (on fine-grained vision datasets) of a monolithic model (no splits), and allows (8x) faster forgetting compared monolithic model with a maximum FID increase of 1%. When applied to text-to-image generation, CDMs improve alignment (TIFA) by 14.33% over a monolithic model trained on MSCOCO. CDMs also allow determining the importance of a subset of the data (attribution) in generating particular samples, and reduce memorization.

翻译：我们提出分区式扩散模型（CDM），该方法能在不同数据源上训练不同的扩散模型（或提示词），并在推理时任意组合这些模型。各模型可独立训练，在不同时间、不同分布和领域上完成训练，随后组合后能达到与同时在所有数据上训练的典范模型相当的性能。此外，每个模型仅包含其训练时接触过的数据子集信息，从而实现了多种形式的训练数据保护。具体而言，CDM能在大规模扩散模型中实现完美的选择性遗忘和持续学习，并允许根据用户的访问权限提供定制化模型。实验表明，类别条件CDM（8分区）的质量（FID）相比单一模型（无分区）的差距在10%以内（在细粒度视觉数据集上），且遗忘速度（8倍）快于单一模型，同时FID最大增量为1%。当应用于文本到图像生成时，CDM在MSCOCO数据集上的对齐度（TIFA）比单一模型提升14.33%。CDM还能确定特定样本生成过程中数据子集的重要性（归因），并减少记忆化现象。