The emergence of various adapters, including Low-Rank Adaptation (LoRA) applied from the field of natural language processing, has allowed diffusion models to personalize image generation at a low cost. However, due to the various challenges including limited datasets and shortage of regularization and computation resources, adapter training often results in unsatisfactory outcomes, leading to the corruption of the backbone model's prior knowledge. One of the well known phenomena is the loss of diversity in object generation, especially within the same class which leads to generating almost identical objects with minor variations. This poses challenges in generation capabilities. To solve this issue, we present Contrastive Adapter Training (CAT), a simple yet effective strategy to enhance adapter training through the application of CAT loss. Our approach facilitates the preservation of the base model's original knowledge when the model initiates adapters. Furthermore, we introduce the Knowledge Preservation Score (KPS) to evaluate CAT's ability to keep the former information. We qualitatively and quantitatively compare CAT's improvement. Finally, we mention the possibility of CAT in the aspects of multi-concept adapter and optimization.

翻译：各类适配器（包括源自自然语言处理领域的低秩适配LoRA）的出现，使得扩散模型能够以较低成本实现个性化图像生成。然而，由于数据集有限、正则化与计算资源不足等挑战，适配器训练往往产生不理想的结果，导致骨干模型的先验知识受损。一个广为人知的现象是目标生成多样性的丧失——尤其在同一类别内，模型会生成近乎完全相同、仅存在细微差异的目标，这严重制约了生成能力。为解决该问题，我们提出对比适配器训练（CAT），这是一种通过应用CAT损失来增强适配器训练的简单高效策略。本方法有助于在模型初始化适配器时保留基础模型的原始知识。此外，我们引入知识保留分数（KPS）来评估CAT保留先前信息的能力。通过定性与定量比较，我们验证了CAT的改进效果。最后，探讨了CAT在多概念适配器与优化方面的应用潜力。