Recent text-to-image (T2I) generation models have demonstrated impressive capabilities in creating images from text descriptions. However, these T2I generation models often fall short of generating images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM's in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multiple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build a joint multi-skill T2I model that can generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Moreover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that fine-tuning with images from a weaker T2I model can help improve the generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models.

翻译：近期文本到图像生成模型在根据文本描述创建图像方面展现出令人瞩目的能力。然而，这些文本到图像生成模型在生成精确匹配文本输入细节的图像时仍显不足，例如错误的空间关系或缺失对象。本文提出SELMA：技能特定专家学习与自动生成数据融合（Skill-Specific Expert Learning and Merging with Auto-Generated Data），这是一种通过利用自动生成的多技能图像-文本数据集进行微调，并结合技能特定专家学习与融合，以提升文本到图像模型忠实度的新范式。首先，SELMA利用大型语言模型的上下文学习能力生成多个涵盖不同技能的文本提示数据集，随后基于这些提示使用文本到图像模型生成图像。接着，SELMA通过学习多个单技能LoRA（低秩适配）专家并进行专家融合，使文本到图像模型适应新技能。独立的专家微调使不同模型专精于不同技能，而专家融合有助于构建联合的多技能文本到图像模型，该模型能够在多样化的文本提示下生成忠实图像，同时缓解不同数据集间的知识冲突。我们通过实验证明，SELMA在多个基准测试（TIFA提升2.1%，DSG提升6.9%）、人类偏好指标（PickScore、ImageReward和HPS）以及人工评估中，显著提升了最先进文本到图像扩散模型的语义对齐度和文本忠实度。此外，使用SELMA自动收集的图像-文本对进行微调，其性能与使用真实数据微调相当。最后，我们表明使用较弱文本到图像模型生成的图像进行微调，有助于提升较强文本到图像模型的生成质量，这为文本到图像模型中的弱到强泛化提供了有前景的启示。