Large-scale text-to-image generation models have achieved remarkable progress in synthesizing high-quality, feature-rich images with high resolution guided by texts. However, these models often struggle with novel concepts, eg, new styles, object entities, etc. Although recent attempts have employed fine-tuning or prompt-tuning strategies to teach the pre-trained diffusion model novel concepts from a reference image set,they have the drawback of overfitting to the given reference images, particularly in one-shot applications, which is harmful to generate diverse and high-quality images while maintaining generation controllability. To tackle this challenge, we present a simple yet effective method called DreamArtist, which employs a positive-negative prompt-tuning learning strategy. Specifically, DreamArtist incorporates both positive and negative embeddings and jointly trains them. The positive embedding aggressively captures the salient characteristics of the reference image to drive diversified generation and the negative embedding rectifies inadequacies from the positive embedding. It learns not only what is correct, but also what can be avoided or improved. We have conducted extensive experiments and evaluated the proposed method from image similarity and diversity, generation controllability, and style cloning. And our DreamArtist has achieved a superior generation performance over existing methods. Besides, our additional evaluation on extended tasks, including concept compositions and prompt-guided image editing, demonstrates its effectiveness for more applications.

翻译：大规模文本到图像生成模型在根据文本指导合成高质量、高分辨率且富含特征的图像方面取得了显著进展。然而，这些模型在处理新概念（如新风格、新物体实体等）时仍面临挑战。尽管近期尝试通过微调或提示微调策略，让预训练扩散模型从参考图像集中学习新概念，但这些方法存在对给定参考图像过度拟合的问题——尤其在单样本应用中，这不利于在保持生成可控性的同时生成多样且高质量的图像。为应对这一挑战，我们提出一种简洁高效的方法DreamArtist，其采用正-负提示微调学习策略。具体而言，DreamArtist融合正嵌入与负嵌入进行联合训练：正嵌入积极捕捉参考图像的显著特征以驱动多样化生成，负嵌入则校正正嵌入的不足。该方法不仅学习"何为正确"，更习得"何为可避免或可改进"。我们开展了大量实验，从图像相似度与多样性、生成可控性及风格克隆等维度评估所提方法。DreamArtist在生成性能上全面超越现有方法。此外，针对概念组合与提示引导图像编辑等扩展任务的额外评估，进一步验证了该方法在更广泛应用中的有效性。