Diffusion-based models have demonstrated impressive capabilities for text-to-image generation and are expected for personalized applications of subject-driven generation, which require the generation of customized concepts with one or a few reference images. However, existing methods based on fine-tuning fail to balance the trade-off between subject learning and the maintenance of the generation capabilities of pretrained models. Moreover, other methods that utilize additional image encoders tend to lose important details of the subject due to encoding compression. To address these challenges, we propose DreamTurner, a novel method that injects reference information from coarse to fine to achieve subject-driven image generation more effectively. DreamTurner introduces a subject-encoder for coarse subject identity preservation, where the compressed general subject features are introduced through an attention layer before visual-text cross-attention. We then modify the self-attention layers within pretrained text-to-image models to self-subject-attention layers to refine the details of the target subject. The generated image queries detailed features from both the reference image and itself in self-subject-attention. It is worth emphasizing that self-subject-attention is an effective, elegant, and training-free method for maintaining the detailed features of customized subjects and can serve as a plug-and-play solution during inference. Finally, with additional subject-driven fine-tuning, DreamTurner achieves remarkable performance in subject-driven image generation, which can be controlled by a text or other conditions such as pose. For further details, please visit the project page at https://dreamtuner-diffusion.github.io/.

翻译：基于扩散的模型在文本到图像生成领域展现出卓越能力，并被期望应用于主体驱动生成的个性化场景——即通过一张或少数参考图像生成定制化概念。然而，现有基于微调的方法难以在主体学习与预训练模型生成能力维持之间取得平衡。此外，其他利用额外图像编码器的方法因编码压缩而容易丢失主体的重要细节。为解决这些挑战，我们提出DreamTuner，一种从粗到细注入参考信息以实现更有效主体驱动图像生成的新颖方法。DreamTuner引入主体编码器以保持粗粒度主体身份，通过视觉-文本交叉注意力前的注意力层引入压缩后的通用主体特征。随后，我们将预训练文本到图像模型中的自注意力层修改为自-主体注意力层，以细化目标主体的细节。在自-主体注意力中，生成图像从参考图像及其自身查询细节特征。值得强调的是，自-主体注意力是一种高效、优雅且无需训练的方法，用于保持定制化主体的细节特征，并可在推理阶段作为即插即用方案。最终，结合额外的主体驱动微调，DreamTuner在主体驱动图像生成中展现出卓越性能，且可通过文本或姿势等其他条件进行控制。更多详情请访问项目页面：https://dreamtuner-diffusion.github.io/。