Personalized text-to-image generation using diffusion models has recently emerged and garnered significant interest. This task learns a novel concept (e.g., a unique toy), illustrated in a handful of images, into a generative model that captures fine visual details and generates photorealistic images based on textual embeddings. In this paper, we present ViCo, a novel lightweight plug-and-play method that seamlessly integrates visual condition into personalized text-to-image generation. ViCo stands out for its unique feature of not requiring any fine-tuning of the original diffusion model parameters, thereby facilitating more flexible and scalable model deployment. This key advantage distinguishes ViCo from most existing models that necessitate partial or full diffusion fine-tuning. ViCo incorporates an image attention module that conditions the diffusion process on patch-wise visual semantics, and an attention-based object mask that comes at no extra cost from the attention module. Despite only requiring light parameter training (~6% compared to the diffusion U-Net), ViCo delivers performance that is on par with, or even surpasses, all state-of-the-art models, both qualitatively and quantitatively. This underscores the efficacy of ViCo, making it a highly promising solution for personalized text-to-image generation without the need for diffusion model fine-tuning. Code: https://github.com/haoosz/ViCo

翻译：基于扩散模型的个性化文本到图像生成技术近期兴起并引起广泛关注。该任务旨在将少量图像中呈现的新颖概念（如独特玩具）习得至生成模型中，使其能够捕捉精细视觉细节，并基于文本嵌入生成逼真图像。本文提出ViCo——一种轻量级即插即用方法，可无缝地将视觉条件融入个性化文本到图像生成过程。ViCo的独特优势在于无需微调原始扩散模型参数，从而支持更灵活、可扩展的模型部署。这一关键特性使其区别于现有需要部分或完全微调扩散模型的主流方法。ViCo通过图像注意力模块实现基于图像块视觉语义的条件扩散过程，并利用该注意力模块以零额外代价生成基于注意力的对象掩码。尽管仅需训练少量参数（约为扩散U-Net参数的6%），ViCo在定性与定量评估中均能达到甚至超越所有最先进模型的性能。这充分验证了ViCo的有效性，使其成为无需扩散模型微调的个性化文本到图像生成领域极具前景的解决方案。代码：https://github.com/haoosz/ViCo