Recent text-to-image generation models have demonstrated incredible success in generating images that faithfully follow input prompts. However, the requirement of using words to describe a desired concept provides limited control over the appearance of the generated concepts. In this work, we address this shortcoming by proposing an approach to enable personalization capabilities in existing text-to-image diffusion models. We propose a novel architecture (BootPIG) that allows a user to provide reference images of an object in order to guide the appearance of a concept in the generated images. The proposed BootPIG architecture makes minimal modifications to a pretrained text-to-image diffusion model and utilizes a separate UNet model to steer the generations toward the desired appearance. We introduce a training procedure that allows us to bootstrap personalization capabilities in the BootPIG architecture using data generated from pretrained text-to-image models, LLM chat agents, and image segmentation models. In contrast to existing methods that require several days of pretraining, the BootPIG architecture can be trained in approximately 1 hour. Experiments on the DreamBooth dataset demonstrate that BootPIG outperforms existing zero-shot methods while being comparable with test-time finetuning approaches. Through a user study, we validate the preference for BootPIG generations over existing methods both in maintaining fidelity to the reference object's appearance and aligning with textual prompts.

翻译：近期文本到图像生成模型已展现出根据输入提示生成遵循语义图像的惊人能力。然而，使用文字描述所需概念的方式对生成图像外观的控制能力有限。本研究针对这一缺陷，提出一种为现有文本到图像扩散模型赋予个性化能力的方法。我们设计了一种新型架构（BootPIG），允许用户提供参考图像以引导生成图像中特定概念的外观。所提BootPIG架构对预训练文本到图像扩散模型仅做最小修改，并利用独立UNet模型将生成过程导向目标外观。我们提出了一种训练流程，通过使用预训练文本到图像模型、大语言模型聊天代理及图像分割模型生成的数据，自举式地赋予BootPIG架构个性化能力。与需要数天预训练的现有方法相比，BootPIG架构仅需约1小时即可完成训练。在DreamBooth数据集上的实验表明，BootPIG性能优于现有零样本方法，并与测试时微调方法表现相当。通过用户研究，我们验证了相较于现有方法，BootPIG在保持参考对象外观忠实度与文本提示对齐方面均获得更优偏好。