Subject-driven text-to-image generation aims to generate customized images of the given subject based on the text descriptions, which has drawn increasing attention recently. Existing methods mainly resort to finetuning a pretrained generative model, where the identity-relevant information and the identity-irrelevant information are entangled in the latent embedding space. However, the highly entangled latent embedding may lead to the failure of subject-driven text-to-image generation as follows: (i) the identity-irrelevant information hidden in the entangled embedding may dominate the generation process, resulting in the generated images heavily dependent on the irrelevant information while ignoring the given text descriptions; (ii) the identity-relevant information carried in the entangled embedding can not be appropriately preserved, resulting in identity change of the subject in the generated images. To tackle the problems, we propose DisenBooth, an identity-preserving disentangled tuning framework for subject-driven text-to-image generation in this paper. Specifically, DisenBooth finetunes the pretrained diffusion model in the denoising process. Different from previous works that utilize an entangled embedding to denoise each image, DisenBooth instead utilizes disentangled embeddings to respectively preserve the subject identity and capture the identity-irrelevant information. We further design the novel weak denoising and contrastive embedding auxiliary tuning objectives to achieve the disentanglement. Extensive experiments show that our proposed DisenBooth framework outperforms baseline models for subject-driven text-to-image generation with the identity-preserved embedding. Additionally, by combining the identity-preserved embedding and identity-irrelevant embedding, DisenBooth demonstrates more generation flexibility and controllability.

翻译：主题驱动文本到图像生成旨在基于文本描述生成给定主题的定制化图像，近年来受到广泛关注。现有方法主要通过对预训练生成模型进行微调，但身份相关信息与身份无关信息在潜在嵌入空间中相互纠缠。然而，高度纠缠的潜在嵌入可能导致主题驱动文本到图像生成失败，具体表现为：（i）纠缠嵌入中隐藏的身份无关信息可能主导生成过程，导致生成图像严重依赖无关信息而忽略给定文本描述；（ii）纠缠嵌入携带的身份相关信息无法得到适当保留，导致生成图像中主题身份发生变化。为解决这些问题，本文提出DisenBooth——一种用于主题驱动文本到图像生成的保身份解耦调优框架。具体而言，DisenBooth在去噪过程中对预训练扩散模型进行微调。与以往利用纠缠嵌入对每张图像进行去噪的工作不同，DisenBooth采用解耦嵌入分别保留主题身份并捕获身份无关信息。我们进一步设计了新颖的弱去噪和对比嵌入辅助调优目标来实现解耦。大量实验表明，所提出的DisenBooth框架在基于保身份嵌入的主题驱动文本到图像生成中优于基线模型。此外，通过结合保身份嵌入与身份无关嵌入，DisenBooth展现出更强的生成灵活性和可控性。