Subject-driven text-to-image generation aims to generate customized images of the given subject based on the text descriptions, which has drawn increasing attention. Existing methods mainly resort to finetuning a pretrained generative model, where the identity-relevant information (e.g., the boy) and the identity-irrelevant information (e.g., the background or the pose of the boy) are entangled in the latent embedding space. However, the highly entangled latent embedding may lead to the failure of subject-driven text-to-image generation as follows: (i) the identity-irrelevant information hidden in the entangled embedding may dominate the generation process, resulting in the generated images heavily dependent on the irrelevant information while ignoring the given text descriptions; (ii) the identity-relevant information carried in the entangled embedding can not be appropriately preserved, resulting in identity change of the subject in the generated images. To tackle the problems, we propose DisenBooth, an identity-preserving disentangled tuning framework for subject-driven text-to-image generation. Specifically, DisenBooth finetunes the pretrained diffusion model in the denoising process. Different from previous works that utilize an entangled embedding to denoise each image, DisenBooth instead utilizes disentangled embeddings to respectively preserve the subject identity and capture the identity-irrelevant information. We further design the novel weak denoising and contrastive embedding auxiliary tuning objectives to achieve the disentanglement. Extensive experiments show that our proposed DisenBooth framework outperforms baseline models for subject-driven text-to-image generation with the identity-preserved embedding. Additionally, by combining the identity-preserved embedding and identity-irrelevant embedding, DisenBooth demonstrates more generation flexibility and controllability

翻译：主题驱动文本到图像生成旨在基于文本描述生成给定主题的定制化图像，近年来受到广泛关注。现有方法主要依赖微调预训练生成模型，其中身份相关信息（如男孩）与身份无关信息（如背景或姿态）在潜在嵌入空间中相互纠缠。然而，高度纠缠的潜在嵌入可能引发以下主题驱动文本到图像生成失败：（i）纠缠嵌入中隐藏的身份无关信息可能主导生成过程，导致生成图像过度依赖无关信息而忽略给定文本描述；（ii）纠缠嵌入携带的身份相关信息无法被恰当保留，导致生成图像中主题身份发生改变。为解决上述问题，我们提出DisenBooth——一种面向主题驱动文本到图像生成的保身份解耦调优框架。具体而言，DisenBooth在去噪过程中微调预训练扩散模型，区别于以往使用纠缠嵌入对每张图像去噪的方法，它采用解耦嵌入分别保留主题身份和捕获身份无关信息。我们进一步设计新颖的弱去噪与对比嵌入辅助调优目标以实现解耦。大量实验表明，所提出的DisenBooth框架通过保身份嵌入在主题驱动文本到图像生成中优于基线模型。此外，结合保身份嵌入与身份无关嵌入，DisenBooth展现出更强的生成灵活性与可控性。