Recent advancements in personalizing text-to-image (T2I) diffusion models have shown the capability to generate images based on personalized visual concepts using a limited number of user-provided examples. However, these models often struggle with maintaining high visual fidelity, particularly in manipulating scenes as defined by textual inputs. Addressing this, we introduce ComFusion, a novel approach that leverages pretrained models generating composition of a few user-provided subject images and predefined-text scenes, effectively fusing visual-subject instances with textual-specific scenes, resulting in the generation of high-fidelity instances within diverse scenes. ComFusion integrates a class-scene prior preservation regularization, which leverages composites the subject class and scene-specific knowledge from pretrained models to enhance generation fidelity. Additionally, ComFusion uses coarse generated images, ensuring they align effectively with both the instance image and scene texts. Consequently, ComFusion maintains a delicate balance between capturing the essence of the subject and maintaining scene fidelity.Extensive evaluations of ComFusion against various baselines in T2I personalization have demonstrated its qualitative and quantitative superiority.

翻译：近期文本到图像（T2I）扩散模型个性化技术的进展，已展现出利用有限用户提供样本生成个性化视觉概念图像的能力。然而，这些模型在保持高视觉保真度方面仍面临挑战，尤其是在根据文本输入操控场景时。针对这一问题，我们提出ComFusion——一种创新方法，利用预训练模型生成由少量用户提供的主体图像与预定义文本场景复合而成的图像，有效融合视觉主体实例与文本特定场景，从而在多样化场景中生成高保真实例。ComFusion整合了类-场景先验保持正则化技术，通过复合主体类别与预训练模型中的场景特定知识来提升生成保真度。此外，ComFusion采用粗生成图像，确保其与实例图像及场景文本有效对齐。该方法在捕捉主体精髓与保持场景保真度之间取得了精妙平衡。通过将ComFusion与多种T2I个性化基线方法进行广泛评估，证实了其在定性与定量指标上的优越性。