Text-to-image (T2I) diffusion models, when fine-tuned on a few personal images, are able to generate visuals with a high degree of consistency. However, they still lack in synthesizing images of different scenarios or styles that are possible in the original pretrained models. To address this, we propose to fine-tune the T2I model by maximizing consistency to reference images, while penalizing the deviation from the pretrained model. We devise a novel training objective for T2I diffusion models that minimally fine-tunes the pretrained model to achieve consistency. Our method, dubbed \emph{Direct Consistency Optimization}, is as simple as regular diffusion loss, while significantly enhancing the compositionality of personalized T2I models. Also, our approach induces a new sampling method that controls the tradeoff between image fidelity and prompt fidelity. Lastly, we emphasize the necessity of using a comprehensive caption for reference images to further enhance the image-text alignment. We show the efficacy of the proposed method on the T2I personalization for subject, style, or both. In particular, our method results in a superior Pareto frontier to the baselines. Generated examples and codes are in our project page( https://dco-t2i.github.io/).

翻译：文本到图像（T2I）扩散模型在针对少量个人图像进行微调后，能够生成具有高度一致性的视觉内容。然而，它们仍难以合成原始预训练模型中可能存在的不同场景或风格的图像。为解决此问题，我们提出通过最大化与参考图像的一致性来微调T2I模型，同时惩罚对预训练模型的偏离。我们设计了一种新颖的T2I扩散模型训练目标，该目标以最小化对预训练模型的微调实现一致性。我们的方法名为“直接一致性优化”（Direct Consistency Optimization），其简便性与常规扩散损失相当，同时显著增强了个性化T2I模型的组合能力。此外，该方法引入了一种新的采样方式，可控制图像保真度与提示保真度之间的权衡。最后，我们强调使用参考图像的完整描述来进一步提升图文对齐的必要性。我们在以主体、风格或两者为目标的T2I个性化任务上验证了所提方法的有效性。具体而言，我们的方法在帕累托前沿（Pareto frontier）上优于基线方法。生成示例及代码见项目页面（https://dco-t2i.github.io/）。