Advancements in text-to-image diffusion models have broadened extensive downstream practical applications, but such models often encounter misalignment issues between text and image. Taking the generation of a combination of two disentangled concepts as an example, say given the prompt "a tea cup of iced coke", existing models usually generate a glass cup of iced coke because the iced coke usually co-occurs with the glass cup instead of the tea one during model training. The root of such misalignment is attributed to the confusion in the latent semantic space of text-to-image diffusion models, and hence we refer to the "a tea cup of iced coke" phenomenon as Latent Concept Misalignment (LC-Mis). We leverage large language models (LLMs) to thoroughly investigate the scope of LC-Mis, and develop an automated pipeline for aligning the latent semantics of diffusion models to text prompts. Empirical assessments confirm the effectiveness of our approach, substantially reducing LC-Mis errors and enhancing the robustness and versatility of text-to-image diffusion models. The code and dataset are here: https://github.com/RossoneriZhao/iced_coke.

翻译：文本到图像扩散模型的进展拓宽了广泛的下游实际应用，但此类模型经常遇到文本与图像之间的错位问题。以生成两个解耦概念的组合为例，例如给定提示词"一杯冰可乐的茶杯"，现有模型通常会生成一杯装有冰可乐的玻璃杯，因为在模型训练过程中，冰可乐通常与玻璃杯而非茶杯共同出现。这种错位的根源被归因于文本到图像扩散模型潜在语义空间的混淆，因此我们将"一杯冰可乐的茶杯"现象称为潜在概念错位（LC-Mis）。我们利用大型语言模型（LLMs）深入研究了LC-Mis的范围，并开发了一个自动化流程，用于将扩散模型的潜在语义与文本提示对齐。实证评估证实了我们方法的有效性，显著减少了LC-Mis错误，并增强了文本到图像扩散模型的鲁棒性和通用性。代码和数据集位于：https://github.com/RossoneriZhao/iced_coke。