The rapid advancement of Text-to-Image(T2I) generative models has enabled the synthesis of high-quality images guided by textual descriptions. Despite this significant progress, these models are often susceptible in generating contents that contradict the input text, which poses a challenge to their reliability and practical deployment. To address this problem, we introduce a novel diffusion-based framework to significantly enhance the alignment of generated images with their corresponding descriptions, addressing the inconsistency between visual output and textual input. Our framework is built upon a comprehensive analysis of inconsistency phenomena, categorizing them based on their manifestation in the image. Leveraging a state-of-the-art large language module, we first extract objects and construct a knowledge graph to predict the locations of these objects in potentially generated images. We then integrate a state-of-the-art controllable image generation model with a visual text generation module to generate an image that is consistent with the original prompt, guided by the predicted object locations. Through extensive experiments on an advanced multimodal hallucination benchmark, we demonstrate the efficacy of our approach in accurately generating the images without the inconsistency with the original prompt. The code can be accessed via https://github.com/TruthAI-Lab/PCIG.

翻译：文本到图像（T2I）生成模型的快速发展使得根据文本描述合成高质量图像成为可能。尽管取得了这一显著进展，这些模型在生成内容时常常容易与输入文本相矛盾，这对其可靠性和实际部署构成了挑战。为解决此问题，我们引入了一种新颖的基于扩散的框架，以显著增强生成图像与其对应描述之间的一致性，解决视觉输出与文本输入之间的不一致问题。我们的框架建立在对不一致现象进行全面分析的基础上，根据其在图像中的表现形式对其进行分类。利用先进的大语言模块，我们首先提取对象并构建知识图谱，以预测这些对象在可能生成的图像中的位置。随后，我们集成最先进的可控图像生成模型与视觉文本生成模块，在预测的对象位置引导下，生成一幅与原始提示保持一致的图像。通过在先进的多模态幻觉基准上进行大量实验，我们证明了我们的方法能够准确生成图像，且不与原始提示产生不一致性。代码可通过 https://github.com/TruthAI-Lab/PCIG 访问。