Recent advances in text-to-image diffusion models have achieved remarkable success in generating high-quality, realistic images from textual descriptions. However, these approaches have faced challenges in precisely aligning the generated visual content with the textual concepts described in the prompts. In this paper, we propose a two-stage coarse-to-fine semantic re-alignment method, named RealignDiff, aimed at improving the alignment between text and images in text-to-image diffusion models. In the coarse semantic re-alignment phase, a novel caption reward, leveraging the BLIP-2 model, is proposed to evaluate the semantic discrepancy between the generated image caption and the given text prompt. Subsequently, the fine semantic re-alignment stage employs a local dense caption generation module and a re-weighting attention modulation module to refine the previously generated images from a local semantic view. Experimental results on the MS-COCO benchmark demonstrate that the proposed two-stage coarse-to-fine semantic re-alignment method outperforms other baseline re-alignment techniques by a substantial margin in both visual quality and semantic similarity with the input prompt.

翻译：近期文本到图像扩散模型在根据文本描述生成高质量、逼真图像方面取得了显著进展。然而，这些方法在精确对齐生成视觉内容与提示词所描述的文本概念方面仍面临挑战。本文提出一种名为RealignDiff的两阶段粗到细语义重对齐方法，旨在提升文本到图像扩散模型中文本与图像的对齐效果。在粗语义重对齐阶段，我们提出一种利用BLIP-2模型的新型标题奖励机制，用于评估生成图像标题与给定文本提示之间的语义差异。随后，在细语义重对齐阶段，通过局部密集标题生成模块和重加权注意力调制模块，从局部语义视角对先前生成的图像进行精细化调整。在MS-COCO基准上的实验结果表明，所提出的两阶段粗到细语义重对齐方法在视觉质量和与输入提示的语义相似度方面均显著优于其他基线重对齐技术。