Text-to-image (T2I) models have recently experienced rapid development, achieving astonishing performance in terms of fidelity and textual alignment capabilities. However, given a long paragraph (up to 512 words), these generation models still struggle to achieve strong alignment and are unable to generate images depicting complex scenes. In this paper, we introduce an information-enriched diffusion model for paragraph-to-image generation task, termed ParaDiffusion, which delves into the transference of the extensive semantic comprehension capabilities of large language models to the task of image generation. At its core is using a large language model (e.g., Llama V2) to encode long-form text, followed by fine-tuning with LORA to alignthe text-image feature spaces in the generation task. To facilitate the training of long-text semantic alignment, we also curated a high-quality paragraph-image pair dataset, namely ParaImage. This dataset contains a small amount of high-quality, meticulously annotated data, and a large-scale synthetic dataset with long text descriptions being generated using a vision-language model. Experiments demonstrate that ParaDiffusion outperforms state-of-the-art models (SD XL, DeepFloyd IF) on ViLG-300 and ParaPrompts, achieving up to 15% and 45% human voting rate improvements for visual appeal and text faithfulness, respectively. The code and dataset will be released to foster community research on long-text alignment.

翻译：文本到图像（T2I）模型近期发展迅速，在保真度和文本对齐能力方面取得了惊人表现。然而，面对长达512词的长段落输入时，这些生成模型仍难以实现强对齐，无法生成描绘复杂场景的图像。本文针对段落到图像生成任务，提出一种信息增强扩散模型ParaDiffusion，致力于将大语言模型强大的语义理解能力迁移至图像生成任务。其核心在于使用大语言模型（如Llama V2）编码长文本，并通过LORA微调对齐生成任务中的文图特征空间。为促进长文本语义对齐训练，我们构建了高质量段落-图像对数据集ParaImage。该数据集包含少量高质量人工标注数据，以及利用视觉-语言模型生成的带长文本描述的大规模合成数据集。实验表明，ParaDiffusion在ViLG-300和ParaPrompts基准上优于当前最优模型（SD XL、DeepFloyd IF），视觉吸引力和文本忠实度的人类投票率分别提升15%和45%。代码与数据集将开源以促进长文本对齐领域的社区研究。