Text-to-image (T2I) models have recently experienced rapid development, achieving astonishing performance in terms of fidelity and textual alignment capabilities. However, given a long paragraph (up to 512 words), these generation models still struggle to achieve strong alignment and are unable to generate images depicting complex scenes. In this paper, we introduce an information-enriched diffusion model for paragraph-to-image generation task, termed ParaDiffusion, which delves into the transference of the extensive semantic comprehension capabilities of large language models to the task of image generation. At its core is using a large language model (e.g., Llama V2) to encode long-form text, followed by fine-tuning with LORA to alignthe text-image feature spaces in the generation task. To facilitate the training of long-text semantic alignment, we also curated a high-quality paragraph-image pair dataset, namely ParaImage. This dataset contains a small amount of high-quality, meticulously annotated data, and a large-scale synthetic dataset with long text descriptions being generated using a vision-language model. Experiments demonstrate that ParaDiffusion outperforms state-of-the-art models (SD XL, DeepFloyd IF) on ViLG-300 and ParaPrompts, achieving up to 15% and 45% human voting rate improvements for visual appeal and text faithfulness, respectively. The code and dataset will be released to foster community research on long-text alignment.

翻译：文本到图像（T2I）模型近年来发展迅速，在保真度和文本对齐能力方面取得了令人瞩目的成果。然而，面对长段落（长达512词）输入时，这些生成模型仍难以实现强对齐，且无法生成描绘复杂场景的图像。本文提出了一种面向段落到图像生成任务的信息增强扩散模型——ParaDiffusion，该模型致力于将大语言模型强大的语义理解能力迁移至图像生成任务。其核心在于利用大语言模型（如Llama V2）编码长文本，并通过低秩适应（LoRA）微调对齐生成任务中的文本-图像特征空间。为促进长文本语义对齐的训练，我们还构建了高质量段落-图像对数据集ParaImage，该数据集包含少量高质量、精细标注的数据，以及一个利用视觉语言模型生成的长文本描述的大规模合成数据集。实验表明，ParaDiffusion在ViLG-300和ParaPrompts数据集上均优于现有最优模型（SD XL、DeepFloyd IF），在视觉吸引力和文本保真度方面分别实现了高达15%和45%的人类投票率提升。相关代码与数据集将公开发布，以推动长文本对齐领域的社区研究。