One critical prerequisite for faithful text-to-image generation is the accurate understanding of text inputs. Existing methods leverage the text encoder of the CLIP model to represent input prompts. However, the pre-trained CLIP model can merely encode English with a maximum token length of 77. Moreover, the model capacity of the text encoder from CLIP is relatively limited compared to Large Language Models (LLMs), which offer multilingual input, accommodate longer context, and achieve superior text representation. In this paper, we investigate LLMs as the text encoder to improve the language understanding in text-to-image generation. Unfortunately, training text-to-image generative model with LLMs from scratch demands significant computational resources and data. To this end, we introduce a three-stage training pipeline that effectively and efficiently integrates the existing text-to-image model with LLMs. Specifically, we propose a lightweight adapter that enables fast training of the text-to-image model using the textual representations from LLMs. Extensive experiments demonstrate that our model supports not only multilingual but also longer input context with superior image generation quality.

翻译：对于高保真度的文生图生成而言，一个关键前提是准确理解文本输入。现有方法通常使用CLIP模型的文本编码器来表示输入提示词。然而，预训练CLIP模型仅能编码最大标记长度为77的英文文本。此外，与大语言模型（LLMs）相比，CLIP文本编码器的模型容量相对有限，而LLMs不仅支持多语言输入、能容纳更长的上下文，还能实现更优的文本表征。本文探究了将LLMs用作文本编码器以提升文生图生成任务中的语言理解能力。但直接使用LLMs从头训练文生图生成模型需要消耗大量的计算资源和数据。为此，我们提出了一种三阶段训练流程，能够高效、有效地将现有文生图模型与LLMs结合。具体而言，我们设计了一个轻量级适配器，使得文生图模型能够利用LLMs的文本表征进行快速训练。大量实验表明，我们的模型不仅支持多语言输入，还能生成更优质量的图像，且输入文本的上下文长度得以显著扩展。