Text-to-image (T2I) models based on diffusion processes have achieved remarkable success in controllable image generation using user-provided captions. However, the tight coupling between the current text encoder and image decoder in T2I models makes it challenging to replace or upgrade. Such changes often require massive fine-tuning or even training from scratch with the prohibitive expense. To address this problem, we propose GlueGen, which applies a newly proposed GlueNet model to align features from single-modal or multi-modal encoders with the latent space of an existing T2I model. The approach introduces a new training objective that leverages parallel corpora to align the representation spaces of different encoders. Empirical results show that GlueNet can be trained efficiently and enables various capabilities beyond previous state-of-the-art models: 1) multilingual language models such as XLM-Roberta can be aligned with existing T2I models, allowing for the generation of high-quality images from captions beyond English; 2) GlueNet can align multi-modal encoders such as AudioCLIP with the Stable Diffusion model, enabling sound-to-image generation; 3) it can also upgrade the current text encoder of the latent diffusion model for challenging case generation. By the alignment of various feature representations, the GlueNet allows for flexible and efficient integration of new functionality into existing T2I models and sheds light on X-to-image (X2I) generation.

翻译：基于扩散过程的文本到图像（T2I）模型在利用用户提供的描述进行可控图像生成方面取得了显著成功。然而，当前T2I模型中文本编码器与图像解码器之间的紧密耦合使得替换或升级面临挑战。此类变更通常需要大量微调，甚至以高昂代价从头开始训练。为解决该问题，我们提出GlueGen，该方法利用新提出的GlueNet模型将单模态或多模态编码器的特征与现有T2I模型的潜在空间对齐。该方案引入一种新的训练目标，通过并行语料库实现不同编码器表示空间的对齐。实验结果表明，GlueNet可高效训练，并实现超越先前最优模型的多种能力：1）多语言语言模型（如XLM-Roberta）可与现有T2I模型对齐，从而基于非英语描述生成高质量图像；2）GlueNet可将多模态编码器（如AudioCLIP）与Stable Diffusion模型对齐，实现声音到图像生成；3）该模型还能升级潜在扩散模型的当前文本编码器以处理挑战性案例生成。通过多种特征表示的对齐，GlueNet实现了灵活高效地向现有T2I模型集成新功能，并为X到图像（X2I）生成开辟了道路。