We propose a method to fuse frozen text-only large language models (LLMs) with pre-trained image encoder and decoder models, by mapping between their embedding spaces. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue. Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs. To achieve strong performance on image generation, we propose an efficient mapping network to ground the LLM to an off-the-shelf text-to-image generation model. This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs. Our approach outperforms baseline generation models on tasks with longer and more complex language. In addition to novel image generation, our model is also capable of image retrieval from a prespecified dataset, and decides whether to retrieve or generate at inference time. This is done with a learnt decision module which conditions on the hidden representations of the LLM. Our model exhibits a wider range of capabilities compared to prior multimodal language models. It can process image-and-text inputs, and produce retrieved images, generated images, and generated text -- outperforming non-LLM based generation models across several text-to-image tasks that measure context dependence.

翻译：我们提出一种方法，通过映射预训练文本大语言模型（LLMs）与图像编码器及解码器模型之间的嵌入空间，实现冻结文本大语言模型与视觉模型的融合。该模型展现广泛的多模态能力：图像检索、新型图像生成及多模态对话。这是首个能够基于任意交错的图像与文本输入生成连贯图像（及文本）输出的方法。为在图像生成中取得优异性能，我们设计了一个高效映射网络，将LLM与现成文生图模型对接。该网络将文本隐表示转化为视觉模型的嵌入空间，从而利用LLM强大的文本表征生成视觉输出。在涉及更长、更复杂语言的任务中，我们的方法优于基准生成模型。除新型图像生成外，模型还能从预设数据集中检索图像，并在推理时通过一个基于LLM隐表示学习的决策模块自主选择检索或生成模式。相较于先前多模态语言模型，我们的模型具备更广泛的能力，可处理图文混合输入并生成检索图像、生成图像及生成文本——在多个衡量上下文依赖性的文生图任务中超越非LLM基础生成模型。