We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images. Our method leverages the abilities of language models learnt from large scale text-only pretraining, such as in-context learning and free-form text generation. We keep the language model frozen, and finetune input and output linear layers to enable cross-modality interactions. This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images. We achieve strong zero-shot performance on grounded tasks such as contextual image retrieval and multimodal dialogue, and showcase compelling interactive abilities. Our approach works with any off-the-shelf language model and paves the way towards an effective, general solution for leveraging pretrained language models in visually grounded settings.

翻译：我们提出一种高效方法，将预训练的纯文本语言模型接地到视觉领域，使其能够处理任意交错排列的图像-文本数据，并生成与检索图像交错的文本。该方法利用语言模型从大规模纯文本预训练中习得的能力（如上下文学习和自由文本生成）。我们保持语言模型参数冻结，仅微调输入和输出线性层以实现跨模态交互，从而使模型能够处理任意交错的图像-文本输入，并生成与检索图像交织的自由文本。在上下文图像检索和多模态对话等接地任务中，我们取得了强大的零样本性能，并展示了引人注目的交互能力。该方法适用于任何现成的语言模型，为在视觉接地场景中有效利用预训练语言模型的通用解决方案奠定了基础。