The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map. We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https://github.com/jmerullo/limber

翻译：纯文本语言模型（LMs）在多大程度上能够学习表征非语言世界的特征，这是一个悬而未决的问题。先前的研究表明，当视觉模型的参数被优化以在语言空间中编码图像时，预训练的语言模型可以被教会为图像生成描述。我们检验了一个更强的假设：冻结的纯文本模型与纯视觉模型所习得的概念表征足够相似，以至于可以通过线性映射实现这一目标。我们证明，仅通过训练一个单一的线性投影，即可将视觉模型的图像表征作为连续提示传递给冻结的语言模型。与同时调整图像编码器和文本解码器的模型（如MAGMA模型）相比，使用这些提示来驱动语言模型在图像描述和视觉问答任务上取得了具有竞争力的性能。我们比较了三个在预训练过程中接受不同程度语言监督的图像编码器：BEIT（无语言信息）、NF-ResNet（词汇类别信息）和CLIP（完整自然语言描述）。我们发现，所有三个编码器在将视觉属性信息（例如，动物是大还是小）传递给语言模型时表现同样出色，但经过语言监督预训练的编码器能更显著地编码类别信息（例如，区分河马与大象），因此在基准语言-视觉任务上表现显著更优。我们的结果表明，语言模型编码概念信息的方式在结构上与基于视觉的模型（甚至那些仅在图像上训练的模型）相似。代码可在此处获取：https://github.com/jmerullo/limber