The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map. We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https://github.com/jmerullo/limber

翻译：纯文本语言模型（LMs）在多大程度上能够学习表征非语言世界的特征是一个悬而未决的问题。先前研究表明，当视觉模型的参数被优化以在语言空间中编码图像时，预训练语言模型可以学会为图像生成描述。我们验证一个更强的假设：冻结的纯文本模型与纯视觉模型所学到的概念表征足够相似，以至于可以通过线性映射实现这一目标。我们证明，仅训练单个线性投影，即可将视觉模型的图像表征作为连续提示传递给冻结的语言模型。与同时微调图像编码器和文本解码器的模型（如MAGMA模型）相比，使用这些提示引导语言模型在图像描述和视觉问答任务上取得了具有竞争力的性能。我们比较了预训练过程中接收不同语言监督量的三种图像编码器：BEIT（无语言信息）、NF-ResNet（词汇类别信息）和CLIP（完整自然语言描述）。我们发现，所有三种编码器在向语言模型传递视觉属性信息（例如动物体型大小）方面表现相当，但经过语言监督预训练的编码器能更显著地编码类别信息（例如区分河马与大象），因此在标准语言-视觉基准任务上表现显著更优。我们的结果表明，语言模型对概念的表征方式在结构上与基于视觉的模型（甚至那些仅通过图像训练的模型）相似。代码见：https://github.com/jmerullo/limber