Most humans use visual imagination to understand and reason about language, but models such as BERT reason about language using knowledge acquired during text-only pretraining. In this work, we investigate whether vision-and-language pretraining can improve performance on text-only tasks that involve implicit visual reasoning, focusing primarily on zero-shot probing methods. We propose a suite of visual language understanding (VLU) tasks for probing the visual reasoning abilities of text encoder models, as well as various non-visual natural language understanding (NLU) tasks for comparison. We also contribute a novel zero-shot knowledge probing method, Stroop probing, for applying models such as CLIP to text-only tasks without needing a prediction head such as the masked language modelling head of models like BERT. We show that SOTA multimodally trained text encoders outperform unimodally trained text encoders on the VLU tasks while being underperformed by them on the NLU tasks, lending new context to previously mixed results regarding the NLU capabilities of multimodal models. We conclude that exposure to images during pretraining affords inherent visual reasoning knowledge that is reflected in language-only tasks that require implicit visual reasoning. Our findings bear importance in the broader context of multimodal learning, providing principled guidelines for the choice of text encoders used in such contexts.

翻译：摘要：大多数人类依赖视觉想象来理解和推理语言，但诸如BERT之类的模型仅通过纯文本预训练获取的知识来进行语言推理。本研究旨在探究视觉与语言预训练能否提升涉及隐性视觉推理的纯文本任务性能，并主要聚焦于零样本探测方法。我们提出了一套视觉语言理解（VLU）任务集，用于探测文本编码模型的视觉推理能力，同时准备了多种非视觉自然语言理解（NLU）任务作为对照。此外，我们创新性地引入了一种零样本知识探测方法——斯特鲁普探测（Stroop probing），该方法使CLIP等模型无需依赖BERT类模型中的掩码语言建模预测头即可应用于纯文本任务。实验表明，当前最先进的多模态训练文本编码器在VLU任务上优于单模态训练文本编码器，但在NLU任务上表现不及后者，这为先前关于多模态模型NLU能力的混合结论提供了新的解释视角。我们认为，预训练阶段对图像的接触赋予了模型固有的视觉推理知识，这种知识能体现在需要隐性视觉推理的纯语言任务中。本研究的发现对多模态学习的更广泛领域具有重要意义，为这类场景中文本编码器的选择提供了原则性指导。