Vision (image and video) - Language (VL) pre-training is the recent popular paradigm that achieved state-of-the-art results on multi-modal tasks like image-retrieval, video-retrieval, visual question answering etc. These models are trained in an unsupervised way and greatly benefit from the complementary modality supervision. In this paper, we explore if the language representations trained using vision supervision perform better than vanilla language representations on Natural Language Understanding and commonsense reasoning benchmarks. We experiment with a diverse set of image-text models such as ALBEF, BLIP, METER and video-text models like ALPRO, Frozen-in-Time (FiT), VIOLET. We compare the performance of language representations of stand-alone text encoders of these models to the language representations of text encoders learnt through vision supervision. Our experiments suggest that vanilla language representations show superior performance on most of the tasks. These results shed light on the current drawbacks of the vision-language models.

翻译：视觉（图像和视频）-语言（VL）预训练是近期流行的范式，在图像检索、视频检索、视觉问答等多模态任务上取得了最先进的结果。这些模型以无监督方式训练，并显著受益于互补模态的监督信号。本文探究了利用视觉监督训练的语言表示，是否在自然语言理解和常识推理基准测试中优于纯语言表示。我们实验了多种图像-文本模型（如ALBEF、BLIP、METER）和视频-文本模型（如ALPRO、Frozen-in-Time (FiT)、VIOLET），比较了这些模型独立文本编码器的语言表示与通过视觉监督学习的文本编码器语言表示的性能。实验结果表明，纯语言表示在大部分任务上表现更优。这些结果揭示了当前视觉-语言模型的缺陷。