Despite the impressive advancements achieved through vision-and-language pretraining, it remains unclear whether this joint learning paradigm can help understand each individual modality. In this work, we conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models by probing a broad range of tasks, aiming to assess the quality of the learned representations in a nuanced manner. Interestingly, our empirical observations suggest that vision-and-language models are better at label prediction tasks like object and attribute prediction, while vision-only models are stronger at dense prediction tasks that require more localized information. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models. Code will be released at https://github.com/Lizw14/visual_probing

翻译：尽管视觉-语言预训练取得了令人瞩目的进展，但这种联合学习范式是否有助于理解每个单独模态仍不清楚。在本研究中，我们通过探究一系列广泛的任务，对现有视觉-语言模型和纯视觉模型中的视觉表征进行比较分析，旨在以细致的方式评估所学表征的质量。有趣的是，我们的实证观察表明，视觉-语言模型在对象和属性预测等标签预测任务上表现更佳，而纯视觉模型在需要更多局部信息的密集预测任务上更强。我们希望我们的研究能揭示语言在视觉学习中的作用，并为各种预训练模型提供经验指导。代码将发布在https://github.com/Lizw14/visual_probing。