The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.

翻译：视觉语言模型（VLM）在推理能力方面的缺失始终是研究讨论的焦点。我们认为，这种行为源于其训练数据中存在的报告偏差。具体而言，人们在描述视觉内容时，默认会省略监督某些类型推理所需的隐含信息；例如，“今天在比赛现场！”比“一张37人站在场地后面的照片”更可能成为图像描述。我们通过语用学理论的视角，研究了主流视觉语言模型OpenCLIP、LLaVA-1.5和Molmo的训练数据，发现尽管这些语料库达到网络规模且/或经过合成生成，报告偏差仍导致四种推理技能（空间、时间、否定与计数）的表征不足。通过一组精心设计的基准测试，我们证明：（一）视觉语言模型在训练数据中因报告偏差而被抑制的上述推理类型上表现不佳；（二）与普遍观点相反，扩大数据规模、模型规模乃至扩展至多语言环境，并不会默认促使这些能力涌现；但值得期待的是，（三）引入专门为获取隐含信息而收集的标注数据是有效的。我们的研究结果强调，需要采用更具针对性的训练数据构建方法，而非单纯依赖规模扩张来催生推理能力。