Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode compositional information. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order. ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO & Flickr30k-Order, to test for order sensitivity. ARO is orders of magnitude larger than previous benchmarks of compositionality, with more than 50,000 test cases. We show where state-of-the-art VLMs have poor relational understanding, can blunder when linking objects to their attributes, and demonstrate a severe lack of order sensitivity. VLMs are predominantly trained and evaluated on large datasets with rich compositional structure in the images and captions. Yet, training on these datasets has not been enough to address the lack of compositional understanding, and evaluating on these datasets has failed to surface this deficiency. To understand why these limitations emerge and are not represented in the standard tests, we zoom into the evaluation and training procedures. We demonstrate that it is possible to perform well on retrieval over existing datasets without using the composition and order information. Given that contrastive pretraining optimizes for retrieval on datasets with similar shortcuts, we hypothesize that this can explain why the models do not need to learn to represent compositional information. This finding suggests a natural solution: composition-aware hard negative mining. We show that a simple-to-implement modification of contrastive learning significantly improves the performance on tasks requiring understanding of order and compositionality.

翻译：尽管大型视觉语言模型（VLMs）在许多下游应用中取得了成功，但它们编码组合信息的能力尚不明确。为此，我们构建了归因、关系与顺序（ARO）基准测试，系统评估VLMs理解不同类型关系、属性和顺序的能力。ARO包含Visual Genome归因测试（检验对物体属性的理解）、Visual Genome关系测试（检验关系理解能力）以及COCO与Flickr30k-顺序测试（检验顺序敏感性）。该基准测试的规模比以往的组合性基准测试大数个数量级，包含超过50,000个测试案例。我们发现，最先进的VLMs在关系理解方面表现不佳，在将物体与其属性关联时可能犯严重错误，并且严重缺乏顺序敏感性。尽管VLMs主要在富含图像与标题中组合结构的大型数据集上训练和评估，但仅在这些数据集上训练并不足以弥补组合理解的不足，且基于这些数据集的评估亦未能揭示这一缺陷。为探明这些局限为何出现且未在标准测试中显现，我们深入分析了评估与训练流程。结果表明，在不使用组合与顺序信息的情况下，模型仍能在现有数据集上的检索任务中表现优异。鉴于对比预训练优化的是在存在类似捷径的数据集上的检索性能，我们推测这解释了模型为何无需学习表征组合信息。这一发现提出了一种自然解决方案：组合感知难负样本挖掘。我们证明，对对比学习进行简单易行的修改，便能显著提升模型在需要理解顺序与组合性的任务上的表现。