Compositional reasoning is a hallmark of human visual intelligence. Yet, despite the size of large vision-language models, they struggle to represent simple compositions by combining objects with their attributes. To measure this lack of compositional capability, we design Cola, a text-to-image retrieval benchmark to Compose Objects Localized with Attributes. To solve Cola, a model must retrieve images with the correct configuration of attributes and objects and avoid choosing a distractor image with the same objects and attributes but in the wrong configuration. Cola contains about 1.2k composed queries of 168 objects and 197 attributes on around 30K images. Our human evaluation finds that Cola is 83.33% accurate, similar to contemporary compositionality benchmarks. Using Cola as a testbed, we explore empirical modeling designs to adapt pre-trained vision-language models to reason compositionally. We explore 6 adaptation strategies on 2 seminal vision-language models, using compositionality-centric test benchmarks - Cola and CREPE. We find the optimal adaptation strategy is to train a multi-modal attention layer that jointly attends over the frozen pre-trained image and language features. Surprisingly, training multimodal layers on CLIP performs better than tuning a larger FLAVA model with already pre-trained multimodal layers. Furthermore, our adaptation strategy improves CLIP and FLAVA to comparable levels, suggesting that training multimodal layers using contrastive attribute-object data is key, as opposed to using them pre-trained. Lastly, we show that Cola is harder than a closely related contemporary benchmark, CREPE, since simpler fine-tuning strategies without multimodal layers suffice on CREPE but not on Cola. However, we still see a significant gap between our best adaptation and human accuracy, suggesting considerable room for further research.

翻译：组合式推理是人类视觉智能的标志性能力。然而，尽管大规模视觉语言模型规模庞大，它们在通过组合对象及其属性来表示简单组合关系时仍存在困难。为衡量这种组合能力的缺失，我们设计了Cola——一个基于对象属性定位的文本到图像检索基准测试。要解决Cola任务，模型必须检索到具有正确属性-对象配置的图像，同时避免选择那些包含相同对象和属性但配置错误的干扰图像。Cola包含约1200个组合查询，涉及168个对象和197个属性，覆盖约3万张图像。人工评估表明Cola的准确率达83.33%，与当代组合性基准测试相当。以Cola为测试平台，我们探索了使预训练视觉语言模型具备组合推理能力的经验性建模方案。我们基于两种经典视觉语言模型，在组合性核心测试基准（Cola和CREPE）上评估了6种适配策略。研究发现最佳适配策略是：训练一个能联合关注冻结的预训练图像特征与语言特征的多模态注意力层。令人意外的是，在CLIP上训练多模态层的效果优于微调已具备多模态层的更大模型FLAVA。此外，我们的适配策略使CLIP和FLAVA达到可比水平，这表明使用对比属性-对象数据训练多模态层是关键机制，而非直接使用预训练的多模态层。最后，我们发现Cola的难度高于同期相近基准CREPE——无需多模态层的简单微调策略足以应对CREPE，但在Cola上却失效。不过，当前最佳适配方案与人类准确率之间仍存在显著差距，这为后续研究留下了广阔空间。