Compositional reasoning is a hallmark of human visual intelligence; yet despite the size of large vision-language models, they struggle to represent simple compositions by combining objects with their attributes. To measure this lack of compositional capability, we design Cola, a text-to-image retrieval benchmark to Compose Objects Localized with Attributes. To solve Cola, a model must retrieve images with the correct configuration of attributes and objects, and avoid choosing a distractor image with the same objects and attributes but in the wrong configuration. Cola contains about 1.2k composed queries of 168 objects and 197 attributes on around 30K images. Our human evaluation finds that Cola is 83.33% accurate, similar to contemporary compositionality benchmarks. Using Cola as a testbed, we explore empirical modeling designs to adapt pre-trained vision-language models to reason compositionally. We explore 6 adaptation strategies on 2 seminal vision-language models, using compositionality-centric test benchmarks - Cola and CREPE. We find the optimal adaptation strategy is to train a multimodal attention layer that jointly attends over the frozen pre-trained image and language features. Surprisingly, training multimodal layers on CLIP performs better than tuning a larger FLAVA model with already pre-trained multimodal layers. Furthermore, our adaptation strategy improves CLIP and FLAVA to comparable levels, suggesting that training multimodal layers using contrastive attribute-object data is key, as opposed to using them pre-trained. Lastly, we show that Cola is harder than a closely related contemporary benchmark, CREPE, since simpler fine-tuning strategies without multimodal layers suffice on CREPE, but not on Cola. However, we still see a significant gap between our best adaptation and human accuracy, suggesting considerable room for further research.

翻译：摘要：组合推理是人类视觉智能的标志性特征；然而，尽管大型视觉-语言模型规模庞大，它们在通过将对象与属性结合来表示简单组合方面仍存在困难。为衡量这种组合能力的不足，我们设计了Cola——一个通过属性定位对象组合的文本到图像检索基准。要解决Cola任务，模型必须检索出具有正确属性与对象配置的图像，并避免选择包含相同对象与属性但配置错误的干扰图像。Cola包含约1200个组合查询，涉及168个对象和197个属性，覆盖约3万张图像。人工评估显示，Cola的准确率为83.33%，与当代组合性基准相当。以Cola为测试平台，我们探索了使预训练视觉-语言模型具备组合推理能力的经验性建模设计。针对两种开创性视觉-语言模型，我们基于组合性核心测试基准（Cola和CREPE）探索了6种适配策略。研究发现，最优适配策略是训练一个多模态注意力层，使其联合关注冻结的预训练图像与语言特征。出乎意料的是，在CLIP上训练多模态层的效果优于微调已预训练多模态层的更大规模FLAVA模型。此外，我们的适配策略使CLIP和FLAVA达到可比水平，这表明关键因素在于使用对比性属性-对象数据训练多模态层，而非直接使用预训练的多模态层。最后，我们证明Cola比紧密相关的当代基准CREPE更具挑战性：无需多模态层的简单微调策略足以应对CREPE，但无法解决Cola。然而，最优适配方案与人类准确率之间仍存在显著差距，这表明该领域仍有大量研究空间。