In this paper, we explore the potential of Vision-Language Models (VLMs), specifically CLIP, in predicting visual object relationships, which involves interpreting visual features from images into language-based relations. Current state-of-the-art methods use complex graphical models that utilize language cues and visual features to address this challenge. We hypothesize that the strong language priors in CLIP embeddings can simplify these graphical models paving for a simpler approach. We adopt the UVTransE relation prediction framework, which learns the relation as a translational embedding with subject, object, and union box embeddings from a scene. We systematically explore the design of CLIP-based subject, object, and union-box representations within the UVTransE framework and propose CREPE (CLIP Representation Enhanced Predicate Estimation). CREPE utilizes text-based representations for all three bounding boxes and introduces a novel contrastive training strategy to automatically infer the text prompt for union-box. Our approach achieves state-of-the-art performance in predicate estimation, mR@5 27.79, and mR@20 31.95 on the Visual Genome benchmark, achieving a 15.3\% gain in performance over recent state-of-the-art at mR@20. This work demonstrates CLIP's effectiveness in object relation prediction and encourages further research on VLMs in this challenging domain.

翻译：本文探索了视觉语言模型（VLMs），特别是CLIP，在预测视觉对象关系方面的潜力，该任务涉及将图像中的视觉特征解释为基于语言的关系。当前最先进的方法采用复杂的图模型，利用语言线索和视觉特征来应对这一挑战。我们假设CLIP嵌入中的强语言先验能够简化这些图模型，从而催生一种更简洁的方法。我们采用UVTransE关系预测框架，该框架将关系学习为场景中主语、宾语和联合框嵌入的平移向量。我们系统性地探索了在UVTransE框架内基于CLIP的主语、宾语和联合框表示的设计，并提出了CREPE（CLIP表示增强的谓词估计）。CREPE对所有三个边界框采用基于文本的表示，并引入了一种新颖的对比训练策略来自动推断联合框的文本提示。我们的方法在谓词估计上达到了最先进性能，在Visual Genome基准上mR@5为27.79，mR@20为31.95，相较于近期最先进方法在mR@20上取得了15.3%的性能提升。这项工作证明了CLIP在对象关系预测中的有效性，并鼓励在此具有挑战性的领域内进一步研究VLMs。