Understanding relations between objects is crucial for understanding the semantics of a visual scene. It is also an essential step in order to bridge visual and language models. However, current state-of-the-art computer vision models still lack the ability to perform spatial reasoning well. Existing datasets mostly cover a relatively small number of spatial relations, all of which are static relations that do not intrinsically involve motion. In this paper, we propose the Spatial and Temporal Understanding of Prepositions Dataset (STUPD) -- a large-scale video dataset for understanding static and dynamic spatial relationships derived from prepositions of the English language. The dataset contains 150K visual depictions (videos and images), consisting of 30 distinct spatial prepositional senses, in the form of object interaction simulations generated synthetically using Unity3D. In addition to spatial relations, we also propose 50K visual depictions across 10 temporal relations, consisting of videos depicting event/time-point interactions. To our knowledge, no dataset exists that represents temporal relations through visual settings. In this dataset, we also provide 3D information about object interactions such as frame-wise coordinates, and descriptions of the objects used. The goal of this synthetic dataset is to help models perform better in visual relationship detection in real-world settings. We demonstrate an increase in the performance of various models over 2 real-world datasets (ImageNet-VidVRD and Spatial Senses) when pretrained on the STUPD dataset, in comparison to other pretraining datasets.

翻译：理解物体间的关系对于把握视觉场景的语义至关重要，也是连接视觉与语言模型的关键步骤。然而，当前最先进的计算机视觉模型仍缺乏良好的空间推理能力。现有数据集大多覆盖数量较少的空间关系，且均为静态关系，本质不涉及运动。本文提出空间与时间介词理解数据集（STUPD）——一个大规模视频数据集，用于理解源自英语介词的静态与动态空间关系。该数据集包含150K个视觉表征（视频和图像），涵盖30种不同的空间介词义，以使用Unity3D合成生成的物体交互模拟形式呈现。除空间关系外，我们还提出涵盖10种时间关系的50K个视觉表征，由描述事件/时间点交互的视频组成。据我们所知，尚无数据集以视觉形式表征时间关系。本数据集中还提供物体交互的三维信息（如逐帧坐标）及所用物体的描述。该合成数据集旨在辅助模型在真实场景中提升视觉关系检测性能。我们证明，相较于其他预训练数据集，在STUPD数据集上预训练的各模型在两个真实场景数据集（ImageNet-VidVRD和Spatial Senses）上的性能均有所提升。