This paper shows that text-only Language Models (LM) can learn to ground spatial relations like "left of" or "below" if they are provided with explicit location information of objects and they are properly trained to leverage those locations. We perform experiments on a verbalized version of the Visual Spatial Reasoning (VSR) dataset, where images are coupled with textual statements which contain real or fake spatial relations between two objects of the image. We verbalize the images using an off-the-shelf object detector, adding location tokens to every object label to represent their bounding boxes in textual form. Given the small size of VSR, we do not observe any improvement when using locations, but pretraining the LM over a synthetic dataset automatically derived by us improves results significantly when using location tokens. We thus show that locations allow LMs to ground spatial relations, with our text-only LMs outperforming Vision-and-Language Models and setting the new state-of-the-art for the VSR dataset. Our analysis show that our text-only LMs can generalize beyond the relations seen in the synthetic dataset to some extent, learning also more useful information than that encoded in the spatial rules we used to create the synthetic dataset itself.

翻译：本文表明，纯文本语言模型（LM）能够学习将“左侧”或“下方”等空间关系具象化，前提是为其提供对象的明确位置信息，并经过适当训练以利用这些位置信息。我们在视觉空间推理（VSR）数据集的文本化版本上进行实验，该数据集将图像与包含图像中两个对象之间真实或虚假空间关系的文本陈述配对。我们使用现成的目标检测器将图像文本化，为每个对象标签添加位置标记，以文本形式表示其边界框。由于VSR规模较小，使用位置信息时未观察到显著改进，但通过在由我们自动生成的合成数据集上对LM进行预训练，使用位置标记后结果显著提升。因此，我们证明位置信息使LM能够具象化空间关系，且我们的纯文本LM在VSR数据集上超越了视觉-语言模型，创下新的最优性能。分析表明，我们的纯文本LM能在一定程度上泛化至合成数据集之外的未见关系，并学习到比我们用于创建合成数据集的空间规则中所编码信息更丰富的内容。