We investigate the knowledge of object affordances in pre-trained language models (LMs) and pre-trained Vision-Language models (VLMs). Transformers-based large pre-trained language models (PTLM) learn contextual representation from massive amounts of unlabeled text and are shown to perform impressively in downstream NLU tasks. In parallel, a growing body of literature shows that PTLMs fail inconsistently and non-intuitively, showing a lack of reasoning and grounding. To take a first step toward quantifying the effect of grounding (or lack thereof), we curate a novel and comprehensive dataset of object affordances -- GrAFFORD, characterized by 15 affordance classes. Unlike affordance datasets collected in vision and language domains, we annotate in-the-wild sentences with objects and affordances. Experimental results reveal that PTLMs exhibit limited reasoning abilities when it comes to uncommon object affordances. We also observe that pre-trained VLMs do not necessarily capture object affordances effectively. Through few-shot fine-tuning, we demonstrate improvement in affordance knowledge in PTLMs and VLMs. Our research contributes a novel dataset for language grounding tasks, and presents insights into LM capabilities, advancing the understanding of object affordances. Codes and data are available at https://github.com/sayantan11995/Affordance

翻译：我们研究了预训练语言模型（LMs）和预训练视觉-语言模型（VLMs）中的物体功能知识。基于Transformer的大型预训练语言模型（PTLM）通过大规模无标注文本学习上下文表征，并在下游自然语言理解任务中表现优异。与此同时，越来越多的文献表明PTLM会出现不一致且反直觉的失败，暴露出其缺乏推理和具身性。为初步量化具身性（或其缺失）的影响，我们精心构建了一个新颖且全面的物体功能数据集——GrAFFORD，包含15个功能类别。不同于视觉和语言领域现有的功能数据集，我们对野外句子中的物体及其功能进行了标注。实验结果表明，PTLM在处理不常见物体功能时推理能力有限。我们还观察到预训练VLM并未有效捕捉物体功能。通过少样本微调，我们证明了PTLM和VLM在功能知识方面的提升。本研究为语言具身任务提供了新数据集，并揭示了LM能力的洞见，推进了对物体功能的理解。代码与数据见https://github.com/sayantan11995/Affordance。