Physical reasoning is important for effective robot manipulation. Recent work has investigated both vision and language modalities for physical reasoning; vision can reveal information about objects in the environment and language serves as an abstraction and communication medium for additional context. Although these works have demonstrated success on a variety of physical reasoning tasks, they are limited to physical properties that can be inferred from visual or language inputs. In this work, we investigate combining tactile perception with language, which enables embodied systems to obtain physical properties through interaction and apply common-sense reasoning. We contribute a new dataset PhysiCleAR, which comprises both physical/property reasoning tasks and annotated tactile videos obtained using a GelSight tactile sensor. We then introduce Octopi, a system that leverages both tactile representation learning and large vision-language models to predict and reason about tactile inputs with minimal language fine-tuning. Our evaluations on PhysiCleAR show that Octopi is able to effectively use intermediate physical property predictions to improve physical reasoning in both trained tasks and for zero-shot reasoning. PhysiCleAR and Octopi are available on https://github.com/clear-nus/octopi.

翻译：物理推理对于机器人高效操作至关重要。近期研究从视觉和语言两个模态探讨了物理推理问题：视觉可揭示环境中物体的信息，而语言作为抽象与通信媒介，用于传递额外上下文。尽管这些研究已在多种物理推理任务中取得成果，但它们仅限于可从视觉或语言输入推断的物理属性。在本工作中，我们探索了将触觉感知与语言相结合的方法，使具身系统能够通过交互获取物理属性并应用常识推理。我们贡献了一个新数据集PhysiCleAR，其中包含物理/属性推理任务以及使用GelSight触觉传感器采集的标注触觉视频。随后，我们提出Octopi系统，该系统融合触觉表征学习与大视觉语言模型，以最少语言微调实现触觉输入的预测与推理。在PhysiCleAR上的评估表明，Octopi能够有效利用中间物理属性预测，提升在训练任务与零样本推理中的物理推理能力。PhysiCleAR与Octopi代码已开源至https://github.com/clear-nus/octopi。