Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. To address this limitation, we propose PhysObjects, an object-centric dataset of 36.9K crowd-sourced and 417K automated physical concept annotations of common household objects. We demonstrate that fine-tuning a VLM on PhysObjects improves its understanding of physical object concepts, by capturing human priors of these concepts from visual appearance. We incorporate this physically-grounded VLM in an interactive framework with a large language model-based robotic planner, and show improved planning performance on tasks that require reasoning about physical object concepts, compared to baselines that do not leverage physically-grounded VLMs. We additionally illustrate the benefits of our physically-grounded VLM on a real robot, where it improves task success rates. We release our dataset and provide further details and visualizations of our results at https://iliad.stanford.edu/pg-vlm/.

翻译：近年来，视觉-语言模型（VLM）在视觉问答和图像描述等任务中取得了显著进展，使其能够对物理世界进行推理，特别是在机器人操作等应用领域。然而，当前VLM对常见物体的物理概念（如材质、易碎性）理解有限，这限制了其在需要物体交互和物理推理的机器人操作任务中的实用性。为解决这一局限，我们提出PhysObjects数据集——一个包含36.9万条众包标注和41.7万条自动化标注的常见家居物体物理概念数据集。实验表明，通过微调VLM学习PhysObjects中的视觉外观与物理概念的人类先验，可显著提升模型对物理物体概念的理解。我们将该物理具身VLM集成到基于大语言模型的机器人规划交互框架中，相较于未使用物理具身VLM的基线方法，在需要物理概念推理的任务中展现出更优的规划性能。此外，我们通过实体机器人实验验证了物理具身VLM在提升任务成功率方面的优势。相关数据集、算法细节及可视化结果已开源至 https://iliad.stanford.edu/pg-vlm/ 。