Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. To address this limitation, we propose PhysObjects, an object-centric dataset of 39.6K crowd-sourced and 417K automated physical concept annotations of common household objects. We demonstrate that fine-tuning a VLM on PhysObjects improves its understanding of physical object concepts, including generalization to held-out concepts, by capturing human priors of these concepts from visual appearance. We incorporate this physically-grounded VLM in an interactive framework with a large language model-based robotic planner, and show improved planning performance on tasks that require reasoning about physical object concepts, compared to baselines that do not leverage physically-grounded VLMs. We additionally illustrate the benefits of our physically-grounded VLM on a real robot, where it improves task success rates. We release our dataset and provide further details and visualizations of our results at https://iliad.stanford.edu/pg-vlm/.

翻译：近期视觉-语言模型（VLM）的进展显著提升了其在视觉问答和图像描述等任务上的表现。因此，这类模型已具备对物理世界进行推理的能力，尤其在机器人操控等应用领域。然而，当前VLM对常见物体的物理概念（如材质、易碎性）理解有限，这制约了其在涉及物体交互与物理推理的机器人操控任务中的实用性。为解决这一局限，我们提出PhysObjects数据集——包含39.6万条众包标注和41.7万条自动化标注的日常家居物体物理概念数据集。实验表明，通过在该数据集上微调VLM，模型能够通过视觉外观捕捉人类对物理概念的先验知识，从而提升对物理物体概念的理解（包括对未见过概念泛化的能力）。我们将此物理具身VLM集成至基于大语言模型的机器人规划器交互框架中，实验证明：相比未使用物理具身VLM的基线方法，我们的方法在需要推理物体物理概念的任务中规划性能显著提升。此外，通过在真实机器人上的部署验证，该物理具身VLM有效提升了任务成功率。我们已公开数据集，更多细节与可视化结果请访问https://iliad.stanford.edu/pg-vlm/。