Physical reasoning, which involves the interpretation, understanding, and prediction of object behavior in dynamic environments, remains a significant challenge for current Vision-Language Models (VLMs). In this work, we propose two methods to enhance VLMs' physical reasoning capabilities using simulated data. First, we fine-tune a pre-trained VLM using question-answer (QA) pairs generated from simulations relevant to physical reasoning tasks. Second, we introduce Physics Context Builders (PCBs), specialized VLMs fine-tuned to create scene descriptions enriched with physical properties and processes. During physical reasoning tasks, these PCBs can be leveraged as context to assist a Large Language Model (LLM) to improve its performance. We evaluate both of our approaches using multiple benchmarks, including a new stability detection QA dataset called Falling Tower, which includes both simulated and real-world scenes, and CLEVRER. We demonstrate that a small QA fine-tuned VLM can significantly outperform larger state-of-the-art foundational models. We also show that integrating PCBs boosts the performance of foundational LLMs on physical reasoning tasks. Using the real-world scenes from the Falling Tower dataset, we also validate the robustness of both approaches in Sim2Real transfer. Our results highlight the utility that simulated data can have in the creation of learning systems capable of advanced physical reasoning.

翻译：物理推理涉及对动态环境中物体行为的解释、理解与预测，这对当前视觉-语言模型（VLMs）仍构成重大挑战。本研究提出两种利用仿真数据增强VLMs物理推理能力的方法。首先，我们使用与物理推理任务相关的仿真生成的问答对（QA）对预训练VLM进行微调。其次，我们引入物理上下文构建器（PCBs）——专门微调的VLMs，用于生成富含物理属性与过程的场景描述。在执行物理推理任务时，可将PCBs作为上下文辅助大型语言模型（LLM）以提升其性能。我们通过多个基准测试评估这两种方法，包括包含仿真与现实场景的新型稳定性检测问答数据集“坠落高塔”（Falling Tower）以及CLEVRER。实验表明，经少量问答数据微调的VLM能显著超越规模更大的前沿基础模型。我们还证明，集成PCBs能提升基础LLMs在物理推理任务上的表现。通过“坠落高塔”数据集的真实场景，我们进一步验证了两种方法在仿真到现实迁移中的鲁棒性。研究结果凸显了仿真数据在构建具备高级物理推理能力的学习系统中的实用价值。