Large language models have shown impressive results for multi-hop mathematical reasoning when the input question is only textual. Many mathematical reasoning problems, however, contain both text and image. With the ever-increasing adoption of vision language models (VLMs), understanding their reasoning abilities for such problems is crucial. In this paper, we evaluate the reasoning capabilities of VLMs along various axes through the lens of geometry problems. We procedurally create a synthetic dataset of geometry questions with controllable difficulty levels along multiple axes, thus enabling a systematic evaluation. The empirical results obtained using our benchmark for state-of-the-art VLMs indicate that these models are not as capable in subjects like geometry (and, by generalization, other topics requiring similar reasoning) as suggested by previous benchmarks. This is made especially clear by the construction of our benchmark at various depth levels, since solving higher-depth problems requires long chains of reasoning rather than additional memorized knowledge. We release the dataset for further research in this area.

翻译：大型语言模型在处理仅包含文本输入的多跳数学推理问题方面已展现出令人瞩目的成果。然而，许多数学推理问题同时包含文本和图像。随着视觉语言模型（VLM）的日益普及，理解它们在此类问题上的推理能力至关重要。本文以几何问题为切入点，从多个维度评估了VLM的推理能力。我们通过程序化方式生成了一组具有可控难度维度的合成几何问题数据集，从而实现系统性的评估。使用我们构建的基准对当前最先进的VLM进行实证分析的结果表明，这些模型在几何（以及推而广之的其他需要类似推理的领域）等学科上的能力并不如以往基准所显示的那么出色。这一点尤其体现在我们按不同深度层级构建的基准上，因为解决更深层次的问题需要长链推理，而非额外的记忆性知识。我们已发布该数据集，以支持该领域的进一步研究。