Recent advancements in large language models (LLMs) and multi-modal models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2,000 problems, a 750 problems subset focusing on backward reasoning, an augmented subset of 2,000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs in solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67\% accuracy rate on the main subset but only a 6.00\% accuracy on the hard subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities.

翻译：近期，大语言模型（LLMs）和多模态模型（MMs）在问题求解方面展现出卓越能力。然而，它们在几何数学问题上的熟练度——这类问题需要同时理解文本和视觉信息——尚未得到全面评估。为弥补这一空白，我们提出了GeoEval基准数据集，该综合数据集包含2000道题目的主测试集、750道侧重逆向推理的题目子集、2000道增强题目子集以及300道难题子集。这一基准能够深入探究LLMs和MMs在几何数学问题求解中的表现。我们对十个LLMs和MMs在各子集上的评估显示，WizardMath模型表现最佳，在主测试集上达到55.67%的准确率，但在难题子集上仅达6.00%。这凸显了针对模型未预训练数据集进行测试的迫切需求。此外，研究结果表明，GPT系列模型在回答经自身重新表述的问题时表现更优，这为提升模型能力提供了一种可行方法。