Large language models (LLMs) have shown remarkable proficiency in human-level reasoning and generation capabilities, which encourages extensive research on their application in mathematical problem solving. However, current work has been largely focused on text-based mathematical problems, with limited investigation in problems involving geometric information. Addressing this gap, we aim to enable LLMs to solve geometric problems by understanding image input. We first analyze the limitations of current Multimodal Large Language Models (MLLMs) in this area: they struggle to accurately comprehending basic geometric elements and their relationships. To overcome these challenges, we take advantage of the unique characteristics of geometric problems (such as unique geometric logical form, and geometric scalability) and the capacity of the textual LLMs to build an enriched multimodal geometry dataset based on existing data. The augmented dataset, Geo170K, contains more than 170K geometric image-caption and question-answer pairs. Utilizing our constructed Geo170K dataset, we develop G-LLaVA, which demonstrates exceptional performance in solving geometric problems, significantly outperforming GPT-4-V on the MathVista benchmark with only 7B parameters.

翻译：大型语言模型（LLMs）在类人推理与生成能力方面展现出卓越性能，这推动了其在数学问题求解领域的广泛应用研究。然而，现有工作主要聚焦于基于文本的数学问题，对涉及几何信息的问题研究有限。为填补这一空白，我们旨在通过理解图像输入使LLMs能够解决几何问题。本文首先分析了当前多模态大语言模型（MLLMs）在此领域的局限性：它们难以准确理解基本几何元素及其关系。为解决这些挑战，我们利用几何问题的独特性（如独特的几何逻辑形式与几何可扩展性）以及文本LLMs的能力，基于现有数据构建了增强的多模态几何数据集。该增强数据集Geo170K包含超过17万对几何图像-描述与问答对。基于自主构建的Geo170K数据集，我们开发了G-LLaVA模型，该模型在仅含70亿参数的情况下，在MathVista基准测试中求解几何问题的性能显著优于GPT-4-V。