We present NoReGeo, a novel benchmark designed to evaluate the intrinsic geometric understanding of large language models (LLMs) without relying on reasoning or algebraic computation. Unlike existing benchmarks that primarily assess models' proficiency in reasoning-based geometry-where solutions are derived using algebraic methods-NoReGeo focuses on evaluating whether LLMs can inherently encode spatial relationships and recognize geometric properties directly. Our benchmark comprises 2,500 trivial geometric problems spanning 25 categories, each carefully crafted to be solvable purely through native geometric understanding, assuming known object locations. We assess a range of state-of-the-art models on NoReGeo, including frontier models like GPT-4, observing that even the most advanced systems achieve an overall maximum of 65% accuracy in binary classification tasks. Further, our ablation experiments demonstrate that such geometric understanding does not emerge through fine-tuning alone, indicating that effective training for geometric comprehension requires a specialized approach from the outset. Our findings highlight a significant gap in current LLMs' ability to natively grasp geometric concepts, providing a foundation for future research toward models with true geometric cognition.

翻译：我们提出了NoReGeo，这是一个新颖的基准测试，旨在评估大型语言模型（LLMs）在不依赖推理或代数计算情况下的内在几何理解能力。与现有的主要评估模型在基于推理的几何问题（即使用代数方法求解）上熟练度的基准不同，NoReGeo专注于评估LLMs是否能够内在地编码空间关系并直接识别几何属性。我们的基准包含跨越25个类别的2500个简单几何问题，每个问题都经过精心设计，假设已知对象位置，仅通过本征的几何理解即可求解。我们评估了一系列在NoReGeo上的最先进模型，包括像GPT-4这样的前沿模型，观察到即使是最先进的系统在二元分类任务中也仅能达到最高65%的准确率。此外，我们的消融实验表明，这种几何理解能力并非仅通过微调就能产生，这表明有效的几何理解训练需要从一开始就采用专门的方法。我们的研究结果突显了当前LLMs在本征掌握几何概念能力方面存在的显著差距，为未来研究实现具有真正几何认知能力的模型奠定了基础。