In this work, we introduce a novel evaluation paradigm for Large Language Models, one that challenges them to engage in meta-reasoning. This approach addresses critical shortcomings in existing math problem-solving benchmarks, traditionally used to evaluate the cognitive capabilities of agents. Our paradigm shifts the focus from result-oriented assessments, which often overlook the reasoning process, to a more holistic evaluation that effectively differentiates the cognitive capabilities among models. For example, in our benchmark, GPT-4 demonstrates a performance five times better than GPT3-5. The significance of this new paradigm lies in its ability to reveal potential cognitive deficiencies in LLMs that current benchmarks, such as GSM8K, fail to uncover due to their saturation and lack of effective differentiation among varying reasoning abilities. Our comprehensive analysis includes several state-of-the-art math models from both open-source and closed-source communities, uncovering fundamental deficiencies in their training and evaluation approaches. This paper not only advocates for a paradigm shift in the assessment of LLMs but also contributes to the ongoing discourse on the trajectory towards Artificial General Intelligence (AGI). By promoting the adoption of meta-reasoning evaluation methods similar to ours, we aim to facilitate a more accurate assessment of the true cognitive abilities of LLMs.

翻译：本文提出了一种全新的大型语言模型评估范式，要求模型进行元推理。该范式针对现有数学问题求解基准测试（传统上用于评估智能体认知能力）的关键缺陷进行了改进。我们的范式将关注点从结果导向评估（往往忽略推理过程）转向更全面的评估方式，从而有效区分不同模型之间的认知能力差异。例如，在我们的基准测试中，GPT-4的表现比GPT-3.5高出五倍。这一新范式的重要性在于，它能够揭示当前基准测试（如GSM8K）因过度饱和且无法有效区分不同推理能力而未能发现的潜在认知缺陷。我们全面分析了来自开源和闭源社区的多个最先进数学推理模型，揭示了其训练与评估方法中的根本性缺陷。本文不仅倡导大型语言模型评估的范式转变，也为当前关于通用人工智能（AGI）发展路径的讨论提供了新视角。通过推广类似我们这种基于元推理的评估方法，我们旨在更准确地评估大型语言模型的真实认知能力。