In this work, we introduce a novel evaluation paradigm for Large Language Models, one that challenges them to engage in meta-reasoning. This approach addresses critical shortcomings in existing math problem-solving benchmarks, traditionally used to evaluate the cognitive capabilities of agents. Our paradigm shifts the focus from result-oriented assessments, which often overlook the reasoning process, to a more holistic evaluation that effectively differentiates the cognitive capabilities among models. For example, in our benchmark, GPT-4 demonstrates a performance five times better than GPT3-5. The significance of this new paradigm lies in its ability to reveal potential cognitive deficiencies in LLMs that current benchmarks, such as GSM8K, fail to uncover due to their saturation and lack of effective differentiation among varying reasoning abilities. Our comprehensive analysis includes several state-of-the-art math models from both open-source and closed-source communities, uncovering fundamental deficiencies in their training and evaluation approaches. This paper not only advocates for a paradigm shift in the assessment of LLMs but also contributes to the ongoing discourse on the trajectory towards Artificial General Intelligence (AGI). By promoting the adoption of meta-reasoning evaluation methods similar to ours, we aim to facilitate a more accurate assessment of the true cognitive abilities of LLMs.

翻译：本文提出了一种新颖的大语言模型评估范式，旨在促使模型进行元推理。该范式针对现有数学问题求解基准（传统上用于评估智能体认知能力）的关键缺陷进行了改进。我们将评估焦点从结果导向（往往忽略推理过程）转向更全面的评估，从而有效区分不同模型的认知能力。例如，在我们的基准测试中，GPT-4的表现比GPT-3.5高出五倍。这一新范式的意义在于，它能够揭示当前基准（如GSM8K）因饱和且无法有效区分不同推理能力而未能发现的LLM潜在认知缺陷。我们的综合分析涵盖来自开源和闭源社区的多个最先进数学推理模型，揭示了其训练和评估方法中的根本性缺陷。本文不仅倡导大语言模型评估范式的转变，也为当前关于通用人工智能发展路径的讨论提供了贡献。通过推广类似本研究的元推理评估方法，我们力求更准确地评估大语言模型的真实认知能力。