In this work, we introduce a novel evaluation paradigm for Large Language Models, one that challenges them to engage in meta-reasoning. This approach addresses critical shortcomings in existing math problem-solving benchmarks, traditionally used to evaluate the cognitive capabilities of agents. Our paradigm shifts the focus from result-oriented assessments, which often overlook the reasoning process, to a more holistic evaluation that effectively differentiates the cognitive capabilities among models. For example, in our benchmark, GPT-4 demonstrates a performance ten times more accurate than GPT3-5. The significance of this new paradigm lies in its ability to reveal potential cognitive deficiencies in LLMs that current benchmarks, such as GSM8K, fail to uncover due to their saturation and lack of effective differentiation among varying reasoning abilities. Our comprehensive analysis includes several state-of-the-art math models from both open-source and closed-source communities, uncovering fundamental deficiencies in their training and evaluation approaches. This paper not only advocates for a paradigm shift in the assessment of LLMs but also contributes to the ongoing discourse on the trajectory towards Artificial General Intelligence (AGI). By promoting the adoption of meta-reasoning evaluation methods similar to ours, we aim to facilitate a more accurate assessment of the true cognitive abilities of LLMs.

翻译：在本研究中，我们引入了一种针对大语言模型的新型评估范式，要求它们参与元推理。该方法弥补了现有数学问题求解基准（传统上用于评估智能体认知能力）中的关键缺陷。该范式将关注点从常忽略推理过程的结果导向评估，转向能有效区分模型间认知能力的整体性评估。例如，在我们的基准测试中，GPT-4的准确率比GPT-3.5高出十倍。此新范式的重要性在于其能揭示当前基准（如GSM8K）因饱和及缺乏对推理能力的有效区分而无法发现的潜在认知缺陷。我们全面分析了来自开源和闭源社区的多个最先进数学模型，揭示了其训练与评估方法中的根本性缺陷。本文不仅倡导大语言模型评估范式转变，还推动了关于迈向通用人工智能路径的持续讨论。通过推广类似于我们的元推理评估方法，我们旨在更准确地评估大语言模型的真实认知能力。