We propose novel evaluations for mathematical reasoning capabilities of Large Language Models (LLMs) based on mathematical misconceptions. Our primary approach is to simulate LLMs as a novice learner and an expert tutor, aiming to identify the incorrect answer to math question resulted from a specific misconception and to recognize the misconception(s) behind an incorrect answer, respectively. Contrary to traditional LLMs-based mathematical evaluations that focus on answering math questions correctly, our approach takes inspirations from principles in educational learning sciences. We explicitly ask LLMs to mimic a novice learner by answering questions in a specific incorrect manner based on incomplete knowledge; and to mimic an expert tutor by identifying misconception(s) corresponding to an incorrect answer to a question. Using simple grade-school math problems, our experiments reveal that, while LLMs can easily answer these questions correctly, they struggle to identify 1) the incorrect answer corresponding to specific incomplete knowledge (misconceptions); 2) the misconceptions that explain particular incorrect answers. Our study indicates new opportunities for enhancing LLMs' math reasoning capabilities, especially on developing robust student simulation and expert tutoring models in the educational applications such as intelligent tutoring systems.

翻译：我们提出基于数学误解对大语言模型（LLMs）的数学推理能力进行新颖评估。主要方法是将LLMs模拟为新手学习者和专家导师，分别针对特定误解导致的数学问题错误答案进行识别，以及分析错误答案背后的误解。不同于传统基于LLMs的数学评估（聚焦于正确回答数学问题），我们的方法借鉴教育学习科学原理，明确要求LLMs模仿新手学习者基于不完整知识以特定错误方式回答问题；同时模仿专家导师识别与问题错误答案对应的误解。通过简单的初等数学问题实验发现，尽管LLMs能轻松正确回答这些问题，但在以下两方面存在困难：1）识别特定不完整知识（误解）对应的错误答案；2）解释特定错误答案的误解。本研究揭示了增强LLMs数学推理能力的新机遇，特别是在智能辅导系统等教育应用中开发稳健的学生模拟与专家辅导模型方面。