Consider the math problem: "Lily received 3 cookies from her best friend yesterday and ate 5 for breakfast. Today, her friend gave her 3 more cookies. How many cookies does Lily have now?" Many large language models (LLMs) in previous research approach this problem by calculating the answer "1" using the equation "3 - 5 + 3." However, from a human perspective, we recognize the inherent flaw in this problem: Lily cannot eat 5 cookies if she initially only had 3. This discrepancy prompts a key question: Are current LLMs merely Blind Solver that apply mathematical operations without deeper reasoning, or can they function as Logical Thinker capable of identifying logical inconsistencies? To explore this question, we propose a benchmark dataset, FaultyMath, which includes faulty math problems of rich diversity: i) multiple mathematical categories, e.g., algebra, geometry, number theory, etc., ii) varying levels of difficulty, and iii) different origins of faultiness -- ranging from violations of common sense and ambiguous statements to mathematical contradictions and more. We evaluate a broad spectrum of LLMs, including open-source, closed-source, and math-specialized models, using FaultyMath across three dimensions: (i) How accurately can the models detect faulty math problems without being explicitly prompted to do so? (ii) When provided with hints -- either correct or misleading -- about the validity of the problems, to what extent do LLMs adapt to become reliable Logical Thinker? (iii) How trustworthy are the explanations generated by LLMs when they recognize a math problem as flawed? Through extensive experimentation and detailed analysis, our results demonstrate that existing LLMs largely function as Blind Solver and fall short of the reasoning capabilities required to perform as Logical Thinker.

翻译：考虑以下数学问题："莉莉昨天从她最好的朋友那里收到了3块饼干，早餐吃了5块。今天，她的朋友又给了她3块饼干。莉莉现在有多少块饼干？" 先前研究中的许多大语言模型（LLMs）通过计算"3 - 5 + 3"得出答案"1"。然而，从人类视角来看，我们认识到这个问题存在内在缺陷：如果莉莉最初只有3块饼干，她不可能吃掉5块。这种差异引出了一个关键问题：当前的大语言模型仅仅是应用数学运算而不进行深度推理的"盲解者"，还是能够识别逻辑不一致性的"逻辑思考者"？为探究此问题，我们提出了一个基准数据集FaultyMath，其中包含多样化的缺陷数学问题：i) 涵盖多个数学类别，如代数、几何、数论等；ii) 具有不同难度级别；以及iii) 缺陷来源各异——从违反常识和表述模糊，到数学矛盾等。我们使用FaultyMath从三个维度评估了广泛的大语言模型，包括开源、闭源和数学专用模型：(i) 在未明确提示的情况下，模型检测缺陷数学问题的准确度如何？(ii) 当提供关于问题有效性的提示（无论是正确还是误导性提示）时，大语言模型在多大程度上能适应成为可靠的"逻辑思考者"？(iii) 当大语言模型识别出数学问题存在缺陷时，其生成的解释可信度如何？通过大量实验和详细分析，我们的结果表明，现有的大语言模型主要充当"盲解者"，尚未达到作为"逻辑思考者"所需的推理能力。