The ability of Large Language Models (LLMs) to critique and refine their reasoning is crucial for their application in evaluation, feedback provision, and self-improvement. This paper introduces CriticBench, a comprehensive benchmark designed to assess LLMs' abilities to critique and rectify their reasoning across a variety of tasks. CriticBench encompasses five reasoning domains: mathematical, commonsense, symbolic, coding, and algorithmic. It compiles 15 datasets and incorporates responses from three LLM families. Utilizing CriticBench, we evaluate and dissect the performance of 17 LLMs in generation, critique, and correction reasoning, i.e., GQC reasoning. Our findings reveal: (1) a linear relationship in GQC capabilities, with critique-focused training markedly enhancing performance; (2) a task-dependent variation in correction effectiveness, with logic-oriented tasks being more amenable to correction; (3) GQC knowledge inconsistencies that decrease as model size increases; and (4) an intriguing inter-model critiquing dynamic, where stronger models are better at critiquing weaker ones, while weaker models can surprisingly surpass stronger ones in their self-critique. We hope these insights into the nuanced critique-correct reasoning of LLMs will foster further research in LLM critique and self-improvement.

翻译：大型语言模型（LLMs）对其推理过程进行批判与修正的能力，在其应用于评估、反馈提供及自我改进等场景中至关重要。本文提出CriticBench——一个旨在系统评估LLMs在多类型任务中批判修正推理能力的综合基准。CriticBench涵盖数学推理、常识推理、符号推理、代码生成与算法推理五大推理领域，整合了15个数据集并纳入三个LLM家族的响应结果。借助CriticBench，我们评估并剖析了17个LLMs在生成、批判与修正推理（即GQC推理）中的表现。研究发现：(1) GQC能力呈现线性相关关系，面向批判的专项训练可显著提升模型表现；(2) 修正效果存在任务依赖性，逻辑导向型任务更易通过修正获得提升；(3) 随着模型规模扩大，GQC知识不一致性逐渐降低；(4) 存在有趣的跨模型批判动态：强模型更善于批判弱模型，而弱模型在自我批判中反而可能超越强模型。我们期望这些关于LLMs批判性纠错推理的精细洞察，能推动LLM批判机制与自我改进领域的进一步研究。