The ability of Large Language Models (LLMs) to critique and refine their reasoning is crucial for their application in evaluation, feedback provision, and self-improvement. This paper introduces CriticBench, a comprehensive benchmark designed to assess LLMs' abilities to critique and rectify their reasoning across a variety of tasks. CriticBench encompasses five reasoning domains: mathematical, commonsense, symbolic, coding, and algorithmic. It compiles 15 datasets and incorporates responses from three LLM families. Utilizing CriticBench, we evaluate and dissect the performance of 17 LLMs in generation, critique, and correction reasoning, i.e., GQC reasoning. Our findings reveal: (1) a linear relationship in GQC capabilities, with critique-focused training markedly enhancing performance; (2) a task-dependent variation in correction effectiveness, with logic-oriented tasks being more amenable to correction; (3) GQC knowledge inconsistencies that decrease as model size increases; and (4) an intriguing inter-model critiquing dynamic, where stronger models are better at critiquing weaker ones, while weaker models can surprisingly surpass stronger ones in their self-critique. We hope these insights into the nuanced critique-correct reasoning of LLMs will foster further research in LLM critique and self-improvement.

翻译：大语言模型（LLMs）对其推理过程进行批判与优化的能力，对于其在评估、反馈提供及自我改进等领域的应用至关重要。本文提出CriticBench——一个旨在系统评估LLMs在不同任务中批判并修正自身推理能力的综合基准。CriticBench涵盖数学、常识、符号、编程与算法五大推理领域，整合15个数据集，并纳入三类LLM家族的响应。借助CriticBench，我们对17个LLM在生成、批判与修正推理（即GQC推理）中的性能进行评估与剖析。研究发现：（1）GQC能力呈现线性关系，以批判为核心的训练显著提升性能；（2）修正效果因任务而异，逻辑导向型任务更易修正；（3）GQC知识不一致性随模型规模增大而降低；（4）存在有趣的跨模型批判动态：强模型更善于批判弱模型，而弱模型在自我批判中反而可能超越强模型。我们期望这些关于LLMs精细批判性推理的洞见，能推动LLM批判与自我改进领域的进一步研究。