Recently, o1-like models have drawn significant attention, where these models produce the long Chain-of-Thought (CoT) reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). In this paper, to understand the qualities of these long CoTs and measure the critique abilities of existing LLMs on these long CoTs, we introduce the DeltaBench, including the generated long CoTs from different o1-like models (e.g., QwQ, DeepSeek-R1) for different reasoning tasks (e.g., Math, Code, General Reasoning), to measure the ability to detect errors in long CoT reasoning. Based on DeltaBench, we first perform fine-grained analysis of the generated long CoTs to discover the effectiveness and efficiency of different o1-like models. Then, we conduct extensive evaluations of existing process reward models (PRMs) and critic models to detect the errors of each annotated process, which aims to investigate the boundaries and limitations of existing PRMs and critic models. Finally, we hope that DeltaBench could guide developers to better understand the long CoT reasoning abilities of their models.

翻译：近期，o1类模型引起了广泛关注，这些模型通过生成长链思维推理步骤来提升现有大型语言模型的推理能力。本文为理解这些长链思维推理的质量并评估现有大型语言模型对其的批判能力，提出了DeltaBench基准测试集。该数据集包含不同o1类模型（如QwQ、DeepSeek-R1）针对各类推理任务（如数学、代码、通用推理）生成的长链思维推理过程，用于衡量模型在长链思维推理中检测错误的能力。基于DeltaBench，我们首先对生成的长链思维推理过程进行细粒度分析，以探究不同o1类模型的有效性与效率。随后，我们对现有过程奖励模型和批判模型进行了广泛评估，检测每个标注推理过程中的错误，旨在探索现有过程奖励模型与批判模型的边界与局限。最后，我们希望DeltaBench能够帮助开发者更好地理解其模型的长链思维推理能力。