Generating accurate step-by-step reasoning is essential for Large Language Models (LLMs) to address complex problems and enhance robustness and interpretability. Despite the flux of research on developing advanced reasoning approaches, systematically analyzing the diverse LLMs and reasoning strategies in generating reasoning chains remains a significant challenge. The difficulties stem from the lack of two key elements: (1) an automatic method for evaluating the generated reasoning chains on different tasks, and (2) a unified formalism and implementation of the diverse reasoning approaches for systematic comparison. This paper aims to close the gap: (1) We introduce AutoRace for fully automated reasoning chain evaluation. Existing metrics rely on expensive human annotations or pre-defined LLM prompts not adaptable to different tasks. In contrast, AutoRace automatically creates detailed evaluation criteria tailored for each task, and uses GPT-4 for accurate evaluation following the criteria. (2) We develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms, under a unified formulation of the search, reward, and world model components. With the new evaluation and library, (3) we conduct extensive study of different reasoning approaches (e.g., CoT, ToT, RAP). The analysis reveals interesting findings about different factors contributing to reasoning, including the reward-guidance, breadth-vs-depth in search, world model, and prompt formats, etc.

翻译：生成准确的逐步推理对于大语言模型处理复杂问题、增强鲁棒性与可解释性至关重要。尽管针对开发先进推理方法的研究层出不穷，但系统分析不同大语言模型及推理策略在生成推理链方面的能力仍面临重大挑战。这一困境源于两个关键要素的缺失：（1）跨任务自动评估生成推理链的方法；（2）对不同推理方法进行系统比较的统一形式化框架与实现。本文旨在弥合这一差距：（1）我们提出AutoRace——全自动推理链评估方法。现有指标依赖昂贵的人工标注或预定义的大语言模型提示，难以适应不同任务。而AutoRace可为每项任务自动生成细粒度评估准则，并利用GPT-4依据准则进行精准评估。（2）我们构建LLM Reasoners工具库，基于搜索、奖励与世界模型组件的统一形式化框架，实现现有及新推理算法的标准化模块化开发。借助新评估方法与工具库，（3）我们开展了涵盖CoT、ToT、RAP等多种推理方法的系统性研究。分析揭示了影响推理质量的多重因素，包括奖励导向机制、搜索中的广度与深度权衡、世界模型构建以及提示格式等。