To comprehensively gauge the capacity of current models for complex reasoning, it is crucial to assess their step-by-step reasoning in a scalable manner. Established reference-based evaluation metrics rely on human-annotated reasoning chains as references to assess the model-derived chains. However, such "gold-standard" human-written reasoning chains may not be unique and their acquisition is often labor-intensive. Existing reference-free reasoning evaluation metrics, while eliminating the need for human-crafted reasoning chains as references, often require fine-tuning with human-derived chains before evaluation, complicating the process and questioning their adaptability to other datasets. To address these challenges, we harness GPT-4 to automatically evaluate reasoning chain quality, thereby removing the dependency on human-written reasoning chains for both model fine-tuning and evaluative purposes. Leveraging the Socratic method, we develop SocREval ({\bf Soc}ratic Method-Inspired {\bf R}easoning {\bf Eval}uation), a novel approach for prompt design in reference-free reasoning evaluation. Empirical results from four human annotated datasets reveal that SocREval significantly improves GPT-4's performance, surpassing existing reference-free and reference-based reasoning evaluation metrics. Beyond its demonstrated efficacy, SocREval, proves to be both cost-efficient and robust to prompt writing and example selection, as substantiated by our in-depth analysis.

翻译：为全面衡量当前模型复杂推理能力，以可扩展方式评估其逐步推理过程至关重要。传统的基于参考的评估指标依赖人工标注的推理链作为参照来评估模型生成的推理链，然而这类"黄金标准"人工推理链并非唯一且获取往往耗费人力。现有无参考推理评估指标虽消除了对人工推理链的依赖，却通常需要在评估前使用人工推理链进行微调，这不仅增加了流程复杂性，也使其对其他数据集的适应性存疑。针对这些挑战，我们利用GPT-4自动评估推理链质量，从而完全消除对人工推理链的依赖——无论是模型微调还是评估环节。受苏格拉底法启发，我们提出SocREval（苏格拉底式推理评估）这一新颖的无参考推理评估提示设计方法。在四个人工标注数据集上的实验结果表明，SocREval显著提升了GPT-4的推理评估性能，超越了现有无参考与基于参考的推理评估指标。除有效性验证外，深度分析还证实SocREval具有成本效益高、对提示设计和示例选择具有鲁棒性的优势。