Enhancing the mathematical reasoning capabilities of Large Language Models (LLMs) is of great scientific and practical significance. Researchers typically employ process-supervised reward models (PRMs) to guide the reasoning process, effectively improving the models' reasoning abilities. However, existing methods for constructing process supervision training data, such as manual annotation and per-step Monte Carlo estimation, are often costly or suffer from poor quality. To address these challenges, this paper introduces a framework called EpicPRM, which annotates each intermediate reasoning step based on its quantified contribution and uses an adaptive binary search algorithm to enhance both annotation precision and efficiency. Using this approach, we efficiently construct a high-quality process supervision training dataset named Epic50k, consisting of 50k annotated intermediate steps. Compared to other publicly available datasets, the PRM trained on Epic50k demonstrates significantly superior performance. Getting Epic50k at https://github.com/xiaolizh1/EpicPRM.

翻译：提升大型语言模型（LLM）的数学推理能力具有重要的科学意义和实用价值。研究者通常采用过程监督奖励模型（PRM）来引导推理过程，从而有效提升模型的推理能力。然而，现有的过程监督训练数据构建方法，例如人工标注和逐步蒙特卡洛估计，往往成本高昂或质量欠佳。为应对这些挑战，本文提出了一个名为EpicPRM的框架，该框架基于每个中间推理步骤的量化贡献对其进行标注，并采用自适应二分搜索算法来提升标注的精确性和效率。利用此方法，我们高效地构建了一个名为Epic50k的高质量过程监督训练数据集，其中包含5万个已标注的中间步骤。与其他公开可用的数据集相比，基于Epic50k训练的PRM展现出显著优越的性能。可通过 https://github.com/xiaolizh1/EpicPRM 获取Epic50k。