Large language models (LLMs) have achieved remarkable success in a wide range of tasks. However, their reasoning capabilities, particularly in complex domains like mathematics, remain a significant challenge. Value-based process verifiers, which estimate the probability of a partial reasoning chain leading to a correct solution, are a promising approach for improving reasoning. Nevertheless, their effectiveness is often hindered by estimation error in their training annotations, a consequence of the limited number of Monte Carlo (MC) samples feasible due to the high cost of LLM inference. In this paper, we identify that the estimation error primarily arises from high variance rather than bias, and the MC estimator is a Minimum Variance Unbiased Estimator (MVUE). To address the problem, we propose the \textsc{Com}pound \textsc{M}onte \textsc{C}arlo \textsc{S}ampling (ComMCS) method, which constructs an unbiased estimator by linearly combining the MC estimators from the current and subsequent steps. Theoretically, we show that our method leads to a predictable reduction in variance, while maintaining an unbiased estimation without additional LLM inference cost. We also perform empirical experiments on the MATH-500 and GSM8K benchmarks to demonstrate the effectiveness of our method. Notably, ComMCS outperforms regression-based optimization method by 2.8 points, the non-variance-reduced baseline by 2.2 points on MATH-500 on Best-of-32 sampling experiment.

翻译：大型语言模型（LLM）在广泛的任务中取得了显著成功。然而，其推理能力，特别是在数学等复杂领域，仍然是一个重大挑战。基于价值的过程验证器通过评估部分推理链导向正确解的概率，是改进推理的一种有前景的方法。然而，其有效性常常受到训练标注中估计误差的阻碍，这是由于LLM推理成本高昂导致可行的蒙特卡洛（MC）样本数量有限所致。本文指出，估计误差主要源于高方差而非偏差，且MC估计量是最小方差无偏估计量（MVUE）。为解决此问题，我们提出了复合蒙特卡洛采样（ComMCS）方法，该方法通过线性组合当前及后续步骤的MC估计量来构建一个无偏估计量。理论上，我们证明了该方法能在不引入额外LLM推理成本、保持估计无偏的同时，实现可预测的方差降低。我们还在MATH-500和GSM8K基准上进行了实证实验，以验证我们方法的有效性。值得注意的是，在Best-of-32采样实验中，ComMCS在MATH-500上分别优于基于回归的优化方法2.8个百分点，以及未进行方差削减的基线方法2.2个百分点。