Large Language Models (LLMs) have demonstrated significant potential in handling complex reasoning tasks through step-by-step rationale generation. However, recent studies have raised concerns regarding the hallucination and flaws in their reasoning process. Substantial efforts are being made to improve the reliability and faithfulness of the generated rationales. Some approaches model reasoning as planning, while others focus on annotating for process supervision. Nevertheless, the planning-based search process often results in high latency due to the frequent assessment of intermediate reasoning states and the extensive exploration space. Additionally, supervising the reasoning process with human annotation is costly and challenging to scale for LLM training. To address these issues, in this paper, we propose a framework to learn planning-based reasoning through direct preference optimization (DPO) on collected trajectories, which are ranked according to synthesized process rewards. Our results on challenging logical reasoning benchmarks demonstrate the effectiveness of our learning framework, showing that our 7B model can surpass the strong counterparts like GPT-3.5-Turbo.

翻译：大型语言模型通过逐步推理生成在处理复杂推理任务中展现出显著潜力。然而，近期研究对其推理过程中的幻觉与缺陷提出了担忧。目前已有大量研究致力于提升推理生成的可靠性与忠实度：部分方法将推理建模为规划过程，另一些则聚焦于过程监督的标注工作。然而，基于规划的搜索过程因需要频繁评估中间推理状态且探索空间庞大，常导致高延迟问题。此外，通过人工标注进行推理过程监督既成本高昂又难以扩展至大规模模型训练。为应对这些挑战，本文提出一种通过直接偏好优化在收集轨迹上学习规划推理的框架——该框架依据合成过程奖励对轨迹进行排序。我们在具有挑战性的逻辑推理基准测试上的结果表明，该学习框架具有有效性，且使用7B参数规模的模型即可超越GPT-3.5-Turbo等强基模型。