Many of the recent capabilities demonstrated by Large Language Models (LLMs) arise primarily from their ability to exploit contextual information. In this paper, we explore ways to improve reasoning capabilities of LLMs through (1) exploration of different chains of thought and (2) validation of the individual steps of the reasoning process. We propose three general principles that a model should adhere to while reasoning: (i) Relevance, (ii) Mathematical Accuracy, and (iii) Logical Consistency. We apply these constraints to the reasoning steps generated by the LLM to improve the accuracy of the final generation. The constraints are applied in the form of verifiers: the model itself is asked to verify if the generated steps satisfy each constraint. To further steer the generations towards high-quality solutions, we use the perplexity of the reasoning steps as an additional verifier. We evaluate our method on 4 distinct types of reasoning tasks, spanning a total of 9 different datasets. Experiments show that our method is always better than vanilla generation, and, in 6 out of the 9 datasets, it is better than best-of N sampling which samples N reasoning chains and picks the lowest perplexity generation.

翻译：大型语言模型（LLMs）近期展现的许多能力主要源于其利用上下文信息的能力。本文通过以下两种方式探索提升LLMs推理能力的方法：（1）探索不同的思维链；（2）验证推理过程各步骤的合理性。我们提出模型在推理时应遵循的三个通用原则：（i）相关性、（ii）数学准确性、（iii）逻辑一致性。我们将这些约束应用于LLM生成的推理步骤，以提升最终输出的准确性。约束以验证器的形式实施：模型自身被要求验证生成的步骤是否满足每项约束。为进一步引导生成过程趋向高质量解决方案，我们利用推理步骤的困惑度作为附加验证器。我们在涵盖9个不同数据集的4类推理任务上评估了该方法。实验表明，我们的方法始终优于原始生成方法，且在9个数据集的6个中优于最优N采样（采样N条推理链并选取最低困惑度输出）方法。