Recent works like Tree-of-Thought (ToT) and Reasoning via Planning (RAP) aim to augment the reasoning capabilities of LLMs by using tree-search algorithms to guide multi-step reasoning. These methods rely on prompting a pre-trained model to serve as a value function and focus on problems with low search depth. As a result, these methods will not work in domains where the pre-trained LLM does not have enough knowledge to serve as an effective value function or in domains that require long-horizon planning. To address these limitations, we present an AlphaZero-like tree-search learning framework for LLMs (termed TS-LLM), systematically illustrating how tree-search with a learned value function can guide LLM decoding. TS-LLM distinguishes itself in two key ways. (1) Leveraging a learned value function and AlphaZero-like algorithms, our approach can be generally adaptable to a wide range of tasks, language models of any size, and tasks of varying search depths. (2) Our approach can guide LLMs during both inference and training, iteratively improving the LLM. Empirical results across reasoning, planning, alignment, and decision-making tasks show that TS-LLM outperforms existing approaches and can handle trees with a depth of 64.

翻译：近期如思维树（ToT）和基于规划的推理（RAP）等工作，旨在通过树搜索算法指导多步推理，增强大语言模型的推理能力。这些方法依赖于对预训练模型进行提示以充当价值函数，且主要针对搜索深度较低的问题。因此，在预训练大语言模型缺乏足够知识充当有效价值函数的领域，或需要长时域规划的领域，这些方法将无法适用。为解决上述局限，我们提出一种面向大语言模型的类AlphaZero树搜索学习框架（TS-LLM），系统阐述了如何利用带学习价值函数的树搜索指导大语言模型解码。TS-LLM在以下两方面具有独特优势：（1）通过利用学习所得的价值函数与类AlphaZero算法，我们的方法可广泛适用于各类任务、任意规模的语言模型以及不同搜索深度的任务；（2）我们的方法可在推理与训练阶段同时指导大语言模型，实现模型的迭代优化。在推理、规划、对齐及决策制定等任务上的实验结果表明，TS-LLM不仅优于现有方法，且能有效处理深度达64的搜索树。