Large language models (LLMs) have demonstrated impressive capability in reasoning and planning when integrated with tree-search-based prompting methods. However, since these methods ignore the previous search experiences, they often make the same mistakes in the search process. To address this issue, we introduce Reflection on search Trees (RoT), an LLM reflection framework designed to improve the performance of tree-search-based prompting methods. It uses a strong LLM to summarize guidelines from previous tree search experiences to enhance the ability of a weak LLM. The guidelines are instructions about solving this task through tree search which can prevent the weak LLMs from making similar mistakes in the past search process. In addition, we proposed a novel state selection method, which identifies the critical information from historical search processes to help RoT generate more specific and meaningful guidelines. In our extensive experiments, we find that RoT significantly improves the performance of LLMs in reasoning or planning tasks with various tree-search-based prompting methods (e.g., BFS and MCTS). Non-tree-search-based prompting methods such as Chain-of-Thought (CoT) can also benefit from RoT guidelines since RoT can provide task-specific knowledge collected from the search experience.

翻译：大语言模型（LLMs）在与基于树搜索的提示方法结合时，在推理与规划方面展现出令人瞩目的能力。然而，由于这些方法忽略了先前的搜索经验，它们在搜索过程中常会重复犯相同的错误。为解决这一问题，我们提出了搜索树反思（RoT），这是一种旨在提升基于树搜索提示方法性能的大语言模型反思框架。它利用一个强LLM总结先前树搜索经验中的指导原则，以增强弱LLM的能力。这些指导原则是关于如何通过树搜索解决该任务的指令，可防止弱LLM在历史搜索过程中重蹈覆辙。此外，我们提出了一种新颖的状态选择方法，能从历史搜索过程中识别关键信息，帮助RoT生成更具针对性和意义的指导原则。在大量实验中，我们发现RoT能显著提升LLM在各种基于树搜索提示方法（如BFS和MCTS）上的推理或规划任务性能。诸如思维链（CoT）等非树搜索提示方法也能从RoT指导原则中受益，因为RoT能提供从搜索经验中收集的任务特定知识。