The field of natural language processing (NLP) has witnessed significant progress in recent years, with a notable focus on improving large language models' (LLM) performance through innovative prompting techniques. Among these, prompt engineering coupled with structures has emerged as a promising paradigm, with designs such as Chain-of-Thought, Tree of Thoughts, or Graph of Thoughts, in which the overall LLM reasoning is guided by a structure such as a graph. As illustrated with numerous examples, this paradigm significantly enhances the LLM's capability to solve numerous tasks, ranging from logical or mathematical reasoning to planning or creative writing. To facilitate the understanding of this growing field and pave the way for future developments, we devise a general blueprint for effective and efficient LLM reasoning schemes. For this, we conduct an in-depth analysis of the prompt execution pipeline, clarifying and clearly defining different concepts. We then build the first taxonomy of structure-enhanced LLM reasoning schemes. We focus on identifying fundamental classes of harnessed structures, and we analyze the representations of these structures, algorithms executed with these structures, and many others. We refer to these structures as reasoning topologies, because their representation becomes to a degree spatial, as they are contained within the LLM context. Our study compares existing prompting schemes using the proposed taxonomy, discussing how certain design choices lead to different patterns in performance and cost. We also outline theoretical underpinnings, relationships between prompting and others parts of the LLM ecosystem such as knowledge bases, and the associated research challenges. Our work will help to advance future prompt engineering techniques.

翻译：自然语言处理领域近年来取得了显著进展，其中通过创新提示技术提升大型语言模型性能成为关注焦点。在诸多方法中，基于结构化提示的工程范式崭露头角，例如思维链、思维树或思维图等设计，其核心是通过图等结构引导LLM的完整推理过程。大量实例表明，该范式显著增强了LLM在逻辑推理、数学运算、规划编排乃至创意写作等多元任务中的求解能力。为促进该新兴领域的理解并为未来发展铺平道路，我们构建了一套适用于高效LLM推理方案的通用蓝图。为此，我们深入剖析了提示执行流程，厘清并明确定义了相关概念，进而首次建立了结构化增强型LLM推理方案分类体系。研究聚焦于识别所采用结构的基本类别，分析这些结构的表征方式、基于此类结构的算法执行机制及其他关键要素。我们将这些结构称为推理拓扑，因其表征在LLM上下文环境中呈现出一定程度的空间特性。本研究运用所提出的分类体系对现有提示方案进行对比，探讨特定设计选择如何导致性能与成本的不同模式。同时，我们阐述了理论基础、提示与知识库等LLM生态系统其他组成部分的关联，以及相关研究挑战。本工作将推动未来提示工程技术的发展。