Chain-of-thought (CoT) prompting, which offers step-by-step problem-solving rationales, has impressively unlocked the reasoning potential of large language models (LLMs). Yet, the standard CoT is less effective in problems demanding multiple reasoning steps. This limitation arises from the complex reasoning process in multi-step problems: later stages often depend on the results of several steps earlier, not just the results of the immediately preceding step. Such complexities suggest the reasoning process is naturally represented as a graph. The almost linear and straightforward structure of CoT prompting, however, struggles to capture this complex reasoning graph. To address this challenge, we propose Residual Connection Prompting (RESPROMPT), a new prompting strategy that advances multi-step reasoning in LLMs. Our key idea is to reconstruct the reasoning graph within prompts. We achieve this by integrating necessary connections-links present in the reasoning graph but missing in the linear CoT flow-into the prompts. Termed "residual connections", these links are pivotal in morphing the linear CoT structure into a graph representation, effectively capturing the complex reasoning graphs inherent in multi-step problems. We evaluate RESPROMPT on six benchmarks across three diverse domains: math, sequential, and commonsense reasoning. For the open-sourced LLaMA family of models, RESPROMPT yields a significant average reasoning accuracy improvement of 12.5% on LLaMA-65B and 6.8% on LLaMA2-70B. Breakdown analysis further highlights RESPROMPT particularly excels in complex multi-step reasoning: for questions demanding at least five reasoning steps, RESPROMPT outperforms the best CoT based benchmarks by a remarkable average improvement of 21.1% on LLaMA-65B and 14.3% on LLaMA2-70B. Through extensive ablation studies and analyses, we pinpoint how to most effectively build residual connections.

翻译：链式思维提示（CoT）通过提供逐步的问题解决推理过程，显著释放了大型语言模型（LLMs）的推理潜力。然而，标准CoT在需要多步推理的问题中效果较差。这一局限性源于多步问题中复杂的推理过程：后续步骤通常依赖于前多步的结果，而不仅仅是紧邻上一步的结果。这种复杂性表明推理过程天然呈现为图结构。而CoT提示近乎线性的简洁结构难以捕捉这种复杂的推理图。为解决这一挑战，我们提出残差连接提示（RESPROMPT）——一种新的提示策略，可推进LLMs的多步推理能力。核心思想是在提示中重构推理图。我们通过将推理图中存在但线性CoT流程中缺失的必要连接链（即“残差连接”）整合到提示中来实现这一目标。这些连接将线性CoT结构转变为图表示，有效捕捉多步问题中固有的复杂推理图。我们在数学、序列和常识推理三个领域的六个基准上评估了RESPROMPT。对于开源的LLaMA系列模型，RESPROMPT在LLaMA-65B上实现了12.5%的平均推理准确率提升，在LLaMA2-70B上提升6.8%。进一步分解分析表明，RESPROMPT在复杂多步推理中表现尤为突出：对于需要至少五个推理步骤的问题，RESPROMPT在LLaMA-65B上比最优CoT基准平均提升21.1%，在LLaMA2-70B上提升14.3%。通过广泛的消融研究和分析，我们明确了最有效的残差连接构建方法。