We study the task of prompting large-scale language models to perform multi-step reasoning. Existing work shows that when prompted with a chain of thoughts (CoT), sequences of short sentences describing intermediate reasoning steps towards a final answer, large language models can generate new reasoning chains and predict answers for new inputs. A central question is which reasoning examples make the most effective prompts. In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning. We show that prompts with higher reasoning complexity, i.e., chains with more reasoning steps, achieve substantially better performance on multi-step reasoning tasks over strong baselines. We further extend our complexity-based criteria from prompting (selecting inputs) to decoding (selecting outputs), where we sample multiple reasoning chains from the model, then choose the majority of generated answers from complex reasoning chains (over simple chains). When used to prompt GPT-3 and Codex, our approach substantially improves multi-step reasoning accuracy and achieves new state-of-the-art (SOTA) performance on three math benchmarks (GSM8K, MultiArith, and MathQA) and two BigBenchHard tasks (Date Understanding and Penguins), with an average +5.3 and up to +18 accuracy improvements. Compared with existing example selection schemes like manual tuning or retrieval-based selection, selection based on reasoning complexity is intuitive, easy to implement, and annotation-efficient. Further results demonstrate the robustness of performance gains from complex prompts under format perturbation and distribution shift.

翻译：我们研究如何通过提示大规模语言模型执行多步推理任务。现有工作表明，当使用思维链（CoT）——描述最终答案中间推理步骤的短句序列——进行提示时，大语言模型能够生成新的推理链并预测新输入的答案。核心问题在于哪些推理示例能构成最有效的提示。本文提出基于复杂性的提示策略，这是一种简单有效的多步推理示例选择方案。研究表明，具有更高推理复杂性的提示（即包含更多推理步骤的链）在多步推理任务上显著优于强基线方法。我们进一步将基于复杂性的标准从提示（输入选择）扩展至解码（输出选择）：从模型中采样多条推理链后，从复杂推理链（而非简单链）生成答案中选取多数结果。使用该方法提示GPT-3和Codex时，我们的策略显著提升多步推理准确率，并在三个数学基准（GSM8K、MultiArith和MathQA）及两个BigBenchHard任务（日期理解和企鹅分类）上实现新的最优结果（平均+5.3，最高+18准确率提升）。与现有示例选择方案（如人工调优或基于检索的选择）相比，基于推理复杂性的选择方法更直观易实现且节省标注成本。进一步实验表明，复杂提示在格式扰动和分布偏移场景下仍保持稳健的性能提升。