Recent works have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks, step-by-step. However, prompt-based CoT methods are dependent on very large models such as GPT-3 175B which are prohibitive to deploy at scale. In this paper, we use these large models as reasoning teachers to enable complex reasoning in smaller models and reduce model size requirements by several orders of magnitude. We propose Fine-tune-CoT, a method that generates reasoning samples from very large teacher models to fine-tune smaller models. We evaluate our method on a wide range of public models and complex tasks. We find that Fine-tune-CoT enables substantial reasoning capability in small models, far outperforming prompt-based baselines and even the teacher model in many tasks. Additionally, we extend our method by leveraging the teacher model's ability to generate multiple distinct rationales for each original sample. Enriching the fine-tuning data with such diverse reasoning results in a substantial performance boost across datasets, even for very small models. We conduct ablations and sample studies to understand the emergence of reasoning capabilities of student models. Our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher.

翻译：近期研究表明，思维链提示能引导语言模型逐步解决复杂推理任务。然而，基于提示的思维链方法依赖GPT-3 175B等超大规模模型，这类模型难以大规模部署。本文利用这些大型模型作为推理导师，使小模型具备复杂推理能力，从而将模型规模需求降低数个数量级。我们提出微调思维链方法——通过从超大规模教师模型生成推理样本，对较小模型进行微调。我们在多种公开模型和复杂任务上评估该方法，发现微调思维链能够在小型模型中显著激发推理能力，其表现远超基于提示的基线方法，甚至在多项任务中超越教师模型。此外，我们通过利用教师模型为每个原始样本生成多组不同推理路径的能力扩展了该方法。用此类多样化推理结果丰富微调数据后，即便极小模型也能在各类数据集中获得显著性能提升。我们通过消融实验和样本研究探究学生模型推理能力的涌现机制。代码实现及数据已开源在https://github.com/itsnamgyu/reasoning-teacher。