Large language models have astounded the world with fascinating new capabilities. However, they currently lack the ability to teach themselves new skills, relying instead on large amounts of human-generated training data. We introduce SECToR (Self-Education via Chain-of-Thought Reasoning), a proof-of-concept demonstration that language models can teach themselves new skills using chain-of-thought reasoning. During the self-learning loop, SECToR asks models to solve addition problems using chain-of-thought reasoning before training the next version of the model to solve those same problems directly without using such reasoning. This process often results in an improved model which can, when again augmented with chain-of-thought reasoning, solve even harder problems than the original model, allowing the self-learning loop to continue. Language models trained via SECToR autonomously learn to add up to the longest-length-digit numbers without access to any ground truth examples beyond an initial supervised fine-tuning phase consisting only of numbers with 6 or fewer digits. Our central hypothesis is that chain-of-thought reasoning can act as a policy improvement operator, similarly to how Monte-Carlo Tree Search is used in AlphaZero (Silver et al., 2017). We hope that this research can lead to new directions in which language models can learn to teach themselves without the need for human demonstrations.

翻译：大语言模型以其令人惊叹的新功能震惊了世界。然而，它们目前缺乏自我学习新技能的能力，反而依赖大量人工生成的训练数据。我们提出SECToR（基于思维链推理的自我教育），这是一个概念验证，证明语言模型可以通过思维链推理自我学习新技能。在自我学习循环中，SECToR要求模型使用思维链推理解决加法问题，随后训练下一版本模型直接解决相同问题而不使用此类推理。该过程通常会得到一个改进后的模型，当再次辅以思维链推理时，该模型能够解决比原始模型更困难的问题，从而使自我学习循环得以持续。通过SECToR训练的语言模型自主学会了计算最长位数数字的加法，除初始监督微调阶段仅包含6位或更少数字外，无需访问任何真实标注示例。我们的核心假设是：思维链推理可充当策略改进算子，类似于AlphaZero（Silver等人，2017年）中蒙特卡洛树搜索的作用。我们希望这项研究能引导语言模型在无需人类示范的情况下自我学习的新方向。