Large language models (LLMs) are typically trained to acquire behaviours from demonstrations or experience, yet much of their training data is declarative: instructions, rules, and descriptions that specify behaviours without showing how to execute them. We introduce Programming by Backprop (PBB): a training regime that enables LLMs to acquire procedural knowledge (i.e., reusable behaviours) from declarative instructions encountered during training. With PBB, instructions in training data provide an opportunity to `program' specific behaviours into model weights. The core principle underpinning PBB is the separation of learning how instructions map to behaviour from internalising new instructions. We devise two distinct PBB curricula that leverage this principle. Through controlled experiments across two domains (algorithmic execution from Python source code and text generation from context-free grammars), we demonstrate the benefit of these curricula over training on a homogeneous data mixture. Crucially, PBB is highly sample efficient, with a single instruction substituting for up to 100 execution examples. Though execution of instructions in training data remains less reliable than when instructions are given in-context, our results demonstrate that procedural knowledge can be noisily `programmed' into LLMs through PBB, with important implications for data curation and safety.

翻译：大语言模型通常通过演示或经验学习行为，然而其训练数据多为陈述性内容：指令、规则和描述仅规定行为而未展示执行方式。我们提出基于反向传播的编程：该训练机制使大语言模型能够从训练过程中遇到的陈述性指令中获取程序性知识（即可复用的行为模式）。通过PBB，训练数据中的指令为将特定行为“编程”到模型权重中提供了契机。PBB的核心原理在于将“学习指令如何映射到行为”与“内化新指令”这两个过程分离。基于此原理，我们设计了两种不同的PBB训练课程。通过在两个领域（Python源代码的算法执行与上下文无关文法的文本生成）的对照实验，我们证明了这些课程相较于同质数据混合训练的优势。关键的是，PBB具有极高的样本效率——单条指令最多可替代100个执行示例。尽管训练数据中指令的执行可靠性仍低于上下文给定指令的情况，但我们的结果表明程序性知识能够通过PBB以带有噪声的方式“编程”到大语言模型中，这对数据策展与安全性具有重要启示。