Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2). To this end, we introduce a fast-slow learning framework for LLMs, with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while parameter-only RL stalls.

翻译：大语言模型通过参数更新（例如使用强化学习）在下游任务上进行训练。然而，更新参数迫使模型吸收任务特定信息，这可能导致灾难性遗忘和可塑性丧失。相比之下，固定大语言模型参数的上下文学习能够以低成本快速适应任务特定要求（例如提示优化），但通常无法单独达到参数更新带来的性能提升。没有充分理由将学习局限于上下文学习或权重更新，并且人类也可能以不同时间尺度进行学习（例如系统1与系统2）。为此，我们为大语言模型引入了一个快慢学习框架，其中模型参数作为“慢”权重，优化后的上下文作为“快”权重。这些快“权重”能够从文本反馈中学习以吸收任务特定信息，同时使慢权重更接近基础模型并保持通用推理行为。在推理任务上，快慢训练相比纯慢学习（强化学习）样本效率提升高达3倍，并且始终能达到更高的性能渐近线。此外，经过快慢训练的模型更接近基础大语言模型（KL散度降低高达70%），从而比强化学习训练产生更少的灾难性遗忘。这种漂移减少也保留了可塑性：在完成一个任务的训练后，快慢训练模型比纯参数训练模型能更有效地适应后续任务。在任务域动态变化的持续学习场景中，快慢训练持续获取每个新任务，而纯参数强化学习则停滞不前。