Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.

翻译：针对新任务训练的模型通常会在先前任务上表现下降，这一现象被称为遗忘。传统上，缓解遗忘需要回放先前任务中存储的样例，这往往不切实际。相比之下，语言模型能够从其自身的训练分布中采样，而我们发现这些自我生成的样本可作为有效的回放数据，几乎消除遗忘。然而，我们注意到当模型剩余容量不足时，遗忘仍会出现：接近饱和训练的预训练模型无法在不覆写先前知识的情况下吸收新信息。当容量并非限制因素时，低学习率可减少遗忘但需要显著更多的训练步数。回放策略打破了这一权衡，使得在不发生遗忘的情况下，能够进行快速、高学习率的微调。