Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English$\rightarrow$English) and a stronger distribution shift (English$\rightarrow$German) at the $405$M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realistic shift for larger-scale experiments, we also find that our continual learning strategies match the re-training baseline for a 10B parameter LLM. Our results demonstrate that LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training baseline using only a fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR re-warming and that are not bound to a fixed token budget.

翻译：大型语言模型通常在数十亿个令牌上进行预训练，一旦新数据可用就得重新开始整个训练过程。一种更高效的解决方案是持续预训练这些模型，相比重新训练可节省大量计算资源。然而，新数据带来的分布偏移通常会导致模型在旧数据上的性能下降或对新数据的适应能力不足。本研究证明，通过结合学习率重新预热、学习率重新衰减以及旧数据重放这一简单且可扩展的策略，就能在最终损失和多个语言模型评估基准平均得分上，达到使用全部可用数据从头重新训练的同等效果。我们针对两种常用大语言模型预训练数据集之间的弱分布偏移（英语→英语）和强分布偏移（英语→德语），在4.05亿参数模型规模和大型数据集（数千亿个令牌）上验证了这一结论。在更大规模实验中选取弱但实际的分布偏移时，我们发现该持续学习策略能使100亿参数的大语言模型达到与重新训练基线相当的效果。研究结果表明，通过简单可扩展的持续学习策略即可成功更新大语言模型，且仅需使用重新训练基线的部分计算资源。最后，受前人研究启发，我们提出了余弦学习率计划表的替代方案，这些方案既能帮助规避学习率重新预热导致的遗忘问题，又不受固定令牌预算的限制。