Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors? In this paper, we analyze the intermediate training checkpoints of differently sized OPT models (Zhang et al.,2022)--from 125M to 175B parameters--on next-token prediction, sequence-level generation, and downstream tasks. We find that 1) at a given perplexity and independent of model sizes, a similar subset of training tokens see the most significant reduction in loss, with the rest stagnating or showing double-descent behavior; 2) early in training, all models learn to reduce the perplexity of grammatical sequences that contain hallucinations, with small models halting at this suboptimal distribution and larger ones eventually learning to assign these sequences lower probabilities; 3) perplexity is a strong predictor of in-context learning performance on 74 multiple-choice tasks from BIG-Bench, and this holds independent of the model size. Together, these results show that perplexity is more predictive of model behaviors than model size or training computation.

翻译：扩大语言模型规模带来了前所未有的性能提升，但关于训练动态如何随模型增大而变化，人们仍知之甚少。不同规模的语言模型在预训练过程中是如何学习的？为什么更大规模的语言模型会展现出更理想的行为？本文分析了不同规模OPT模型（Zhang等人，2022）的中期训练检查点——参数从1.25亿到1750亿不等——并考察了它们在下一个词元预测、序列级生成以及下游任务上的表现。我们发现：1）在给定的困惑度下，与模型规模无关，训练数据中相似的子集在损失上经历了最显著的降低，而其余部分则停滞不前或表现出双下降行为；2）在训练早期，所有模型都学会降低包含幻觉的语法序列的困惑度，小模型停留在这个次优分布上，而大模型最终学会赋予这些序列更低概率；3）困惑度是BIG-Bench中74个多项选择任务上上下文学习性能的强预测指标，且这一关系与模型规模无关。这些结果共同表明，困惑度比模型规模或训练计算量更能预测模型行为。