In this paper, we explore the nature of sudden breakthroughs in language model performance at scale, which stands in contrast to smooth improvements governed by scaling laws. While advocates of "emergence" view abrupt performance gains as capabilities unlocking at specific scales, others have suggested that they are produced by thresholding effects and alleviated by continuous metrics. We propose that breakthroughs are instead driven by continuous changes in the probability distribution of training outcomes, particularly when performance is bimodally distributed across random seeds. In synthetic length generalization tasks, we show that different random seeds can produce either highly linear or emergent scaling trends. We reveal that sharp breakthroughs in metrics are produced by underlying continuous changes in their distribution across seeds. Furthermore, we provide a case study of inverse scaling and show that even as the probability of a successful run declines, the average performance of a successful run continues to increase monotonically. We validate our distributional scaling framework on realistic settings by measuring MMLU performance in LLM populations. These insights emphasize the role of random variation in the effect of scale on LLM capabilities.

翻译：本文探讨了语言模型性能在规模化过程中出现突破性进展的本质，这种现象与受缩放定律支配的平滑改进形成鲜明。虽然“涌现”观点的支持者将性能的突然提升视为特定规模下能力的解锁，但另一些研究者则认为这是由阈值效应产生，并可通过连续度量指标来缓解。我们提出，突破性进展实际上是由训练结果概率分布的连续变化所驱动，尤其是在性能在不同随机种子间呈双峰分布的情况下。在合成长度泛化任务中，我们展示了不同的随机种子可以产生高度线性或涌现性的缩放趋势。我们发现，度量指标的急剧突破是由其在种子间分布的潜在连续变化所产生。此外，我们提供了一个逆向缩放的案例研究，表明即使成功运行的概率下降，成功运行的平均性能仍持续单调递增。我们通过在LLM群体中测量MMLU性能，在现实场景中验证了我们的分布性缩放框架。这些见解强调了随机变异在规模对LLM能力影响中的作用。