Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent ($ρ= .81$ across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining ($R^2 = .68$-$.84$ across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.

翻译：摘要：大型语言模型（LLMs）能执行极为复杂的任务，然而这些能力在预训练过程中如何逐步涌现的细节仍不甚明了。验证损失上的标度定律告诉我们模型随计算量增加提升了多少，却未揭示它按何种顺序习得何种技能。为弥补这一不足，我们提出"隐式课程假说"：预训练遵循一个跨模型与数据混合的、可组合且可预测的课程。我们通过设计一组涵盖检索、形态变换、共指解析、逻辑推理与数学的简单可组合任务来检验该假说。利用这些任务，我们追踪了四个模型族（参数规模从4.1亿到130亿）的涌现时间点。研究发现，模型达到固定精度阈值的涌现顺序高度一致（跨45个模型对时ρ=0.81），且复合任务大多在其组成任务之后涌现。此外，这种结构编码于模型表征中：具有相似函数向量表征的任务在训练过程中也往往遵循类似的轨迹。通过利用从任务集导出的表征空间，我们无需事先评估即可有效预测简单保留组合任务在预训练全程中的训练轨迹（跨模型R²=0.68–0.84）。综合这些结果，表明预训练比损失曲线所揭示的更具结构性：技能以跨模型一致且可从模型内部读取的组合顺序涌现。