Multi-epoch training is becoming the standard now that compute is growing faster than the supply of high-quality text. But pretraining a single model saturates within a few passes, long before the compute budget is exhausted. We argue this calls for a conceptual shift from training a single model toward exploring a population of models and aggregating their predictions. We introduce hyper-epoch pretraining (q0), which turns a multi-epoch budget into a population of diverse models whose combined predictions reach a lower validation loss than a single refined model. q0 reduces to three core primitives. A cyclic schedule with anti-correlated learning rate and weight decay collects diverse models from a few parallel trajectories. Chain distillation trains each model against its predecessor so that model quality compounds across the population. A learned prior, fit on a held out set, selects and weights members for any inference budget. On a 1.8B-parameter model trained on 100M FineWeb tokens, q0 matches a strong 256-epoch ensemble baseline using only ${\sim}56$ epochs (${\sim}4.6\times$ fewer), or ${\sim}67$ epochs (${\sim}3.8\times$ fewer) when matched to the baseline's ensemble size, and continues to improve beyond it. These gains reach cumulative ${\sim}12.9\times$ data efficiency under the Slowrun setting and transfer to downstream benchmarks. Crucially, the optimal allocation shifts with the budget, so we give prescriptive recipes for how to spend a given epoch budget to maximize generalization, from a single epoch up to the largest budgets.

翻译：多周期训练正成为标准做法，因为算力增长快于高质量文本的供应。但单个模型的预训练在几个周期后就会饱和，远早于计算预算耗尽。我们认为，这需要从训练单个模型转向探索模型群体并聚合其预测的概念性转变。我们引入超周期预训练（q0），它将多周期预算转化为多样化的模型群体，其组合预测比单个精炼模型达到更低的验证损失。q0归结为三个核心原语。一个具有反相关学习率和权重衰减的循环调度，从多个并行轨迹中收集多样化模型。链式蒸馏使每个模型针对其前驱进行训练，从而使模型质量在群体中累积提升。一个在保留集上拟合的学习先验，为任意推理预算选择和加权成员。在1.8B参数模型上使用100M FineWeb令牌进行训练，q0仅用约56个周期（减少约4.6倍）或约67个周期（减少约3.8倍，当与基线集成大小匹配时）即可匹配强256周期集成基线，并继续超越。这些增益在Slowrun设置下达到累积约12.9倍的数据效率，并迁移到下游基准。关键在于，最优分配随预算变化，因此我们针对如何花费给定周期预算以最大化泛化给出了规范性配方，范围从单个周期到最大预算。