Data teams at frontier AI companies routinely train small proxy models to make critical decisions about pretraining data recipes for full-scale training runs. However, the community has a limited understanding of whether and when conclusions drawn from small-scale experiments reliably transfer to full-scale model training. In this work, we uncover a subtle yet critical issue in the standard experimental protocol for data recipe assessment: the use of identical small-scale model training configurations across all data recipes in the name of "fair" comparison. We show that the experiment conclusions about data quality can flip with even minor adjustments to training hyperparameters, as the optimal training configuration is inherently data-dependent. Moreover, this fixed-configuration protocol diverges from full-scale model development pipelines, where hyperparameter optimization is a standard step. Consequently, we posit that the objective of data recipe assessment should be to identify the recipe that yields the best performance under data-specific tuning. To mitigate the high cost of hyperparameter tuning, we introduce a simple patch to the evaluation protocol: using reduced learning rates for proxy model training. We show that this approach yields relative performance that strongly correlates with that of fully tuned large-scale LLM pretraining runs. Theoretically, we prove that for random-feature models, this approach preserves the ordering of datasets according to their optimal achievable loss. Empirically, we validate this approach across 23 data recipes covering four critical dimensions of data curation, demonstrating dramatic improvements in the reliability of small-scale experiments.

翻译：前沿AI公司的数据团队通常训练小型代理模型，以对全规模训练运行中的预训练数据配方做出关键决策。然而，社区对于从小规模实验得出的结论是否以及何时能可靠迁移至全规模模型训练的理解仍然有限。本研究发现，在数据配方评估的标准实验协议中存在一个微妙但关键的问题：为追求“公平”比较，所有数据配方均采用相同的小规模模型训练配置。我们证明，即使对训练超参数进行微小调整，关于数据质量的实验结论也可能发生逆转，因为最优训练配置本质上与数据相关。此外，这种固定配置协议与全规模模型开发流程存在差异，后者通常包含超参数优化这一标准步骤。因此，我们认为数据配方评估的目标应是识别在数据特异性调优下能获得最佳性能的配方。为降低超参数调优的高昂成本，我们引入一种简单的评估协议修正方案：在代理模型训练中使用降低的学习率。实验表明，该方法获得的相对性能与完全调优的大规模LLM预训练运行结果呈现强相关性。理论上，我们证明对于随机特征模型，该方法能保持数据集按其最优可达损失的排序关系。实证方面，我们在涵盖数据筛选四个关键维度的23种数据配方上验证了该方法，证明其能显著提升小规模实验的可靠性。