The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

翻译：当前语言模型扩展的趋势涉及同时增加参数量和训练数据集规模。外推这一趋势表明，训练数据集规模可能很快会受到互联网可用文本数据量的限制。受此限制启发，我们研究了数据受限场景下语言模型的扩展。具体而言，我们开展了一系列实验，改变数据重复程度和计算预算，实验范围涵盖高达9000亿训练令牌和90亿参数的模型。我们发现，在固定计算预算的数据受限条件下，与使用唯一数据相比，对重复数据进行最多4个epoch的训练对损失的影响可忽略不计。然而，随着重复次数增加，增加计算的价值最终会衰减至零。我们提出并实证验证了一种考虑重复令牌和多余参数价值递减的计算最优性扩展定律。最后，我们实验了缓解数据稀缺的方法，包括用代码数据增强训练数据集或移除常用过滤器。我们400次训练运行的模型和数据集可在https://github.com/huggingface/datablations 免费获取。