Understanding what constitutes high-quality pre-training data remains a central question in language model training. In this work, we investigate whether benchmark performance is primarily driven by the degree of statistical pattern overlap between pre-training corpora and evaluation datasets. We measure this overlap using word-level unigram cross-entropy and word frequency statistics, and perform controlled experiments across $10$ zero-shot benchmarks, $4$ pre-training datasets spanning $8.5\mathrm{B}$ to $60\mathrm{B}$ tokens, and model sizes ranging from $400\mathrm{M}$ to $3\mathrm{B}$ parameters. Our results demonstrate a robust inverse relationship between word-level unigram cross-entropy and benchmark performance, suggesting that widely used benchmarks are strongly influenced by word overlap between training and evaluation data. Thus, larger pre-training subsets with similar word-level unigram cross-entropy yield improved downstream results, indicating that word frequency statistics play an additional role in shaping benchmark scores. Taken together, these results suggest that many standard benchmarks are only weakly out-of-distribution relative to pre-training corpora, so that simple word-overlap statistics predict benchmark performance.

翻译：理解什么构成高质量预训练数据仍然是语言模型训练中的一个核心问题。在本研究中，我们探究基准测试性能是否主要由预训练语料库与评估数据集之间的统计模式重叠程度驱动。我们使用词级一元交叉熵和词频统计量来衡量这种重叠，并在10个零样本基准测试、4个涵盖8.5B至60B词元的预训练数据集以及参数量从400M到3B的模型规模上进行了受控实验。我们的结果表明，词级一元交叉熵与基准测试性能之间存在稳健的负相关关系，这表明广泛使用的基准测试受到训练数据与评估数据之间词汇重叠的强烈影响。因此，具有相似词级一元交叉熵的更大规模预训练子集会带来下游结果的提升，表明词频统计在塑造基准测试分数方面发挥着额外作用。综上所述，这些结果表明，许多标准基准测试相对于预训练语料库仅呈现弱分布外特性，以至于简单的词汇重叠统计量即可预测基准测试性能。