Large text corpora are the backbone of language models. However, we have a limited understanding of the content of these corpora, including general statistics, quality, social factors, and inclusion of evaluation data (contamination). In this work, we propose What's In My Big Data? (WIMBD), a platform and a set of sixteen analyses that allow us to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities -- count and search -- at scale, which allows us to analyze more than 35 terabytes on a standard compute node. We apply WIMBD to ten different corpora used to train popular language models, including C4, The Pile, and RedPajama. Our analysis uncovers several surprising and previously undocumented findings about these corpora, including the high prevalence of duplicate, synthetic, and low-quality content, personally identifiable information, toxic language, and benchmark contamination. For instance, we find that about 50% of the documents in RedPajama and LAION-2B-en are duplicates. In addition, several datasets used for benchmarking models trained on such corpora are contaminated with respect to important benchmarks, including the Winograd Schema Challenge and parts of GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them: github.com/allenai/wimbd.

翻译：大型文本语料库是语言模型的基石。然而，我们对这些语料库的内容（包括一般统计数据、质量、社会因素以及评估数据的污染情况）的理解十分有限。在这项工作中，我们提出了“我的大数据里有什么？”（WIMBD）——一个平台和一套共十六种分析方法，用于揭示和比较大型文本语料库的内容。WIMBD 建立在两种核心能力之上——大规模计数和搜索，这使得我们能够在标准计算节点上分析超过 35 TB 的数据。我们将 WIMBD 应用于用于训练流行的语言模型的十个不同语料库，包括 C4、The Pile 和 RedPajama。我们的分析揭示了关于这些语料库的一些令人惊讶且此前未有文献记录的发现，包括重复内容、合成内容和低质量内容的高发性，以及个人身份信息、有害语言和基准测试污染。例如，我们发现 RedPajama 和 LAION-2B-en 中约 50% 的文档是重复的。此外，用于对此类语料库上训练的模型进行基准测试的多个数据集在重要基准测试（包括 Winograd 模式挑战赛以及 GLUE 和 SuperGLUE 的组成部分）方面存在污染。我们开源了 WIMBD 的代码和产物，以便为新的基于文本的语料库提供一套标准评估，并鼓励围绕它们进行更多分析和提高透明度：github.com/allenai/wimbd。