This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs. The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Datasets; (3) Preference Datasets; (4) Evaluation Datasets; (5) Traditional Natural Language Processing (NLP) Datasets. The survey sheds light on the prevailing challenges and points out potential avenues for future investigation. Additionally, a comprehensive review of the existing available dataset resources is also provided, including statistics from 444 datasets, covering 8 language categories and spanning 32 domains. Information from 20 dimensions is incorporated into the dataset statistics. The total data size surveyed surpasses 774.5 TB for pre-training corpora and 700M instances for other datasets. We aim to present the entire landscape of LLM text datasets, serving as a comprehensive reference for researchers in this field and contributing to future studies. Related resources are available at: https://github.com/lmmlzn/Awesome-LLMs-Datasets.

翻译：本文系统探讨了大语言模型（LLM）数据集，这些数据集在LLM的显著进展中发挥着关键作用。数据集作为基础性基础设施，如同支撑和培育LLM发展的根系系统。因此，对这些数据集的剖析成为研究中的重要议题。为弥补当前LLM数据集缺乏全面概览与深入分析的不足，并洞察其现状与未来趋势，本综述从五个维度整合并归类了LLM数据集的基本要素：（1）预训练语料库；（2）指令微调数据集；（3）偏好数据集；（4）评估数据集；（5）传统自然语言处理（NLP）数据集。本文揭示了当前面临的挑战，并指出未来研究的潜在方向。此外，还提供了对现有可用数据集资源的全面综述，涵盖来自444个数据集的统计数据，覆盖8种语言类别和32个领域，数据集统计信息包含20个维度。总计调查的预训练语料库数据量超过774.5 TB，其他数据集超过7亿实例。我们旨在呈现LLM文本数据集的完整图景，为领域研究者提供全面参考，并助力后续研究。相关资源可访问：https://github.com/lmmlzn/Awesome-LLMs-Datasets。