Large language models have demonstrated remarkable potential in various tasks, however, there remains a significant scarcity of open-source models and data for specific domains. Previous works have primarily focused on manually specifying resources and collecting high-quality data on specific domains, which significantly consume time and effort. To address this limitation, we propose an efficient data collection method~\textit{Query of CC} based on large language models. This method bootstraps seed information through a large language model and retrieves related data from public corpora. It not only collects knowledge-related data for specific domains but unearths the data with potential reasoning procedures. Through the application of this method, we have curated a high-quality dataset called~\textsc{Knowledge Pile}, encompassing four major domains, including stem and humanities sciences, among others. Experimental results demonstrate that~\textsc{Knowledge Pile} significantly improves the performance of large language models in mathematical and knowledge-related reasoning ability tests. To facilitate academic sharing, we open-source our dataset and code, providing valuable support to the academic community.

翻译：大型语言模型在各种任务中展现出显著潜力，但在特定领域仍严重缺乏开源模型和数据。以往研究主要依赖人工指定资源并收集特定领域的高质量数据，这一过程耗费大量时间和精力。为解决这一局限，我们提出了一种基于大型语言模型的高效数据收集方法——Query of CC。该方法通过大型语言模型引导种子信息，并从公开语料库中检索相关数据，不仅能够收集特定领域的知识相关数据，还能挖掘具有潜在推理过程的数据。通过应用该方法，我们构建了一个名为Knowledge Pile的高质量数据集，涵盖包括理工科和人文科学在内的四大领域。实验结果表明，Knowledge Pile显著提升了大型语言模型在数学和知识相关推理能力测试中的表现。为促进学术共享，我们开源了数据集和代码，为学术界提供宝贵支持。