The immense evolution in Large Language Models (LLMs) has underscored the importance of massive, heterogeneous, and high-quality data. A data recipe is a mixture of data from different sources for training LLMs, which plays a vital role in LLMs' performance. Existing open-source tools for LLM data processing are mostly tailored for specific data recipes. To continuously uncover the potential of LLMs, incorporate data from new sources, and improve LLMs' performance, we build a new system named Data-Juicer, with which we can efficiently generate diverse data recipes, explore different possibilities in forming data mixtures, and evaluate their effects on model performance. Different from traditional data-analytics pipelines, Data-Juicer faces some unique challenges. Firstly, the possible data sources for forming data recipes are truly heterogeneous and massive with various qualities. Secondly, it is extremely expensive to precisely evaluate data recipes' impact on LLMs' performance. Thirdly, the end users of Data-Juicer, model developers, need sufficient flexibility to configure and evaluate different data recipes. Data-Juicer features a fine-grained abstraction of pipelines for constructing data recipes, with over 50 built-in operators for easy composition and extension. By incorporating visualization and auto-evaluation capabilities, Data-Juicer enables a timely feedback loop for both LLM pre-training and fine-tuning. Further, Data-Juicer is optimized and integrated with ecosystems for LLM training, evaluation, and distributed computing. The data recipes derived with Data-Juicer gain notable improvements on state-of-the-art LLMs, by up to 7.45% increase in averaged score across 16 LLM benchmarks and 17.5% higher win rate in pair-wise GPT-4 evaluations. Our system, data recipes, and tutorials are released, calling for broader data-centric research on training and understanding LLMs.

翻译：大型语言模型（LLM）的迅猛发展凸显了海量、异构、高质量数据的重要性。数据配方（data recipe）是用于训练LLM的混合数据源集合，对LLM性能起关键作用。现有开源LLM数据处理工具大多针对特定数据配方定制。为持续挖掘LLM潜力、整合新来源数据并提升LLM性能，我们构建了名为Data-Juicer的新系统，可高效生成多样化数据配方、探索数据混合方案的不同可能性，并评估其对模型性能的影响。与传统数据分析流水线不同，Data-Juicer面临独特挑战：首先，构成数据配方的潜在数据源具有高度异质性与规模性，且质量参差；其次，精确评估数据配方对LLM性能的影响成本极高；最后，作为终端用户的模型开发者需要足够灵活性来配置和评估不同数据配方。Data-Juicer采用细粒度的流水线抽象来构建数据配方，内置50余个算子以支持便捷组合与扩展。通过集成可视化与自动评估功能，系统能为LLM预训练和微调提供及时反馈循环。此外，Data-Juicer经过优化并与LLM训练、评估及分布式计算生态系统深度整合。利用Data-Juicer导出的数据配方在先进LLM上取得显著提升：在16项LLM基准测试中平均得分提升高达7.45%，在GPT-4成对评估中胜率提升17.5%。我们已开源本系统、数据配方及教程，呼吁开展更广泛的数据中心型研究，以推动LLM的训练与理解。