The ability of the foundation models heavily relies on large-scale, diverse, and high-quality pretraining data. In order to improve data quality, researchers and practitioners often have to manually curate datasets from difference sources and develop dedicated data cleansing pipeline for each data repository. Lacking a unified data processing framework, this process is repetitive and cumbersome. To mitigate this issue, we propose a data processing framework that integrates a Processing Module which consists of a series of operators at different granularity levels, and an Analyzing Module which supports probing and evaluation of the refined data. The proposed framework is easy to use and highly flexible. In this demo paper, we first introduce how to use this framework with some example use cases and then demonstrate its effectiveness in improving the data quality with an automated evaluation with ChatGPT and an end-to-end evaluation in pretraining the GPT-2 model. The code and demonstration videos are accessible on GitHub.

翻译：基础模型的能力高度依赖于大规模、多样且高质量的预训练数据。为了提升数据质量，研究人员和实践者通常需要手动从不同来源整理数据集，并为每个数据仓库开发专门的数据清洗流程。由于缺乏统一的数据处理框架，这一过程变得重复且繁琐。为解决此问题，我们提出一个数据处理框架，该框架集成了一个由不同粒度层级运算符组成的处理模块，以及一个支持对精炼数据进行探查与评估的分析模块。所提出的框架易于使用且高度灵活。在本演示论文中，我们首先通过若干示例用例介绍如何使用该框架，随后通过基于ChatGPT的自动评估以及在预训练GPT-2模型上的端到端评估，展示其在提升数据质量方面的有效性。相关代码与演示视频可在GitHub上获取。