To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of custom processors with block-based interface in Dataverse allows users to readily and efficiently use Dataverse to build their own ETL pipeline. We hope that Dataverse will serve as a vital tool for LLM development and open source the entire library to welcome community contribution. Additionally, we provide a concise, two-minute video demonstration of our system, illustrating its capabilities and implementation.

翻译：为应对规模化数据处理所面临的挑战，我们提出Dataverse——一个以用户友好设计为核心、面向大语言模型（LLMs）的统一开源ETL（提取-转换-加载）流水线。通过Dataverse中基于模块化接口的便捷自定义处理器添加功能，用户能够轻松高效地构建专属ETL流水线。我们期望Dataverse能成为大语言模型开发的关键工具，并开源整个程序库以欢迎社区贡献。此外，我们提供了一个简洁的两分钟系统演示视频，以阐释其功能与实现方式。