The paper presents a study of the efficiency of loading and storing data in the three most common Data Lakehouse systems, including Apache Hudi, Apache Iceberg, and Delta Lake, using Apache Spark as a distributed data processing platform. The study analyzes the behavior of each system when processing structured (CSV) and semi-structured (JSON) data of different sizes, including loading files up to 7 GB in size. The purpose of the work is to determine the most optimal Data Lakehouse architecture based on the type and volume of data sources, data loading performance using Apache Spark, and disk size of data for forming analytical data systems. The research covers the development of four sequential ETL processes, which include reading, transforming, and loading data into tables in each of the Data Lakehouse systems. The efficiency of each Lakehouse was evaluated according to two key criteria: data loading time and the volume of tables formed in the file system. For the first time, a comparison of performance and data storage in Apache Iceberg, Apache Hudi, and Delta Lake Data Lakehouse systems was conducted to select the most relevant architecture for building analytical data systems. The practical value of the study consists in the fact that it assists data engineers and architects in choosing the most appropriate Lakehouse architecture, understanding the balance between loading performance and storage efficiency. Experimental results showed that Delta Lake is the most optimal architecture for systems where the priority is the speed of loading data of any volume, while Apache Iceberg is most appropriate for systems where stability and disk space savings are critical. Apache Hudi proved ineffective in data loading and storage evaluation tasks but could potentially be effective in incremental update and streaming processing scenarios.
翻译:本文针对三大主流数据湖仓系统——Apache Hudi、Apache Iceberg及Delta Lake——在采用Apache Spark作为分布式数据处理平台时的数据加载与存储效率展开研究。实验分析了各系统处理不同规模结构化数据(CSV)与半结构化数据(JSON)的行为特征,涵盖最高7 GB文件的加载场景。研究目标在于根据数据源类型与规模、基于Apache Spark的数据加载性能以及用于构建分析数据系统的数据磁盘占用情况,确定最优数据湖仓架构。本文设计了四个连续的ETL流程,包括数据读取、转换及加载至各湖仓系统表。依据数据加载时间与文件系统形成的表空间占用两大关键指标评估每个湖仓的效率。首次对Apache Iceberg、Apache Hudi与Delta Lake三种数据湖仓系统的性能与存储进行对比,以筛选最适合构建分析数据系统的架构。本研究的实践价值在于协助数据工程师与架构师理解加载性能与存储效率的平衡,从而选择最合适的湖仓架构。实验结果表明:Delta Lake是优先重视任意规模数据加载速度的系统最优架构;Apache Iceberg则最适用于对稳定性与磁盘空间节省要求严苛的系统;Apache Hudi在数据加载与存储评估任务中表现不佳,但在增量更新与流处理场景中可能具有潜在优势。