Columnar storage is one of the core components of a modern data analytics system. Although many database management systems (DBMSs) have proprietary storage formats, most provide extensive support to open-source storage formats such as Parquet and ORC to facilitate cross-platform data sharing. But these formats were developed over a decade ago, in the early 2010s, for the Hadoop ecosystem. Since then, both the hardware and workload landscapes have changed significantly. In this paper, we revisit the most widely adopted open-source columnar storage formats (Parquet and ORC) with a deep dive into their internals. We designed a benchmark to stress-test the formats' performance and space efficiency under different workload configurations. From our comprehensive evaluation of Parquet and ORC, we identify design decisions advantageous with modern hardware and real-world data distributions. These include using dictionary encoding by default, favoring decoding speed over compression ratio for integer encoding algorithms, making block compression optional, and embedding finer-grained auxiliary data structures. Our analysis identifies important considerations that may guide future formats to better fit modern technology trends.

翻译：列式存储是现代数据分析系统的核心组件之一。尽管许多数据库管理系统（DBMS）拥有专有存储格式，但大多数系统对Parquet和ORC等开源存储格式提供了广泛支持，以促进跨平台数据共享。然而，这些格式开发于十多年前（2010年代初期）的Hadoop生态系统。自那时起，硬件和工作负载格局均发生了显著变化。本文深入探究了最广泛采用的开源列式存储格式（Parquet和ORC）的内部机制，设计了一套基准测试以评估这些格式在不同工作负载配置下的性能与空间效率。通过对Parquet和ORC的全面评估，我们识别出与现代硬件及真实数据分布相适应的设计决策，包括：默认采用字典编码、整数编码算法优先考虑解码速度而非压缩比、将块压缩设为可选项以及嵌入更细粒度的辅助数据结构。我们的分析揭示了指导未来存储格式更贴合现代技术趋势的关键考量因素。