Data preparation is a trial-and-error process that typically involves countless iterations over the data to define the best pipeline of operators for a given task. With tabular data, practitioners often perform that burdensome activity on local machines by writing ad hoc scripts with libraries based on the Pandas dataframe API and testing them on samples of the entire dataset--the faster the library, the less idle time its users have. In this paper, we evaluate the most popular Python dataframe libraries in general data preparation use cases to assess how they perform on a single machine. To do so, we employ 4 real-world datasets and pipelines with distinct characteristics, covering a variety of scenarios. The insights gained with this experimentation are useful to data scientists who need to choose which of the dataframe libraries best suits their data preparation task at hand. In a nutshell, we found that: for small datasets, Pandas consistently proves to be the best choice with the richest API; when RAM is limited and there is no need to complete compatibility with Pandas API, Polars is the go-to choice thanks to its resource and query optimization; when a GPU is available, CuDF often yields the best performance, while for very large datasets that cannot fit in the GPU memory and RAM, PySpark (thanks to a multi-thread execution and a query optimizer) and Vaex (exploiting a columnar data format) are the best options.

翻译：数据准备是一个反复试错的过程，通常需要对数据进行无数轮迭代，以确定针对特定任务的最佳操作符流水线。在处理表格数据时，从业者往往通过编写基于Pandas数据框架API的临时脚本，并在整个数据集样本上进行测试，在本地机器上完成这一繁重工作——库的运行速度越快，用户的空闲时间就越少。本文评估了在通用数据准备用例中最流行的Python数据框架库，探究它们在单机环境下的性能表现。为此，我们采用了4个具有不同特征的真实数据集和流水线，覆盖了多种应用场景。本实验获得的洞察有助于数据科学家选择最适合当前数据准备任务的数据框架库。简而言之，我们发现：对于小型数据集，Pandas凭借其最丰富的API始终是最优选择；当内存受限且无需完全兼容Pandas API时，Polars凭借其资源与查询优化成为首选；在配备GPU的情况下，CuDF通常能达到最佳性能；而对于无法容纳于GPU内存和RAM的超大规模数据集，PySpark（得益于多线程执行与查询优化器）和Vaex（利用列式数据格式）是最佳方案。