External sorting is at the core of many operations in large-scale database systems, such as ordering and aggregation queries for large result sets, building indexes, sort-merge joins, duplicate removal, sharding, and record clustering. Unlike in-memory sorting, these algorithms need to work together with the OS and the filesystem to efficiently utilize system resources and minimize disk I/O. In this paper we describe ELSAR: a parallel external sorting algorithm that uses an innovative paradigm based on a learned data distribution model. The algorithm leverages the model to arrange the input records into mutually exclusive, monotonic, and equi-depth partitions that, once sorted, can simply be concatenated to form the output. This method completely eliminates the need for multi-way file merging, which is typically used in external sorting. We present thorough benchmarks for uniform and skewed datasets in various storage media, where we measure the sorting rates, size scalability, and energy efficiency of ELSAR and other sorting algorithms. We observed that ELSAR has up to 1.65x higher sorting rates than the next-best external sort (Nsort) on SSD drives and 5.31x higher than the GNU coreutils' sort utility on Intel Optane non-volatile memory. In addition, ELSAR supersedes the current winner of the SortBenchmark for the most energy-efficient external string sorting algorithm by an impressive margin of 41%. These results reinforce the premise that novel learning-enhanced algorithms can provide remarkable performance benefits over traditional ones.

翻译：外部排序是大规模数据库系统中许多操作的核心，例如对大型结果集进行排序和聚合查询、构建索引、排序合并连接、去重、分片以及记录聚类。与内存排序不同，这些算法需要与操作系统和文件系统协同工作，高效利用系统资源并最小化磁盘I/O。本文描述了ELSAR：一种并行外部排序算法，它采用基于学习数据分布模型的创新范式。该算法利用模型将输入记录划分为互斥、单调且等深的区域，这些区域在排序后直接连接形成输出。该方法完全消除了外部排序中通常使用的多路文件合并。我们在各种存储介质上对均匀和倾斜数据集进行了全面基准测试，测量了ELSAR及其他排序算法的排序速率、规模可扩展性和能效。我们观察到，在SSD驱动器上，ELSAR的排序速率比次优的外部排序算法（Nsort）高出1.65倍；在Intel Optane非易失性内存上，比GNU coreutils的sort工具高出5.31倍。此外，ELSAR在SortBenchmark最节能外部字符串排序算法评比中以41%的显著优势超越当前优胜者。这些结果强化了新型学习增强算法相比传统算法能提供显著性能优势的论点。