As the number of pre-trained machine learning (ML) models is growing exponentially, data reduction tools are not catching up. Existing data reduction techniques are not specifically designed for pre-trained model (PTM) dataset files. This is largely due to a lack of understanding of the patterns and characteristics of these datasets, especially those relevant to data reduction and compressibility. This paper presents the first, exhaustive analysis to date of PTM datasets on storage compressibility. Our analysis spans different types of data reduction and compression techniques, from hash-based data deduplication, data similarity detection, to dictionary-coding compression. Our analysis explores these techniques at three data granularity levels, from model layers, model chunks, to model parameters. We draw new observations that indicate that modern data reduction tools are not effective when handling PTM datasets. There is a pressing need for new compression methods that take into account PTMs' data characteristics for effective storage reduction. Motivated by our findings, we design ELF, a simple yet effective, error-bounded, lossy floating-point compression method. ELF transforms floating-point parameters in such a way that the common exponent field of the transformed parameters can be completely eliminated to save storage space. We develop Elves, a compression framework that integrates ELF along with several other data reduction methods. Elves uses the most effective method to compress PTMs that exhibit different patterns. Evaluation shows that Elves achieves an overall compression ratio of $1.52\times$, which is $1.31\times$, $1.32\times$ and $1.29\times$ higher than a general-purpose compressor (zstd), an error-bounded lossy compressor (SZ3), and the uniform model quantization, respectively, with negligible model accuracy loss.

翻译：随着预训练机器学习模型数量呈指数级增长，数据缩减工具却未能跟上步伐。现有数据缩减技术并非专门针对预训练模型数据集文件设计，这主要是由于对这些数据集（尤其是与数据缩减和可压缩性相关的模式与特征）缺乏理解。本文首次对预训练模型数据集在存储可压缩性方面进行了全面分析。我们的研究涵盖从基于哈希的数据去重、数据相似性检测到词典编码压缩等不同类型的数据缩减与压缩技术，并在模型层、模型块、模型参数三个数据粒度层级上探索这些技术。我们提出的新观察表明，现代数据缩减工具在处理预训练模型数据集时效果不佳，迫切需要考虑预训练模型数据特征的新型压缩方法以实现有效存储缩减。受研究结果启发，我们设计了ELF——一种简单而有效、具有误差约束的有损浮点压缩方法。ELF对浮点参数进行变换，使得变换后参数的公共指数域可被完全消除以节省存储空间。我们开发了Elves压缩框架，该框架整合ELF及其他多种数据缩减方法，采用最有效的方式对不同模式下的预训练模型进行压缩。评估表明，Elves实现了1.52倍的总体压缩比，分别比通用压缩器（zstd）、误差约束有损压缩器（SZ3）和均匀模型量化高出1.31倍、1.32倍和1.29倍，且模型精度损失可忽略不计。