Despite the success of Transformers in self- supervised learning with applications to various downstream tasks, the computational cost of training and inference remains a major challenge for applying these models to a wide spectrum of devices. Several isolated attempts have been made to compress Transformers, but the settings and metrics are different across studies. Trade-off at various compression rates are also largely missing in prior work, making it difficult to compare compression techniques. In this work, we aim to provide context for the isolated results, studying several commonly used compression techniques, including weight pruning, head pruning, low-rank approximation, and knowledge distillation. We report trade- off at various compression rate, including wall-clock time, the number of parameters, and the number of multiply-accumulate operations. Our results show that compared to recent approaches, basic compression techniques are strong baselines. We further present several applications of our results, revealing properties of Transformers, such as the significance of diagonal attention heads. In addition, our results lead to a simple combination of compression techniques that improves trade-off over recent approaches. We hope the results would promote more diverse comparisons among model compression techniques and promote the use of model compression as a tool for analyzing models. Our code of compressing speech self-supervised model is available at https://github.com/nervjack2/Speech-SSL-Compression/.

翻译：尽管Transformer在自监督学习及其下游任务中取得了成功，但训练和推理的计算成本仍是将这些模型应用于广泛设备的主要挑战。已有若干孤立的尝试对Transformer进行压缩，但不同研究中的设置和度量标准各不相同。先前工作也大多缺乏不同压缩率下的权衡比较，使得压缩技术难以对比。本研究旨在为这些孤立结果提供背景，系统研究了多种常用压缩技术，包括权重剪枝、头部剪枝、低秩近似和知识蒸馏。我们报告了不同压缩率下的权衡指标，包括实际运行时间、参数量和乘法累加操作数。结果表明，与近期方法相比，基础压缩技术是强有力的基线。我们进一步展示了研究结果的若干应用，揭示了Transformer的特性，例如对角注意力头的重要性。此外，我们的结果提出了一种简单的压缩技术组合，在权衡表现上优于近期方法。希望本研究能促进模型压缩技术间的多样化对比，并推动模型压缩作为分析模型工具的应用。本文的语音自监督模型压缩代码已开源至https://github.com/nervjack2/Speech-SSL-Compression/。