Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques such as pruning aim to reduce the model size and computation without degradation in accuracy. Prior studies focus on the pruning of Transformers; however, speech models not only utilize a stack of Transformer blocks, but also combine a frontend network based on multiple convolutional layers for low-level feature representation learning. This frontend has a small size but a heavy computational cost. In this work, we propose three task-specific structured pruning methods to deal with such heterogeneous networks. Experiments on LibriSpeech and SLURP show that the proposed method is more accurate than the original wav2vec2-base with 10% to 30% less computation, and is able to reduce the computation by 40% to 50% without any degradation.

翻译：自监督语音表示学习（SSL）在各类下游任务中已展现出有效性，但SSL模型通常规模庞大且运行缓慢。模型压缩技术（如剪枝）旨在不降低精度的前提下减少模型尺寸与计算开销。现有研究集中于Transformer的剪枝，然而语音模型不仅使用堆叠的Transformer模块，还结合了基于多层卷积的前端网络用于低层特征表示学习。该前端网络虽尺寸较小但计算成本高昂。本文针对此类异构网络提出三种任务特定的结构化剪枝方法。在LibriSpeech和SLURP上的实验表明，所提方法相较于原始wav2vec2-base在减少10%-30%计算量的情况下具有更高精度，且可在无精度损失前提下将计算量降低40%-50%。