Recent years have witnessed great strides in self-supervised learning (SSL) on the speech processing. The SSL model is normally pre-trained on a great variety of unlabelled data and a large model size is preferred to increase the modeling capacity. However, this might limit its potential applications due to the expensive computation and memory costs introduced by the oversize model. Miniaturization for SSL models has become an important research direction of practical value. To this end, we explore the effective distillation of HuBERT-based SSL models for automatic speech recognition (ASR). First, in order to establish a strong baseline, a comprehensive study on different student model structures is conducted. On top of this, as a supplement to the regression loss widely adopted in previous works, a discriminative loss is introduced for HuBERT to enhance the distillation performance, especially in low-resource scenarios. In addition, we design a simple and effective algorithm to distill the front-end input from waveform to Fbank feature, resulting in 17% parameter reduction and doubling inference speed, at marginal performance degradation.

翻译：近年来，自监督学习（SSL）在语音处理领域取得了显著进展。SSL模型通常在大规模未标注数据上进行预训练，且倾向于采用较大的模型尺寸以增强建模能力。然而，过大的模型会带来高昂的计算和存储成本，限制了其潜在应用。SSL模型的小型化已成为具有实际价值的重要研究方向。为此，我们探索了基于HuBERT的SSL模型在自动语音识别（ASR）中的有效蒸馏方法。首先，为建立强基线，我们对不同学生模型结构进行了全面研究。在此基础上，作为先前工作中广泛采用的回归损失的补充，我们引入了一种判别损失来增强HuBERT的蒸馏性能，尤其在低资源场景下效果显著。此外，我们设计了一种简单高效的算法，将前端输入从波形蒸馏为Fbank特征，在性能几乎无损失的情况下，实现了17%的参数缩减和两倍的推理速度提升。