Recent years have witnessed great strides in self-supervised learning (SSL) on the speech processing. The SSL model is normally pre-trained on a great variety of unlabelled data and a large model size is preferred to increase the modeling capacity. However, this might limit its potential applications due to the expensive computation and memory costs introduced by the oversize model. Miniaturization for SSL models has become an important research direction of practical value. To this end, we explore the effective distillation of HuBERT-based SSL models for automatic speech recognition (ASR). First, in order to establish a strong baseline, a comprehensive study on different student model structures is conducted. On top of this, as a supplement to the regression loss widely adopted in previous works, a discriminative loss is introduced for HuBERT to enhance the distillation performance, especially in low-resource scenarios. In addition, we design a simple and effective algorithm to distill the front-end input from waveform to Fbank feature, resulting in 17% parameter reduction and doubling inference speed, at marginal performance degradation.

翻译：近年来，自监督学习在语音处理领域取得了重大进展。自监督模型通常在大规模无标签数据上进行预训练，且倾向于采用较大的模型规模以提升建模能力。然而，过大的模型会带来昂贵的计算和内存开销，从而限制其潜在应用。自监督模型的小型化已成为具有实用价值的重要研究方向。为此，我们探索基于HuBERT的自监督模型在自动语音识别中的有效蒸馏方法。首先，为建立强基线，我们对不同学生模型结构进行了全面研究。在此基础上，作为以往工作中广泛采用的回归损失的补充，我们引入了一种判别性损失以增强HuBERT的蒸馏性能，尤其在低资源场景下效果显著。此外，我们设计了一种简单有效的算法，将前段输入从波形蒸馏为Fbank特征，在性能轻微下降的情况下实现了17%的参数减少和两倍的推理速度提升。