Model architectures such as wav2vec 2.0 and HuBERT have been proposed to learn speech representations from audio waveforms in a self-supervised manner. When they are combined with downstream tasks such as keyword spotting and speaker verification, they provide state-of-the-art performance. However, these models use a large number of parameters, the smallest version of which has 95 million parameters. This constitutes a challenge for edge AI device deployments. In this paper, we investigate the application of knowledge distillation to speech representation learning (SRL) models followed by joint fine-tuning with multiple downstream voice-activated tasks. In our experiments on two such tasks, our approach results in nearly 75% reduction in model size while suffering only 0.1% accuracy and 0.9% equal error rate degradation compared to the full-size model. In addition, we show that fine-tuning the SRL models results in a significant performance boost compared to using frozen SRL models.

翻译：诸如wav2vec 2.0和HuBERT等模型架构已被提出，用于以自监督方式从音频波形中学习语音表征。当这些模型与关键词识别、说话人验证等下游任务结合时，能够提供最先进的性能。然而，这些模型使用了大量参数，其最小版本也拥有9500万参数，这对边缘AI设备的部署构成了挑战。本文研究了知识蒸馏在语音表征学习（SRL）模型中的应用，并随后与多个激活语音下游任务进行联合微调。在针对其中两项任务的实验中，我们的方法使模型规模减小了近75%，而与完整模型相比，仅损失了0.1%的准确率和0.9%的等错误率下降。此外，我们证明了与使用冻结的SRL模型相比，对SRL模型进行微调能够带来显著的性能提升。