Although large foundation models pre-trained by self-supervised learning have achieved state-of-the-art performance in many tasks including automatic speech recognition (ASR), knowledge distillation (KD) is often required in practice to transfer the knowledge learned by large teacher models into much smaller student models with affordable computation and memory costs. This paper proposes a novel two-stage KD framework to distil the knowledge from multiple speech foundation models as teachers into a single student neural transducer model for ASR. In the first stage, the student model encoder is pre-trained using the embeddings extracted from multiple teacher models. In the second stage, the student encoder is fine-tuned with the audio-text pairs based on the ASR task. Experiments on the LibriSpeech 100-hour subset show that the proposed KD framework improves the performance of both streaming and non-streaming student models when using only one teacher. The performance of the student model can be further enhanced when multiple teachers are used jointly, achieving word error rate reductions (WERRs) of 17.5% and 10.6%. Our proposed framework can be combined with other existing KD methods to achieve further improvements. Further WERRs were obtained by incorporating extra unlabelled data during encoder pre-training, leading to a total relative WERR of 55.0% on the non-streaming student model.

翻译：尽管通过自监督学习预训练的大型基础模型在包括自动语音识别（ASR）在内的许多任务中已取得最先进性能，但在实际应用中，常需采用知识蒸馏（KD）将大型教师模型学到的知识迁移到计算和存储成本更易负担的小型学生模型中。本文提出一种新颖的两阶段知识蒸馏框架，用于将多个语音基础模型作为教师模型的知识蒸馏到单个学生神经换能器模型中，以完成ASR任务。第一阶段，利用从多个教师模型中提取的嵌入向量对学生模型编码器进行预训练；第二阶段，基于ASR任务的音频-文本对对学生编码器进行微调。在LibriSpeech 100小时子集上的实验表明，即使仅使用单个教师模型，所提出的知识蒸馏框架也能提升流式与非流式学生模型的性能。当联合使用多个教师模型时，学生模型性能可进一步提升，词错误率降低（WERR）分别达到17.5%和10.6%。所提出的框架可与其他现有知识蒸馏方法结合，实现进一步改进。通过在编码器预训练阶段引入额外无标注数据，可获取更多词错误率降低，最终使非流式学生模型的相对词错误率总降低达55.0%。