This work studies knowledge distillation (KD) and addresses its constraints for recurrent neural network transducer (RNN-T) models. In hard distillation, a teacher model transcribes large amounts of unlabelled speech to train a student model. Soft distillation is another popular KD method that distills the output logits of the teacher model. Due to the nature of RNN-T alignments, applying soft distillation between RNN-T architectures having different posterior distributions is challenging. In addition, bad teachers having high word-error-rate (WER) reduce the efficacy of KD. We investigate how to effectively distill knowledge from variable quality ASR teachers, which has not been studied before to the best of our knowledge. We show that a sequence-level KD, full-sum distillation, outperforms other distillation methods for RNN-T models, especially for bad teachers. We also propose a variant of full-sum distillation that distills the sequence discriminative knowledge of the teacher leading to further improvement in WER. We conduct experiments on public datasets namely SpeechStew and LibriSpeech, and on in-house production data.

翻译：本研究探讨知识蒸馏（KD）技术并解决其在循环神经网络换能器（RNN-T）模型中的应用限制。在硬蒸馏中，教师模型转录大量无标注语音以训练学生模型。软蒸馏是另一种流行的KD方法，通过蒸馏教师模型的输出logits实现。由于RNN-T对齐的特性，在具有不同后验分布的RNN-T架构间应用软蒸馏颇具挑战性。此外，词错误率（WER）较高的低质量教师模型会降低KD的有效性。我们研究了如何从不同质量的自动语音识别（ASR）教师模型中有效蒸馏知识，据我们所知，这一方向此前尚未被探索。研究表明，序列级KD方法——全和蒸馏——在RNN-T模型中优于其他蒸馏方法，尤其适用于低质量教师模型。我们还提出全和蒸馏的一种变体，通过蒸馏教师模型的序列判别性知识进一步提升词错误率。我们分别在SpeechStew和LibriSpeech公开数据集以及内部生产数据上进行了实验验证。