Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new tasks and deploying them to on-device applications are infeasible. Prior studies propose model compression methods to address this issue, but those works focus on smaller models and less realistic tasks. Thus, we propose Contrastive Layer-to-layer Distillation (CoLLD), a novel knowledge distillation method to compress pre-trained speech encoders by leveraging masked prediction and contrastive learning to train student models to copy the behavior of a large teacher model. CoLLD outperforms prior methods and closes the gap between small and large models on multilingual speech-to-text translation and recognition benchmarks.

翻译：大规模自监督预训练语音编码器在语音识别和翻译任务中表现优于传统方法。由于开发这些大型模型成本高昂，为新任务构建新编码器并将其部署到设备端应用并不可行。先前研究提出了模型压缩方法以解决此问题，但这些工作聚焦于较小模型和非实际任务。因此，我们提出对比逐层蒸馏（CoLLD），一种新型知识蒸馏方法，通过利用掩码预测和对比学习训练学生模型模仿大型教师模型的行为，从而压缩预训练语音编码器。CoLLD优于先前方法，并在多语言语音到文本翻译和识别基准上缩小了小模型与大模型之间的差距。