A simple way to improve the performance of almost any machine learning model is not to train a single but several models with diverse algorithms which will make slightly distinct kinds of predictions and errors on the same data, and thus improve the average predictions and robustness. However, making predictions using a whole ensemble of models is cumbersome and computationally too expensive to allow deployment to a large number of users, especially if the models are large neural nets. In response to this, we introduce a layer and point wise projection mapping, which maps student and teacher representations into an aligned high-dimensional embedding space during training process. The proposed approach combined with LoRA injection reduces the student model trainable parameters to less than 1% of the teacher model, while significantly improving word error rate (WER) compared to other distillation methods, as demonstrated in ablation studies. Unlike a mixture of experts, our method can be trained rapidly and in parallel.

翻译：提升几乎任何机器学习模型性能的一种简单方法，并非训练单一模型，而是使用多种算法训练多个模型。这些模型对相同数据会做出略有不同的预测和产生略有不同的错误，从而提升平均预测准确率和鲁棒性。然而，使用整体集成模型进行预测既繁琐又计算成本过高，难以部署给大量用户，尤其是在模型为大型神经网络的情况下。针对这一问题，我们引入了一种逐层逐点投影映射方法，在训练过程中将学生模型和教师模型的表示映射到一个对齐的高维嵌入空间中。所提出的方法结合LoRA注入，将学生模型的可训练参数减少至教师模型的不足1%，同时相较于其他蒸馏方法，显著降低了词错误率（WER），消融研究已证明这一点。与混合专家方法不同，我们的方法可以快速并行训练。