Adapting a trained Automatic Speech Recognition (ASR) model to new tasks results in catastrophic forgetting of old tasks, limiting the model's ability to learn continually and to be extended to new speakers, dialects, languages, etc. Focusing on End-to-End ASR, in this paper, we propose a simple yet effective method to overcome catastrophic forgetting: weight averaging. By simply taking the average of the previous and the adapted model, our method achieves high performance on both the old and new tasks. It can be further improved by introducing a knowledge distillation loss during the adaptation. We illustrate the effectiveness of our method on both monolingual and multilingual ASR. In both cases, our method strongly outperforms all baselines, even in its simplest form.

翻译：将训练好的自动语音识别（ASR）模型应用于新任务会导致对旧任务的灾难性遗忘，限制了模型持续学习以及扩展到新说话人、方言、语言等的能力。本文聚焦于端到端ASR，提出了一种克服灾难性遗忘的简单有效方法：权重平均。通过简单地对先前模型和微调后模型取平均，我们的方法在新旧任务上均取得了高性能。通过在微调过程中引入知识蒸馏损失，该方法可进一步改进。我们在单语言和多语言ASR上验证了该方法的有效性。两种情况下，即使是最简单的形式，我们的方法也显著优于所有基线方法。