Continual Learning (CL) involves fine-tuning pre-trained models with new data while maintaining the performance on the pre-trained data. This is particularly relevant for expanding multilingual ASR (MASR) capabilities. However, existing CL methods, mainly designed for computer vision and reinforcement learning tasks, often yield sub-optimal results when directly applied to MASR. We hypothesise that this is because CL of the auto-regressive decoder in the MASR model is difficult. To verify this, we propose four optimizations on the decoder. They include decoder-layer gradient surgery, freezing unused token embeddings, suppressing output of newly added tokens, and learning rate re-scaling. Our experiments on adapting Whisper to 10 unseen languages from the Common Voice dataset demonstrate that these optimizations reduce the Average Word Error Rate (AWER) of pretrained languages from 14.2% to 12.4% compared with Experience Replay, without compromising the AWER of new languages.

翻译：持续学习（CL）涉及使用新数据对预训练模型进行微调，同时保持其在预训练数据上的性能。这对于扩展多语言自动语音识别（MASR）能力尤为重要。然而，现有的CL方法主要针对计算机视觉和强化学习任务设计，直接应用于MASR时往往产生次优结果。我们假设这是因为MASR模型中自回归解码器的持续学习存在困难。为验证此假设，我们针对解码器提出了四项优化：解码器层梯度手术、冻结未使用的词元嵌入、抑制新增词元的输出以及学习率重缩放。我们在使用Common Voice数据集的10种未见语言上对Whisper模型进行适配的实验表明，与经验回放方法相比，这些优化将预训练语言的平均词错误率（AWER）从14.2%降低至12.4%，且未损害新语言的AWER。