Despite being trained on massive and diverse datasets, speech self-supervised encoders are generally used for downstream purposes as mere frozen feature extractors or model initializers before fine-tuning. The former severely limits the exploitation of large encoders, while the latter hurts the robustness acquired during pretraining, especially in low-resource scenarios. This work explores middle-ground solutions, conjecturing that reducing the forgetting of the self-supervised task during the downstream fine-tuning leads to better generalization. To prove this, focusing on speech recognition, we benchmark different continual-learning approaches during fine-tuning and show that they improve both in-domain and out-of-domain generalization abilities. Relative performance gains reach 15.7% and 22.5% with XLSR used as the encoder on two English and Danish speech recognition tasks. Further probing experiments show that these gains are indeed linked to less forgetting.

翻译：尽管语音自监督编码器是在海量多样化数据集上训练的，但在下游应用中通常仅作为冻结特征提取器或微调前的模型初始化器使用。前者严重限制了大编码器的潜力挖掘，而后者则会损害预训练阶段获得的鲁棒性，在低资源场景下尤为明显。本研究探索折中方案，提出假设：在下游微调过程中减少对自监督任务的遗忘能够提升泛化性能。为验证此假设，我们聚焦语音识别任务，对微调阶段的不同持续学习方法进行基准测试，结果表明这些方法能同时提升模型在领域内和跨领域的泛化能力。以XLSR作为编码器时，在英语和丹麦语两项语音识别任务上分别实现了15.7%和22.5%的相对性能提升。进一步的探测实验证实，这些性能增益确实与更少的遗忘现象相关。