Recently, large pre-trained multilingual speech models have shown potential in scaling Automatic Speech Recognition (ASR) to many low-resource languages. Some of these models employ language adapters in their formulation, which helps to improve monolingual performance and avoids some of the drawbacks of multi-lingual modeling on resource-rich languages. However, this formulation restricts the usability of these models on code-switched speech, where two languages are mixed together in the same utterance. In this work, we propose ways to effectively fine-tune such models on code-switched speech, by assimilating information from both language adapters at each language adaptation point in the network. We also model code-switching as a sequence of latent binary sequences that can be used to guide the flow of information from each language adapter at the frame level. The proposed approaches are evaluated on three code-switched datasets encompassing Arabic, Mandarin, and Hindi languages paired with English, showing consistent improvements in code-switching performance with at least 10\% absolute reduction in CER across all test sets.

翻译：近期，大规模预训练多语言语音模型在将自动语音识别（ASR）扩展到众多低资源语言方面展现出潜力。其中部分模型在其架构中采用了语言适配器，这不仅有助于提升单语性能，还能避免多语言建模对资源丰富语言带来的某些缺陷。然而，这种架构限制了此类模型在代码转换语音（即同一语句中混合两种语言）中的可用性。在本工作中，我们提出了有效微调此类模型以处理代码转换语音的方法，通过在网络的每个语言适配点整合来自两种语言适配器的信息。同时，我们将代码转换建模为一系列潜在二元序列，用于在帧级别引导各语言适配器的信息流。所提方法在三个涵盖阿拉伯语、普通话和印地语与英语混合的代码转换数据集上进行了评估，所有测试集的字符错误率（CER）均实现至少10%的绝对降幅，显示出代码转换性能的持续提升。