Joint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language. We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models and optimized data-efficiently for spoken language tasks. Particularly, we propose to incorporate cross-modality learning into transfer learning and conduct them simultaneously for downstream tasks in a multi-task learning manner. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks, achieving a new state-of-the-art average BLEU score of 31.5 on the multilingual speech to English text translation task for 21 languages, as measured on the public CoVoST2 evaluation set.

翻译：联合语音-语言训练因训练数据需求大、GPU消耗高以及语音与语言之间的模态差异而面临挑战。我们提出ComSL，一种构建于公开预训练纯语音模型和纯语言模型复合架构之上的语音-语言模型，并以数据高效的方式针对口语任务进行优化。具体而言，我们提出将跨模态学习融入迁移学习，并以多任务学习方式同时为下游任务执行。该方法在端到端语音到文本翻译任务中展现出有效性，在面向21种语言的多语言语音到英语文本翻译任务中，于公开CoVoST2评估集上实现了31.5的平均BLEU分数，达到新的最先进水平。