Large language models (LLMs) have shown incredible proficiency in performing tasks that require semantic understanding of natural language instructions. Recently, many works have further expanded this capability to perceive multimodal audio and text inputs, but their capabilities are often limited to specific fine-tuned tasks such as automatic speech recognition and translation. We therefore develop SpeechVerse, a robust multi-task training and curriculum learning framework that combines pre-trained speech and text foundation models via a small set of learnable parameters, while keeping the pre-trained models frozen during training. The models are instruction finetuned using continuous latent representations extracted from the speech foundation model to achieve optimal zero-shot performance on a diverse range of speech processing tasks using natural language instructions. We perform extensive benchmarking that includes comparing our model performance against traditional baselines across several datasets and tasks. Furthermore, we evaluate the model's capability for generalized instruction following by testing on out-of-domain datasets, novel prompts, and unseen tasks. Our empirical experiments reveal that our multi-task SpeechVerse model is even superior to conventional task-specific baselines on 9 out of the 11 tasks.

翻译：大语言模型（LLMs）在执行需要理解自然语言指令语义的任务中展现出惊人的能力。最近，许多研究进一步扩展了这一能力，使其能够感知多模态音频和文本输入，但这些模型的性能通常局限于特定的微调任务，如自动语音识别与翻译。为此，我们开发了SpeechVerse——一个鲁棒的多任务训练与课程学习框架，通过少量可学习参数结合预训练的语音与文本基础模型，并在训练过程中保持预训练模型参数冻结。该模型利用从语音基础模型中提取的连续潜在表征进行指令微调，从而在面向自然语言指令的多样化语音处理任务中实现最优的零样本性能。我们进行了广泛的基准测试，包括将模型性能与多个数据集和任务上的传统基线模型进行对比。此外，我们还通过域外数据集、新颖提示词及未见任务的测试，评估了模型在通用指令跟随方面的能力。实验结果表明，我们的多任务SpeechVerse模型在11个任务中的9个上甚至优于传统的特定任务基线模型。