Large language models (LLMs) have shown incredible proficiency in performing tasks that require semantic understanding of natural language instructions. Recently, many works have further expanded this capability to perceive multimodal audio and text inputs, but their capabilities are often limited to specific fine-tuned tasks such as automatic speech recognition and translation. We therefore develop SpeechVerse, a robust multi-task training and curriculum learning framework that combines pre-trained speech and text foundation models via a small set of learnable parameters, while keeping the pre-trained models frozen during training. The models are instruction finetuned using continuous latent representations extracted from the speech foundation model to achieve optimal zero-shot performance on a diverse range of speech processing tasks using natural language instructions. We perform extensive benchmarking that includes comparing our model performance against traditional baselines across several datasets and tasks. Furthermore, we evaluate the model's capability for generalized instruction following by testing on out-of-domain datasets, novel prompts, and unseen tasks. Our empirical experiments reveal that our multi-task SpeechVerse model is even superior to conventional task-specific baselines on 9 out of the 11 tasks.

翻译：大型语言模型（LLM）在执行需要理解自然语言指令语义的任务中展现出卓越的能力。近期，许多研究进一步扩展了其感知多模态音频与文本输入的能力，但这些能力通常局限于自动语音识别、翻译等特定的微调任务。为此，我们开发了SpeechVerse——一个鲁棒的多任务训练与课程学习框架，该框架通过少量可学习参数将预训练的语音与文本基础模型相结合，同时在训练过程中保持预训练模型参数冻结。模型使用从语音基础模型中提取的连续潜在表示进行指令微调，从而在基于自然语言指令的多样化语音处理任务上实现最优的零样本性能。我们进行了广泛的基准测试，包括在多个数据集和任务上将我们的模型性能与传统基线模型进行比较。此外，我们通过在域外数据集、新提示词及未见任务上的测试，评估了模型在泛化指令遵循方面的能力。实验结果表明，我们的多任务SpeechVerse模型在11项任务中的9项上甚至优于传统的任务专用基线模型。