The recent advancements in large language models (LLMs) have revolutionized the field of natural language processing, progressively broadening their scope to multimodal perception and generation. However, effectively integrating listening capabilities into LLMs poses significant challenges, particularly with respect to generalizing across varied contexts and executing complex auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach. Leveraging dual encoders, we decouple different types of speech information, utilizing a Whisper encoder to process the semantic content of speech, and a WavLM encoder to capture the unique characteristics of the speaker's identity. Within the curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. Furthermore, our model successfully completes Gaokao tasks without specialized training. The codes, models, audio, and Gaokao evaluation set can be accessed at \url{aka.ms/wavllm}.

翻译：大语言模型（LLM）的最新进展已彻底革新自然语言处理领域，并逐步拓展至多模态感知与生成范畴。然而，将听力能力有效集成至LLM仍面临重大挑战，尤其在跨上下文泛化与执行复杂听觉任务方面。本研究提出WavLLM——一种具有双编码器及提示感知LoRA权重适配器的鲁棒自适应语音大语言模型，并通过两阶段课程学习优化策略实现。利用双编码器设计，我们解耦不同类型的语音信息：采用Whisper编码器处理语音语义内容，WavLM编码器捕捉说话人身份独特特征。在课程学习框架下，WavLLM首先通过优化混合基础单任务构建核心能力，继而面向更复杂任务（如基础任务组合）进行高级多任务训练。为增强对不同任务与指令的灵活适配性，在第二阶段高级多任务训练中引入提示感知LoRA权重适配器。我们在通用语音基准（包括ASR、ST、SV、ER等任务）及专项数据集（如高考英语听力理解集用于口语问答、语音思维链评估集）上验证模型性能。实验表明，在相同模型规模下，本模型在多项语音任务中达到最优性能，并通过思维链方法展现出执行复杂任务的鲁棒泛化能力。此外，模型无需专项训练即可成功完成高考任务。代码、模型、音频及高考评估集可在\url{aka.ms/wavllm}获取。