We present a novel integration of an instruction-tuned large language model (LLM) and end-to-end automatic speech recognition (ASR). Modern LLMs can perform a wide range of linguistic tasks within zero-shot learning when provided with a precise instruction or a prompt to guide the text generation process towards the desired task. We explore using this zero-shot capability of LLMs to extract linguistic information that can contribute to improving ASR performance. Specifically, we direct an LLM to correct grammatical errors in an ASR hypothesis and harness the embedded linguistic knowledge to conduct end-to-end ASR. The proposed model is built on the hybrid connectionist temporal classification (CTC) and attention architecture, where an instruction-tuned LLM (i.e., Llama2) is employed as a front-end of the decoder. An ASR hypothesis, subject to correction, is obtained from the encoder via CTC decoding, which is then fed into the LLM along with an instruction. The decoder subsequently takes as input the LLM embeddings to perform sequence generation, incorporating acoustic information from the encoder output. Experimental results and analyses demonstrate that the proposed integration yields promising performance improvements, and our approach largely benefits from LLM-based rescoring.

翻译：我们提出了一种创新性的集成方法，将指令微调大语言模型（LLM）与端到端自动语音识别（ASR）相结合。现代大语言模型能够在零样本学习框架下，通过精确的指令或提示引导文本生成过程朝向预期任务，从而执行广泛的 linguistic 任务。我们探索利用大语言模型的这种零样本能力来提取有助于提升ASR性能的 linguistic 信息。具体而言，我们引导大语言模型纠正ASR假设中的语法错误，并利用其蕴含的 linguistic 知识实现端到端ASR。所提出的模型基于混合连接时序分类（CTC）与注意力架构，其中采用指令微调大语言模型（即Llama2）作为解码器的前端。通过CTC解码从编码器获取待修正的ASR假设，随后将该假设与指令一同输入大语言模型。解码器随后以大语言模型嵌入作为输入，结合编码器输出的声学信息执行序列生成。实验结果表明，该集成方法能够带来显著的性能提升，且我们的方法主要受益于基于大语言模型的重评分机制。