Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which leverages the rich linguistic knowledge and powerful reasoning ability of LLMs to improve recognition results. The latest work proposes a GER benchmark with HyPoradise dataset to learn the mapping from ASR N-best hypotheses to ground-truth transcription by efficient LLM finetuning, which shows great effectiveness but lacks specificity on noise-robust ASR. In this work, we extend the benchmark to noisy conditions and investigate if we can teach LLMs to perform denoising for GER just like what robust ASR do}, where one solution is introducing noise information as a conditioner into LLM. However, directly incorporating noise embeddings from audio encoder could harm the LLM tuning due to cross-modality gap. To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER. Furthermore, in order to enhance its representation ability of audio noise, we design a knowledge distillation (KD) approach via mutual information estimation to distill the real noise information in audio embeddings to our language embedding. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement in terms of word error rate while with limited training data. Analysis shows that our language-space noise embedding can well represent the noise conditions of source speech, under which off-the-shelf LLMs show strong ability of language-space denoising.

翻译：近期大型语言模型（LLMs）的进展推动了自动语音识别（ASR）的生成式纠错（GER）技术发展，该技术通过利用LLMs丰富的语言知识和强大的推理能力来改进识别结果。最新研究提出基于HyPoradise数据集的GER基准，通过高效LLM微调学习从ASR N-best假设到真实转录的映射，该方法虽展现出显著效果，但在噪声鲁棒ASR方面缺乏针对性。本研究将该基准扩展至噪声环境，探究能否像鲁棒ASR那样教导LLMs执行去噪GER，其中一种解决方案是将噪声信息作为条件引入LLM。然而，由于跨模态差异，直接融合来自音频编码器的噪声嵌入会损害LLM调优效果。为此，我们提出从N-best列表中提取语言空间噪声嵌入以表征源语音的噪声条件，从而促进GER中的去噪过程。此外，为增强其对音频噪声的表征能力，我们设计了基于互信息估计的知识蒸馏（KD）方法，将音频嵌入中的真实噪声信息蒸馏至语言嵌入中。在多种最新LLMs上的实验表明，本方法在仅使用有限训练数据的条件下，实现了词错误率最高达53.9%的纠正改进突破。分析显示，我们提出的语言空间噪声嵌入能有效表征源语音的噪声条件，在此条件下，现成LLMs展现出强大的语言空间去噪能力。