Large pre-trained language models (PLMs) have shown remarkable performance across various natural language understanding (NLU) tasks, particularly in low-resource settings. Nevertheless, their potential in Automatic Speech Recognition (ASR) remains largely unexplored. This study investigates the potential usage of PLMs for language modelling in ASR. We compare the application of large-scale text sampling and probability conversion for approximating GPT-2 into an n-gram model. Furthermore, we introduce a vocabulary-restricted decoding method for random sampling, and evaluate the effects of domain difficulty and data size on the usability of generated text. Our findings across eight domain-specific corpora support the use of sampling-based approximation and show that interpolating with a large sampled corpus improves test perplexity over a baseline trigram by 15%. Our vocabulary-restricted decoding method pushes this improvement further by 5% in domain-specific settings.

翻译：大规模预训练语言模型（PLMs）在各类自然语言理解（NLU）任务中已展现出卓越性能，尤其是在低资源场景下。然而，其在自动语音识别（ASR）领域的潜力仍待深入探索。本研究旨在探讨PLMs在ASR语言建模中的潜在应用价值。我们对比了通过大规模文本采样与概率转换将GPT-2近似为n-gram模型的方法，并提出了一种面向随机采样的词汇受限解码策略，同时评估了领域难度与数据规模对生成文本可用性的影响。基于八个领域特定语料库的实验结果表明：基于采样的近似方法具有可行性，且将模型与大规模采样语料进行插值后，测试困惑度相较三元组基线模型降低了15%；而所提出的词汇受限解码策略在领域特定场景下进一步将这一提升幅度扩大了5%。