Transferring the knowledge of large language models (LLMs) is a promising technique to incorporate linguistic knowledge into end-to-end automatic speech recognition (ASR) systems. However, existing works only transfer a single representation of LLM (e.g. the last layer of pretrained BERT), while the representation of a text is inherently non-unique and can be obtained variously from different layers, contexts and models. In this work, we explore a wide range of techniques to obtain and transfer multiple representations of LLMs into a transducer-based ASR system. While being conceptually simple, we show that transferring multiple representations of LLMs can be an effective alternative to transferring only a single representation.

翻译：将大语言模型（LLMs）的知识迁移到端到端自动语音识别（ASR）系统中，是一种将语言知识融入系统的有前景技术。然而，现有工作仅迁移LLM的单一表示（例如，预训练BERT的最后一层），而文本的表示本质上并非唯一，可以从不同的层、上下文和模型中以多种方式获得。在本研究中，我们探索了广泛的技术，以获取并迁移LLM的多重表示到基于转换器的ASR系统中。尽管概念上简单，我们证明迁移LLM的多重表示可以成为仅迁移单一表示的有效替代方案。