When performing tasks like automatic speech recognition or spoken language understanding for a given utterance, access to preceding text or audio provides contextual information can improve performance. Considering the recent advances in generative large language models (LLM), we hypothesize that an LLM could generate useful context information using the preceding text. With appropriate prompts, LLM could generate a prediction of the next sentence or abstractive text like titles or topics. In this paper, we study the use of LLM-generated context information and propose an approach to distill the generated information during fine-tuning of self-supervised speech models, which we refer to as generative context-aware fine-tuning. This approach allows the fine-tuned model to make improved predictions without access to the true surrounding segments or to the LLM at inference time, while requiring only a very small additional context module. We evaluate the proposed approach using the SLUE and Libri-light benchmarks for several downstream tasks: automatic speech recognition, named entity recognition, and sentiment analysis. The results show that generative context-aware fine-tuning outperforms a context injection fine-tuning approach that accesses the ground-truth previous text, and is competitive with a generative context injection fine-tuning approach that requires the LLM at inference time.

翻译：在执行如自动语音识别或口语理解任务时，对于给定的语句，获取前文文本或音频所提供的上下文信息可以提升性能。鉴于生成式大语言模型（LLM）的最新进展，我们提出假设：LLM可利用前文文本生成有用的上下文信息。通过合适的提示，LLM能够生成下一句的预测，或生成标题、主题等抽象文本。本文研究了LLM生成的上下文信息的使用方法，并提出一种在自监督语音模型微调过程中蒸馏所生成信息的新方法，我们称之为生成式上下文感知微调。该方法使微调后的模型在推理时无需访问真实相邻片段或LLM，仅需一个极小的额外上下文模块，即可实现预测性能的提升。我们使用SLUE和Libri-light基准测试，在自动语音识别、命名实体识别和情感分析等多个下游任务上评估了所提方法。结果表明，生成式上下文感知微调方法优于需访问地面真值历史文本的上下文注入微调方法，且与需在推理时使用LLM的生成式上下文注入微调方法性能相当。