Self-supervised pre-trained transformers have improved the state of the art on a variety of speech tasks. Due to the quadratic time and space complexity of self-attention, they usually operate at the level of relatively short (e.g., utterance) segments. In this paper, we study the use of context, i.e., surrounding segments, during fine-tuning and propose a new approach called context-aware fine-tuning. We attach a context module on top of the last layer of a pre-trained model to encode the whole segment into a context embedding vector which is then used as an additional feature for the final prediction. During the fine-tuning stage, we introduce an auxiliary loss that encourages this context embedding vector to be similar to context vectors of surrounding segments. This allows the model to make predictions without access to these surrounding segments at inference time and requires only a tiny overhead compared to standard fine-tuned models. We evaluate the proposed approach using the SLUE and Libri-light benchmarks for several downstream tasks: Automatic speech recognition (ASR), named entity recognition (NER), and sentiment analysis (SA). The results show that context-aware fine-tuning not only outperforms a standard fine-tuning baseline but also rivals a strong context injection baseline that uses neighboring speech segments during inference.

翻译：自监督预训练Transformer已提升多种语音任务的最新性能。由于自注意力机制的二次时间与空间复杂度，这些模型通常处理相对较短的片段（如话语段）。本文研究微调过程中上下文（即周围片段）的利用方法，并提出一种称为"上下文感知微调"的新方法。我们在预训练模型最后一层之上附加一个上下文模块，将整个片段编码为上下文嵌入向量，该向量随后作为最终预测的附加特征。在微调阶段，我们引入辅助损失函数，促使该上下文嵌入向量与周围片段的上下文向量相似。这使得模型在推理时无需访问这些周围片段即可进行预测，且与标准微调模型相比仅需极小的额外开销。我们使用SLUE和Libri-light基准测试在多个下游任务中评估所提方法：自动语音识别（ASR）、命名实体识别（NER）和情感分析（SA）。结果表明，上下文感知微调不仅优于标准微调基线，还与推理时使用相邻语音段的强上下文注入基线性能相当。