There is a growing interest in cost-effective self-supervised fine-tuning (SSFT) of self-supervised learning (SSL)-based speech models to obtain task-specific representations. These task-specific representations are used for robust performance on various downstream tasks by fine-tuning on the labelled data. This work presents a cost-effective SSFT method named Self-supervised Correspondence (SCORE) fine-tuning to adapt the SSL speech representations for content-related tasks. The proposed method uses a correspondence training strategy, aiming to learn similar representations from perturbed speech and original speech. Commonly used data augmentation techniques for content-related tasks (ASR) are applied to obtain perturbed speech. SCORE fine-tuned HuBERT outperforms the vanilla HuBERT on SUPERB benchmark with only a few hours of fine-tuning (< 5 hrs) on a single GPU for automatic speech recognition, phoneme recognition, and query-by-example tasks, with relative improvements of 1.09%, 3.58%, and 12.65%, respectively. SCORE provides competitive results with the recently proposed SSFT method SPIN, using only 1/3 of the processed speech compared to SPIN.

翻译：针对基于自监督学习（SSL）的语音模型，学界对经济高效的自监督微调（SSFT）方法日益关注，旨在获取任务特定表征。通过在有标注数据上进行微调，这些任务特定表征可为各类下游任务提供稳健性能。本文提出一种经济高效的SSFT方法——自监督对应（SCORE）微调，用于适配SSL语音表征以处理内容相关任务。该方法采用对应训练策略，旨在从扰动语音与原始语音中学习相似表征。通过应用内容相关任务（ASR）中常用的数据增强技术获取扰动语音。在SUPERB基准测试中，经SCORE微调的HuBERT模型在自动语音识别、音素识别及样例查询任务上均优于原始HuBERT，且仅需单GPU数小时（<5小时）微调，相对性能提升分别达1.09%、3.58%和12.65%。相较于近期提出的SSFT方法SPIN，SCORE在仅使用其1/3处理语音量的情况下即可获得具有竞争力的结果。