We propose an unsupervised adaptation framework, Self-TAught Recognizer (STAR), which leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains, such as noise and accents. STAR is developed for prevalent speech foundation models based on Transformer-related architecture with auto-regressive decoding (e.g., Whisper, Canary). Specifically, we propose a novel indicator that empirically integrates step-wise information during decoding to assess the token-level quality of pseudo labels without ground truth, thereby guiding model updates for effective unsupervised adaptation. Experimental results show that STAR achieves an average of 13.5% relative reduction in word error rate across 14 target domains, and it sometimes even approaches the upper-bound performance of supervised adaptation. Surprisingly, we also observe that STAR prevents the adapted model from the common catastrophic forgetting problem without recalling source-domain data. Furthermore, STAR exhibits high data efficiency that only requires less than one-hour unlabeled data, and seamless generality to alternative large speech models and speech translation tasks. Our code aims to open source to the research communities.

翻译：我们提出了一种无监督自适应框架——自学习识别器（STAR），该框架利用未标注数据来增强自动语音识别（ASR）系统在多样化目标领域（如噪声和口音）中的鲁棒性。STAR专为基于Transformer相关架构并采用自回归解码的主流语音基础模型（例如Whisper、Canary）而设计。具体而言，我们提出了一种新颖的指标，该指标在解码过程中经验性地整合步进信息，以在缺乏真实标注的情况下评估伪标签在词元级别的质量，从而指导模型更新以实现有效的无监督自适应。实验结果表明，STAR在14个目标领域上平均实现了13.5%的词错误率相对降低，有时甚至接近有监督自适应的上限性能。令人惊讶的是，我们还观察到STAR在无需召回源领域数据的情况下，防止了自适应模型出现常见的灾难性遗忘问题。此外，STAR展现出较高的数据效率，仅需不足一小时的未标注数据，并对替代性大型语音模型及语音翻译任务具有无缝的泛化能力。我们的代码旨在向研究社区开源。