This paper presents InterMPL, a semi-supervised learning method of end-to-end automatic speech recognition (ASR) that performs pseudo-labeling (PL) with intermediate supervision. Momentum PL (MPL) trains a connectionist temporal classification (CTC)-based model on unlabeled data by continuously generating pseudo-labels on the fly and improving their quality. In contrast to autoregressive formulations, such as the attention-based encoder-decoder and transducer, CTC is well suited for MPL, or PL-based semi-supervised ASR in general, owing to its simple/fast inference algorithm and robustness against generating collapsed labels. However, CTC generally yields inferior performance than the autoregressive models due to the conditional independence assumption, thereby limiting the performance of MPL. We propose to enhance MPL by introducing intermediate loss, inspired by the recent advances in CTC-based modeling. Specifically, we focus on self-conditional and hierarchical conditional CTC, that apply auxiliary CTC losses to intermediate layers such that the conditional independence assumption is explicitly relaxed. We also explore how pseudo-labels should be generated and used as supervision for intermediate losses. Experimental results in different semi-supervised settings demonstrate that the proposed approach outperforms MPL and improves an ASR model by up to a 12.1% absolute performance gain. In addition, our detailed analysis validates the importance of the intermediate loss.

翻译：本文提出InterMPL，一种基于中间监督的端到端自动语音识别（ASR）半监督学习方法。动量伪标签（MPL）通过持续在线生成伪标签并提升其质量，在无标注数据上训练基于连接时序分类（CTC）的模型。与自回归模型（如注意力编码-解码器和换能器）不同，CTC凭借其简单快速的推理算法以及对抗标签坍缩的鲁棒性，天然适用于MPL及基于伪标签的半监督ASR。然而，由于条件独立假设，CTC性能通常劣于自回归模型，从而限制了MPL的表现。受近期CTC建模进展启发，我们通过引入中间损失来增强MPL。具体而言，我们聚焦于自条件CTC与层次条件CTC，这两类方法在中间层应用辅助CTC损失，从而显式放宽条件独立假设。同时，我们探索了伪标签的生成策略及其作为中间损失监督信号的方式。不同半监督设置下的实验结果表明，所提方法优于MPL，可为ASR模型带来最高12.1%的绝对性能提升。此外，详细分析验证了中间损失的重要性。