In this paper, we propose a Unified pre-training Framework for Online and Offline (UFO2) Automatic Speech Recognition (ASR), which 1) simplifies the two separate training workflows for online and offline modes into one process, and 2) improves the Word Error Rate (WER) performance with limited utterance annotating. Specifically, we extend the conventional offline-mode Self-Supervised Learning (SSL)-based ASR approach to a unified manner, where the model training is conditioned on both the full-context and dynamic-chunked inputs. To enhance the pre-trained representation model, stop-gradient operation is applied to decouple the online-mode objectives to the quantizer. Moreover, in both the pre-training and the downstream fine-tuning stages, joint losses are proposed to train the unified model with full-weight sharing for the two modes. Experimental results on the LibriSpeech dataset show that UFO2 outperforms the SSL-based baseline method by 29.7% and 18.2% relative WER reduction in offline and online modes, respectively.

翻译：本文提出了一种面向在线与离线模式的统一预训练框架（UFO2，Unified pre-training Framework for Online and Offline）自动语音识别方法。该方法实现了两点创新：1）将在线与离线两种模式原本分离的训练流程简化为单一流程；2）在有限话语标注条件下提升了词错误率（WER）性能。具体而言，我们将传统的基于自监督学习（SSL）的离线模式ASR方法拓展为统一范式，使模型训练同时依赖于全上下文输入和动态分块输入。为增强预训练表征模型，我们采用停止梯度操作将在线模式目标函数与量化器解耦。此外，在预训练和下游微调阶段均采用联合损失函数，通过全权重共享机制训练两种模式的统一模型。在LibriSpeech数据集上的实验结果表明，UFO2相较于基于SSL的基线方法，在离线模式和在线模式下分别实现了29.7%和18.2%的相对词错误率降低。