There has been an increasing interest in large speech models that can perform multiple speech processing tasks in a single model. Such models usually adopt the encoder-decoder or decoder-only architecture due to their popularity and good performance in many domains. However, autoregressive models can be slower during inference compared to non-autoregressive models and also have potential risks of hallucination. Though prior studies observed promising results of non-autoregressive models for certain tasks at small scales, it remains unclear if they can be scaled to speech-to-text generation in diverse languages and tasks. Inspired by the Open Whisper-style Speech Model (OWSM) project, we propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC). It is trained on 180k hours of public audio data for multilingual automatic speech recognition (ASR), speech translation (ST), and language identification (LID). Compared to encoder-decoder OWSM, our OWSM-CTC achieves competitive results on ASR and up to 25% relative improvement on ST, while it is more robust and 3 to 4 times faster for inference. OWSM-CTC also improves the long-form ASR result with 20x speed-up. We will publicly release our codebase, pre-trained model, and training logs to promote open science in speech foundation models.

翻译：近年来，能够以单一模型执行多种语音处理任务的大规模语音模型日益受到关注。这类模型通常采用编码器-解码器或仅解码器架构，因其在众多领域具有较高流行度和出色性能。然而，自回归模型在推理时速度较非自回归模型更慢，且存在潜在的幻觉风险。尽管先前研究表明非自回归模型在特定任务的小规模应用中展现出良好前景，但其能否扩展至多语言多任务的语音到文本生成场景仍不明确。受开放式Whisper风格语音模型（OWSM）项目启发，我们提出基于连接主义时序分类（CTC）的新型仅编码器语音基础模型OWSM-CTC。该模型利用18万小时公共音频数据训练，可完成多语种自动语音识别（ASR）、语音翻译（ST）及语言识别（LID）任务。与编码器-解码器架构的OWSM相比，OWSM-CTC在ASR任务上取得相当的成果，在ST任务上实现高达25%的相对性能提升，同时具备更强鲁棒性且推理速度提升3至4倍。此外，OWSM-CTC在长语音ASR任务中实现20倍加速。我们将公开发布代码库、预训练模型及训练日志，以推动语音基础模型领域的开放科学。