Pre-training speech models on large volumes of data has achieved remarkable success. OpenAI Whisper is a multilingual multitask model trained on 680k hours of supervised speech data. It generalizes well to various speech recognition and translation benchmarks even in a zero-shot setup. However, the full pipeline for developing such models (from data collection to training) is not publicly accessible, which makes it difficult for researchers to further improve its performance and address training-related issues such as efficiency, robustness, fairness, and bias. This work presents an Open Whisper-style Speech Model (OWSM), which reproduces Whisper-style training using an open-source toolkit and publicly available data. OWSM even supports more translation directions and can be more efficient to train. We will publicly release all scripts used for data preparation, training, inference, and scoring as well as pre-trained models and training logs to promote open science.

翻译：基于大规模数据预训练的语音模型已取得显著成功。OpenAI Whisper是一个在多语言多任务场景下训练的模型，其训练数据包含68万小时的监督语音数据。即使在零样本设置下，该模型也能很好地泛化至各类语音识别和翻译基准测试。然而，此类模型从数据采集到训练的完整开发流程并未公开，这使得研究人员难以进一步优化其性能，也无法解决与训练相关的效率、鲁棒性、公平性和偏差等问题。本研究提出开放Whisper风格的语音模型（OWSM），通过开源工具包和公开可用数据复现Whisper风格训练。OWSM不仅支持更多翻译方向，还能实现更高效的训练。为促进开放科学，我们将公开发布用于数据准备、训练、推理和评分的所有脚本，以及预训练模型和训练日志。