Whisper is one of the recent state-of-the-art multilingual speech recognition and translation models, however, it is not designed for real time transcription. In this paper, we build on top of Whisper and create Whisper-Streaming, an implementation of real-time speech transcription and translation of Whisper-like models. Whisper-Streaming uses local agreement policy with self-adaptive latency to enable streaming transcription. We show that Whisper-Streaming achieves high quality and 3.3 seconds latency on unsegmented long-form speech transcription test set, and we demonstrate its robustness and practical usability as a component in live transcription service at a multilingual conference.

翻译：Whisper是近期最先进的多语言语音识别与翻译模型之一，但其并非为实时转录而设计。本文在Whisper基础上构建了Whisper-Streaming，实现了类Whisper模型的实时语音转录与翻译功能。Whisper-Streaming采用自适应延迟的局部一致性策略，支持流式转录。实验表明，Whisper-Streaming在未分割的长语音转录测试集上实现了高质量转录与3.3秒延迟，同时证明了其作为多语言会议直播转录服务组件的鲁棒性与实用价值。