Recently Whisper has approached human-level robustness and accuracy in English automatic speech recognition (ASR), while in minor language and mixed language speech recognition, there remains a compelling need for further improvement. In this work, we present the impressive results of Whisper-MCE, our finetuned Whisper model, which was trained using our self-collected dataset, Mixed Cantonese and English audio dataset (MCE). Meanwhile, considering word error rate (WER) poses challenges when it comes to evaluating its effectiveness in minor language and mixed-language contexts, we present a novel rating mechanism. By comparing our model to the baseline whisper-large-v2 model, we demonstrate its superior ability to accurately capture the content of the original audio, achieve higher recognition accuracy, and exhibit faster recognition speed. Notably, our model outperforms other existing models in the specific task of recognizing mixed language.

翻译：近期Whisper模型在英语自动语音识别（ASR）任务中已接近人类水平的鲁棒性和准确性，但在小语种及混合语言语音识别领域仍存在显著的改进需求。本文展示了我们微调后的Whisper模型——Whisper-MCE的优异表现，该模型使用自主构建的粤语-英语混合音频数据集（MCE）进行训练。同时，针对词错误率（WER）在评估小语种及混合语言场景下有效性时存在的局限性，我们提出了一种新型评分机制。通过与基线模型whisper-large-v2的对比实验表明，我们的模型在精准捕捉原始音频内容、实现更高识别准确率以及展现更快识别速度方面均展现出优越性能。特别值得注意的是，在混合语言识别这一特定任务中，我们的模型显著优于现有其他模型。