End-to-end automatic speech recognition (ASR) systems often struggle to recognize rare name entities, such as personal names, organizations, and terminologies not frequently encountered in the training data. This paper presents Contextual Biasing Whisper (CB-Whisper), a novel ASR system based on OpenAI's Whisper model that can recognize user-defined name entities by performing open-vocabulary keyword-spotting (OV-KWS) using the hidden states of Whisper encoder. The recognized entities are used as prompts for the Whisper decoder. We first propose a multitask training approach with OV-KWS and ASR tasks to optimize the model. Experiments show that this approach substantially improves the entity recalls compared to the original Whisper model on Chinese Aishell hot word subsets and two internal code-switch test sets. However, we observed a slight increase in mixed-error-rate (MER) on internal test sets due to catastrophic forgetting. To address this problem and use different sizes of the Whisper model without finetuning, we propose to use OV-KWS as a separate module and construct a spoken form prompt to prevent hallucination. The OV-KWS module consistently improves MER and Entity Recall for whisper-small, medium, and large models.

翻译：端到端自动语音识别（ASR）系统在处理训练数据中不常见的命名实体（如人名、组织名称及术语）时往往表现不佳。本文提出上下文偏置Whisper（CB-Whisper），一种基于OpenAI Whisper模型的新型ASR系统，通过利用Whisper编码器的隐藏状态执行开放词汇关键词识别（OV-KWS），可识别用户定义的命名实体，并将识别到的实体作为提示输入至Whisper解码器。我们首先提出一种结合OV-KWS与ASR任务的多任务训练方法以优化模型。实验表明，该方法在中文Aishell热词子集及两个内部代码切换测试集上，相较于原始Whisper模型显著提升了实体召回率。然而我们发现，由于灾难性遗忘，内部测试集的混合错误率（MER）略有上升。为解决该问题并避免对不同规模Whisper模型的微调，我们提出将OV-KWS作为独立模块，并构建口语形式提示以防止幻觉。该OV-KWS模块在whisper-small、medium和large模型上持续改善了MER和实体召回率。