The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting cross-attention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves 2.79% false rejection rate and 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge.

翻译：基于音频模态的关键词识别（KWS）系统，其性能通常以虚警率和误拒率衡量，但在远场和噪声环境下性能显著下降。因此，利用多模态间互补关系的视听关键词识别近期受到广泛关注。然而，当前研究主要集中于融合不同模态的独立学习表征，而非在各自建模过程中探索模态间关系。本文提出一种新颖的视觉模态增强端到端KWS框架（VE-KWS），从两方面融合音频与视觉模态：其一，利用视频中唇部区域获取的说话人位置信息辅助多通道音频波束成形的训练，通过引入波束成形作为音频增强模块，可显著抑制远场或噪声环境造成的声学畸变；其二，在不同模态间进行交叉注意力计算以捕获模态间关系，并辅助各模态的表征学习。在MISP挑战赛语料库上的实验表明，所提模型在Eval集上实现了2.79%的误拒率和2.95%的虚警率，相较于ICASSP2022 MISP挑战赛中的顶级系统取得了新的SOTA性能。