Keyword Spotting (KWS) models are becoming increasingly integrated into various systems, e.g. voice assistants. To achieve satisfactory performance, these models typically rely on a large amount of labelled data, limiting their applications only to situations where such data is available. Self-supervised Learning (SSL) methods can mitigate such a reliance by leveraging readily-available unlabelled data. Most SSL methods for speech have primarily been studied for large models, whereas this is not ideal, as compact KWS models are generally required. This paper explores the effectiveness of SSL on small models for KWS and establishes that SSL can enhance the performance of small KWS models when labelled data is scarce. We pretrain three compact transformer-based KWS models using Data2Vec, and fine-tune them on a label-deficient setup of the Google Speech Commands data set. It is found that Data2Vec pretraining leads to a significant increase in accuracy, with label-deficient scenarios showing an improvement of 8.22% 11.18% absolute accuracy.

翻译：关键词识别（KWS）模型正越来越多地集成到各类系统中，例如语音助手。为实现令人满意的性能，这些模型通常依赖大量标注数据，这限制了其仅在数据可用的场景中应用。自监督学习（SSL）方法可通过利用易获取的未标注数据来缓解这种依赖。大多数面向语音的自监督学习方法主要针对大型模型进行研究，但这并不理想，因为通常需要紧凑型KWS模型。本文探讨了自监督学习在小模型上的有效性，并证实当标注数据稀缺时，自监督学习可提升小型KWS模型的性能。我们使用Data2Vec对三个基于Transformer的紧凑型KWS模型进行预训练，并在Google Speech Commands数据集的标注不足配置下进行微调。研究发现，Data2Vec预训练显著提升了准确率，在标注不足的场景中绝对准确率提高了8.22%-11.18%。