Self-supervised learning (SSL) speech models such as wav2vec and HuBERT have demonstrated state-of-the-art performance on automatic speech recognition (ASR) and proved to be extremely useful in low label-resource settings. However, the success of SSL models has yet to transfer to utterance-level tasks such as speaker, emotion, and language recognition, which still require supervised fine-tuning of the SSL models to obtain good performance. We argue that the problem is caused by the lack of disentangled representations and an utterance-level learning objective for these tasks. Inspired by how HuBERT uses clustering to discover hidden acoustic units, we formulate a factor analysis (FA) model that uses the discovered hidden acoustic units to align the SSL features. The underlying utterance-level representations are disentangled from the content of speech using probabilistic inference on the aligned features. Furthermore, the variational lower bound derived from the FA model provides an utterance-level objective, allowing error gradients to be backpropagated to the Transformer layers to learn highly discriminative acoustic units. When used in conjunction with HuBERT's masked prediction training, our models outperform the current best model, WavLM, on all utterance-level non-semantic tasks on the SUPERB benchmark with only 20% of labeled data.

翻译：自监督学习（SSL）语音模型（如wav2vec和HuBERT）在自动语音识别（ASR）任务中展现出最先进性能，并在低标签资源场景下被证明极为有效。然而，SSL模型的成功尚未迁移至说话人识别、情感识别和语种识别等语句级任务，这些任务仍需对SSL模型进行监督微调才能获得良好性能。我们认为，问题源于缺乏解缠表征及针对这些任务的语句级学习目标。受HuBERT通过聚类发现隐式声学单元的启发，我们构建了一个因子分析（FA）模型，利用所发现的隐式声学单元对齐SSL特征。通过对齐特征进行概率推断，底层语句级表征得以从语音内容中解缠。此外，由FA模型推导出的变分下界提供了语句级目标，使得误差梯度可反向传播至Transformer层以学习高判别力的声学单元。当与HuBERT的掩码预测训练结合使用时，我们的模型在SUPERB基准的所有语句级非语义任务上仅需20%标注数据即可超越当前最优模型WavLM。