Self-supervised learning (SSL) speech models, which can serve as powerful upstream models to extract meaningful speech representations, have achieved unprecedented success in speech representation learning. However, their effectiveness on non-speech datasets is relatively less explored. In this work, we propose an ensemble framework, with a combination of ensemble techniques, to fuse SSL speech models' embeddings. Extensive experiments on speech and non-speech audio datasets are conducted to investigate the representation abilities of our ensemble method and its single constituent model. Ablation studies are carried out to evaluate the performances of different ensemble techniques, such as feature averaging and concatenation. All experiments are conducted during NeurIPS 2021 HEAR Challenge as a standard evaluation pipeline provided by competition officials. Results demonstrate SSL speech models' strong abilities on various non-speech tasks, while we also note that they fail to deal with fine-grained music tasks, such as pitch classification and note onset detection. In addition, feature ensemble is shown to have great potential on producing more holistic representations, as our proposed framework generally surpasses state-of-the-art SSL speech/audio models and has superior performance on various datasets compared with other teams in HEAR Challenge. Our code is available at https://github.com/tony10101105/HEAR-2021-NeurIPS-Challenge -- NTU-GURA.

翻译：自监督学习（SSL）语音模型可作为强大的上游模型来提取有意义的语音表征，已在语音表征学习中取得空前成功。然而，这些模型在非语音数据集上的有效性相对研究不足。本研究提出一种集成框架，结合多种集成技术来融合SSL语音模型的嵌入向量。通过在语音与非语音音频数据集上开展大规模实验，考察了本集成方法及其单一组成模型的表征能力。通过消融实验评估了不同集成技术（如特征平均与特征拼接）的性能表现。所有实验均在NeurIPS 2021 HEAR挑战赛的标准评估流程下进行。结果表明SSL语音模型在各类非语音任务中展现出强大能力，但我们也注意到其在音高分类、音符起始检测等精细音乐任务中表现不佳。此外，特征集成在生成更全面表征方面展现出巨大潜力，所提框架整体超越当前最优的SSL语音/音频模型，并在HEAR挑战赛的多个数据集上取得优于其他参赛团队的卓越表现。我们的代码开源在https://github.com/tony10101105/HEAR-2021-NeurIPS-Challenge -- NTU-GURA。