Recent developments in pre-trained speech representation utilizing self-supervised learning (SSL) have yielded exceptional results on a variety of downstream tasks. One such technique, known as masked predictive coding (MPC), has been employed by some of the most high-performing models. In this study, we investigate the impact of MPC loss on the type of information learnt at various layers in the HuBERT model, using nine probing tasks. Our findings indicate that the amount of content information learned at various layers of the HuBERT model has a positive correlation to the MPC loss. Additionally, it is also observed that any speaker-related information learned at intermediate layers of the model, is an indirect consequence of the learning process, and therefore cannot be controlled using the MPC loss. These findings may serve as inspiration for further research in the speech community, specifically in the development of new pre-training tasks or the exploration of new pre-training criterion's that directly preserves both speaker and content information at various layers of a learnt model.

翻译：近期，利用自监督学习（SSL）的预训练语音表征技术在下游任务中取得了显著成果。其中一种被称为遮蔽预测编码（MPC）的技术已被部分高性能模型采用。本研究通过九项探测任务，探究了MPC损失对HuBERT模型各层学习信息类型的影响。结果表明，HuBERT模型各层学习到的内容信息量与MPC损失呈正相关。此外，研究还发现，模型中间层学习到的任何说话人相关信息都是学习过程的间接结果，因此无法通过MPC损失加以控制。这些发现或可为语音领域的研究提供启示，尤其是在设计新的预训练任务或探索既能直接保留学习模型各层说话人信息又能保留内容信息的新预训练准则方面。