Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models.

翻译：近年来，研究者提出了多种自监督语音模型，它们在预训练目标、输入模态和预训练数据上各有不同。尽管在下游任务中取得了显著成功，但我们对其编码的属性以及模型间的差异仍了解有限。本研究针对多种近期模型的中间表征进行了分析。具体而言，我们采用基于典型相关分析（CCA）的轻量级分析工具，测量各层编码的声学、音素和词汇级别属性。研究发现，这些属性在不同层级间的演化方式因模型而异，且变化与预训练目标的选择相关。我们进一步通过对比属性变化趋势与语音识别及口语理解任务的表现，探讨了本分析对下游任务的实用价值。结果表明，CCA趋势可为下游任务提供可靠的层级选择依据，且单层性能常能达到或超过使用全部层级的效果，这表明预训练模型的使用效率具有提升空间。