Recent self-supervised learning (SSL) models have proven to learn rich representations of speech, which can readily be utilized by diverse downstream tasks. To understand such utilities, various analyses have been done for speech SSL models to reveal which and how information is encoded in the learned representations. Although the scope of previous analyses is extensive in acoustic, phonetic, and semantic perspectives, the physical grounding by speech production has not yet received full attention. To bridge this gap, we conduct a comprehensive analysis to link speech representations to articulatory trajectories measured by electromagnetic articulography (EMA). Our analysis is based on a linear probing approach where we measure articulatory score as an average correlation of linear mapping to EMA. We analyze a set of SSL models selected from the leaderboard of the SUPERB benchmark and perform further layer-wise analyses on two most successful models, Wav2Vec 2.0 and HuBERT. Surprisingly, representations from the recent speech SSL models are highly correlated with EMA traces (best: r = 0.81), and only 5 minutes are sufficient to train a linear model with high performance (r = 0.77). Our findings suggest that SSL models learn to align closely with continuous articulations, and provide a novel insight into speech SSL.

翻译：近期自监督学习（SSL）模型已证明能够学习丰富的语音表征，并可便捷地应用于多种下游任务。为理解此类实用性，研究者对语音SSL模型开展了多维度分析，旨在揭示学习表征中所编码的信息类型及其编码方式。尽管先前分析在声学、音系学和语义学视角上已较为全面，但基于语音产生的物理机制尚未得到充分关注。为弥补这一空白，我们开展了一项综合分析，将语音表征与电磁发音仪（EMA）测量的构音轨迹相关联。本分析基于线性探测方法，通过计算线性映射与EMA的平均相关系数来衡量构音分数。我们选取了SUPERB基准排行榜中的一组SSL模型进行分析，并对两个最成功的模型Wav2Vec 2.0和HuBERT进行了进一步的逐层分析。令人意外的是，当前语音SSL模型的表征与EMA轨迹高度相关（最优值：r=0.81），且仅需5分钟训练数据即可获得高性能线性模型（r=0.77）。我们的发现表明，SSL模型能够紧密对齐连续构音过程，并为语音SSL研究提供了全新视角。