In large part due to their implicit semantic modeling, self-supervised learning (SSL) methods have significantly increased the performance of valence recognition in speech emotion recognition (SER) systems. Yet, their large size may often hinder practical implementations. In this work, we take HuBERT as an example of an SSL model and analyze the relevance of each of its layers for SER. We show that shallow layers are more important for arousal recognition while deeper layers are more important for valence. This observation motivates the importance of additional textual information for accurate valence recognition, as the distilled framework lacks the depth of its large-scale SSL teacher. Thus, we propose an audio-textual distilled SSL framework that, while having only ~20% of the trainable parameters of a large SSL model, achieves on par performance across the three emotion dimensions (arousal, valence, dominance) on the MSP-Podcast v1.10 dataset.

翻译：由于隐式语义建模的特性，自监督学习方法显著提升了语音情感识别系统中效价识别的性能。然而，其庞大的模型体积往往阻碍实际应用落地。本研究以HuBERT作为自监督学习模型的范例，分析其各层对语音情感识别的相关程度。研究表明，浅层网络对唤醒度识别更为关键，而深层网络对效价识别更重要。这一发现揭示了蒸馏框架因缺乏大规模自监督教师模型的深度，使得额外文本信息对准确效价识别具有重要价值。为此，我们提出一种音频-文本蒸馏自监督框架，该框架仅需大型自监督模型约20%的可训练参数，便能在MSP-Podcast v1.10数据集的情感三维度（唤醒度、效价、支配度）上达到同等性能水平。