This paper presents a paradigm that adapts general large-scale pretrained models (PTMs) to speech emotion recognition task. Although PTMs shed new light on artificial general intelligence, they are constructed with general tasks in mind, and thus, their efficacy for specific tasks can be further improved. Additionally, employing PTMs in practical applications can be challenging due to their considerable size. Above limitations spawn another research direction, namely, optimizing large-scale PTMs for specific tasks to generate task-specific PTMs that are both compact and effective. In this paper, we focus on the speech emotion recognition task and propose an improved emotion-specific pretrained encoder called Vesper. Vesper is pretrained on a speech dataset based on WavLM and takes into account emotional characteristics. To enhance sensitivity to emotional information, Vesper employs an emotion-guided masking strategy to identify the regions that need masking. Subsequently, Vesper employs hierarchical and cross-layer self-supervision to improve its ability to capture acoustic and semantic representations, both of which are crucial for emotion recognition. Experimental results on the IEMOCAP, MELD, and CREMA-D datasets demonstrate that Vesper with 4 layers outperforms WavLM Base with 12 layers, and the performance of Vesper with 12 layers surpasses that of WavLM Large with 24 layers.

翻译：摘要：本文提出了一种将通用大规模预训练模型（PTMs）适配至语音情感识别任务的范式。尽管PTMs为人工智能通用领域带来了新曙光，但其构建初衷多面向通用任务，因此在特定任务中的效能仍有提升空间。此外，由于PTMs的庞大体积，将其应用于实际场景存在挑战。上述局限催生了另一研究方向——针对特定任务优化大规模PTMs，以生成兼具紧凑性与高效性的任务专用预训练模型。本文聚焦于语音情感识别任务，提出了一种改进的情感专用预训练编码器Vesper。该模型基于WavLM在语音数据集上进行预训练，充分考虑了情感特征。为增强对情感信息的敏感性，Vesper采用情感引导掩码策略识别需掩码的区域，并通过分层与跨层自监督机制提升其捕捉声学与语义表征的能力——二者对情感识别均至关重要。在IEMOCAP、MELD和CREMA-D数据集上的实验表明，4层Vesper性能超越12层WavLM Base，而12层Vesper的表现优于24层WavLM Large。