Contrastive learning based cross-modality pretraining methods have recently exhibited impressive success in diverse fields. In this paper, we propose GEmo-CLAP, a kind of gender-attribute-enhanced contrastive language-audio pretraining (CLAP) method for speech emotion recognition. Specifically, a novel emotion CLAP model (Emo-CLAP) is first built, utilizing various self-supervised pre-trained models. Second, considering the importance of gender attribute in speech emotion modeling, the soft label based GEmo-CLAP (SL-GEmo-CLAP) and multi-task learning based GEmo-CLAP (ML-GEmo-CLAP) are further proposed to integrate the emotion and gender information of speech signals, forming more reasonable objectives. Extensive experiments on IEMOCAP show that our proposed two GEmo-CLAP models consistently outperform the baseline Emo-CLAP with different pre-trained models, while also achieving the best recognition performance compared with recent state-of-the-art methods. Noticeably, the proposed WavLM-based ML-GEmo-CLAP obtains the best UAR of 80.16\% and WAR of 82.06\%.

翻译：基于对比学习的跨模态预训练方法近年来在多个领域展现出显著的成功。本文提出了一种用于语音情感识别的性别属性增强对比语言-音频预训练方法——GEmo-CLAP。具体而言，首先构建了一个新型的情感对比语言-音频预训练模型（Emo-CLAP），该模型利用多种自监督预训练模型。其次，考虑到性别属性在语音情感建模中的重要性，进一步提出了基于软标签的GEmo-CLAP（SL-GEmo-CLAP）和基于多任务学习的GEmo-CLAP（ML-GEmo-CLAP），以整合语音信号的语义和性别信息，形成更合理的目标函数。在IEMOCAP数据集上的大量实验表明，我们提出的两种GEmo-CLAP模型始终优于使用不同预训练模型的基线Emo-CLAP，同时与近期最先进的方法相比取得了最佳识别性能。值得注意的是，所提出的基于WavLM的ML-GEmo-CLAP模型获得了80.16%的最佳未加权平均召回率和82.06%的加权平均召回率。