Despite the recent progress in speech emotion recognition (SER), state-of-the-art systems are unable to achieve improved performance in cross-language settings. In this paper, we propose a Multimodal Dual Attention Transformer (MDAT) model to improve cross-language SER. Our model utilises pre-trained models for multimodal feature extraction and is equipped with a dual attention mechanism including graph attention and co-attention to capture complex dependencies across different modalities and achieve improved cross-language SER results using minimal target language data. In addition, our model also exploits a transformer encoder layer for high-level feature representation to improve emotion classification accuracy. In this way, MDAT performs refinement of feature representation at various stages and provides emotional salient features to the classification layer. This novel approach also ensures the preservation of modality-specific emotional information while enhancing cross-modality and cross-language interactions. We assess our model's performance on four publicly available SER datasets and establish its superior effectiveness compared to recent approaches and baseline models.

翻译：尽管语音情感识别（SER）近期取得了进展，但最先进的系统仍无法在跨语言场景下实现性能提升。本文提出一种多模态双注意力 Transformer（MDAT）模型以改进跨语言 SER。该模型利用预训练模型进行多模态特征提取，并配备包括图注意力和协同注意力在内的双注意力机制，以捕捉不同模态间的复杂依赖关系，并利用最少的目标语言数据实现更优的跨语言 SER 结果。此外，模型还利用 Transformer 编码器层进行高层特征表示，以提高情感分类精度。通过这种方式，MDAT 在不同阶段对特征表示进行精炼，向分类层提供情感显著特征。这种新颖方法在增强跨模态和跨语言交互的同时，也确保了模态特定情感信息的保留。我们在四个公开的 SER 数据集上评估模型性能，证明其相较于近期方法和基线模型具有更优的效果。