Though multimodal emotion recognition has achieved significant progress over recent years, the potential of rich synergic relationships across the modalities is not fully exploited. In this paper, we introduce Recursive Joint Cross-Modal Attention (RJCMA) to effectively capture both intra-and inter-modal relationships across audio, visual and text modalities for dimensional emotion recognition. In particular, we compute the attention weights based on cross-correlation between the joint audio-visual-text feature representations and the feature representations of individual modalities to simultaneously capture intra- and inter-modal relationships across the modalities. The attended features of the individual modalities are again fed as input to the fusion model in a recursive mechanism to obtain more refined feature representations. We have also explored Temporal Convolutional Networks (TCNs) to improve the temporal modeling of the feature representations of individual modalities. Extensive experiments are conducted to evaluate the performance of the proposed fusion model on the challenging Affwild2 dataset. By effectively capturing the synergic intra- and inter-modal relationships across audio, visual and text modalities, the proposed fusion model achieves a Concordance Correlation Coefficient (CCC) of 0.585 (0.542) and 0.659 (0.619) for valence and arousal respectively on the validation set (test set). This shows a significant improvement over the baseline of 0.24 (0.211) and 0.20 (0.191) for valence and arousal respectively on the validation set (test set) of the valence-arousal challenge of 6th Affective Behavior Analysis in-the-Wild (ABAW) competition.

翻译：尽管多模态情感识别近年来取得了显著进展，但跨模态间丰富的协同关系潜力尚未被充分挖掘。本文提出循环联合跨模态注意力（RJCMA），以有效捕获音频、视觉和文本模态之间的模态内与跨模态关系，实现维度情感识别。具体而言，我们基于联合音频-视觉-文本特征表示与各单模态特征表示之间的互相关计算注意力权重，从而同步捕获跨模态的模态内与跨模态关系。单模态的注意力特征通过循环机制再次输入融合模型，以获得更精细的特征表示。同时，我们探索了时间卷积网络（TCNs）以增强单模态特征表示的时间建模能力。通过在具有挑战性的Affwild2数据集上进行大量实验评估所提融合模型的性能，该模型有效捕获了音频、视觉和文本模态间的协同模态内与跨模态关系，在验证集（测试集）上分别达到了效价0.585（0.542）和唤醒度0.659（0.619）的一致相关系数（CCC）。这相较于第6届野外情感行为分析（ABAW）竞赛的效价-唤醒度挑战基线（验证集效价0.24（0.211）、唤醒度0.20（0.191））实现了显著提升。