Multimodal emotion recognition is a challenging research area that aims to fuse different modalities to predict human emotion. However, most existing models that are based on attention mechanisms have difficulty in learning emotionally relevant parts on their own. To solve this problem, we propose to incorporate external emotion-related knowledge in the co-attention based fusion of pre-trained models. To effectively incorporate this knowledge, we enhance the co-attention model with a Bayesian attention module (BAM) where a prior distribution is estimated using the emotion-related knowledge. Experimental results on the IEMOCAP dataset show that the proposed approach can outperform several state-of-the-art approaches by at least 0.7% unweighted accuracy (UA).

翻译：多模态情感识别是一个具有挑战性的研究领域，其目标是通过融合不同模态来预测人类情感。然而，现有基于注意力机制的大多数模型难以自主学习与情感相关的特征部分。为解决这一问题，我们提出在基于双向注意力机制的融合过程中引入外部情感相关知识。为有效融合这些知识，我们通过贝叶斯注意力模块（BAM）增强双向注意力模型，利用情感相关知识估计先验分布。在IEMOCAP数据集上的实验结果表明，所提方法在未加权准确率（UA）上至少以0.7%的优势超越多种当前最优方法。