Multimodal Emotion Recognition in Conversation (ERC) plays an influential role in the field of human-computer interaction and conversational robotics since it can motivate machines to provide empathetic services. Multimodal data modeling is an up-and-coming research area in recent years, which is inspired by human capability to integrate multiple senses. Several graph-based approaches claim to capture interactive information between modalities, but the heterogeneity of multimodal data makes these methods prohibit optimal solutions. In this work, we introduce a multimodal fusion approach named Graph and Attention based Two-stage Multi-source Information Fusion (GA2MIF) for emotion detection in conversation. Our proposed method circumvents the problem of taking heterogeneous graph as input to the model while eliminating complex redundant connections in the construction of graph. GA2MIF focuses on contextual modeling and cross-modal modeling through leveraging Multi-head Directed Graph ATtention networks (MDGATs) and Multi-head Pairwise Cross-modal ATtention networks (MPCATs), respectively. Extensive experiments on two public datasets (i.e., IEMOCAP and MELD) demonstrate that the proposed GA2MIF has the capacity to validly capture intra-modal long-range contextual information and inter-modal complementary information, as well as outperforms the prevalent State-Of-The-Art (SOTA) models by a remarkable margin.

翻译：多模态对话情感识别（ERC）在人机交互与对话机器人领域具有重要作用，因其能促使机器提供共情服务。受人类整合多种感官能力的启发，多模态数据建模成为近年来新兴的研究方向。尽管基于图的方法声称能够捕捉模态间的交互信息，但多模态数据的异质性使得这些方法难以获得最优解。本文提出了一种名为“基于图与注意力的两阶段多源信息融合”（GA2MIF）的多模态融合方法，用于对话情感检测。所提方法规避了将异构图作为模型输入的问题，同时消除了图构建过程中复杂的冗余连接。GA2MIF通过分别利用多头有向图注意力网络（MDGATs）与多头成对跨模态注意力网络（MPCATs），聚焦于上下文建模与跨模态建模。在两个公开数据集（即IEMOCAP与MELD）上的大量实验表明，所提出的GA2MIF能够有效捕获模态内长程上下文信息与模态间互补信息，并以显著优势优于当前主流的最优（SOTA）模型。