Multimodal Emotion Recognition in Conversation (ERC) has garnered growing attention from research communities in various fields. In this paper, we propose a cross-modal fusion network with emotion-shift awareness (CFN-ESA) for ERC. Extant approaches employ each modality equally without distinguishing the amount of emotional information, rendering it hard to adequately extract complementary and associative information from multimodal data. To cope with this problem, in CFN-ESA, textual modalities are treated as the primary source of emotional information, while visual and acoustic modalities are taken as the secondary sources. Besides, most multimodal ERC models ignore emotion-shift information and overfocus on contextual information, leading to the failure of emotion recognition under emotion-shift scenario. We elaborate an emotion-shift module to address this challenge. CFN-ESA mainly consists of the unimodal encoder (RUME), cross-modal encoder (ACME), and emotion-shift module (LESM). RUME is applied to extract conversation-level contextual emotional cues while pulling together the data distributions between modalities; ACME is utilized to perform multimodal interaction centered on textual modality; LESM is used to model emotion shift and capture related information, thereby guide the learning of the main task. Experimental results demonstrate that CFN-ESA can effectively promote performance for ERC and remarkably outperform the state-of-the-art models.

翻译：多模态对话情绪识别（Multimodal Emotion Recognition in Conversation, ERC）已引起各领域研究团体的日益关注。本文提出一种面向ERC的跨模态融合网络与情感转变感知方法（CFN-ESA）。现有方法对各模态平等对待，未区分情绪信息量的差异，难以充分提取多模态数据中的互补与关联信息。为解决此问题，CFN-ESA将文本模态作为情绪信息的主要来源，视觉与声学模态作为次要来源。此外，多数多模态ERC模型忽视情感转变信息，过度聚焦上下文信息，导致在情感转变场景下情绪识别失效。我们设计了一个情感转变模块来应对这一挑战。CFN-ESA主要由单模态编码器（RUME）、跨模态编码器（ACME）和情感转变模块（LESM）构成。RUME用于提取对话级上下文情绪线索，同时拉近模态间的数据分布；ACME用于执行以文本模态为中心的多模态交互；LESM用于建模情感转变并捕获相关信息，从而引导主任务学习。实验结果表明，CFN-ESA可有效提升ERC性能，并显著优于现有最先进模型。