Emotion recognition is a crucial task for human conversation understanding. It becomes more challenging with the notion of multimodal data, e.g., language, voice, and facial expressions. As a typical solution, the global- and the local context information are exploited to predict the emotional label for every single sentence, i.e., utterance, in the dialogue. Specifically, the global representation could be captured via modeling of cross-modal interactions at the conversation level. The local one is often inferred using the temporal information of speakers or emotional shifts, which neglects vital factors at the utterance level. Additionally, most existing approaches take fused features of multiple modalities in an unified input without leveraging modality-specific representations. Motivating from these problems, we propose the Relational Temporal Graph Neural Network with Auxiliary Cross-Modality Interaction (CORECT), an novel neural network framework that effectively captures conversation-level cross-modality interactions and utterance-level temporal dependencies with the modality-specific manner for conversation understanding. Extensive experiments demonstrate the effectiveness of CORECT via its state-of-the-art results on the IEMOCAP and CMU-MOSEI datasets for the multimodal ERC task Implementation available at: https://github.com/leson502/CORECT\_EMNLP2023

翻译：情绪识别是人际对话理解中的关键任务，而多模态数据（如语言、语音及面部表情）的引入使其更具挑战性。作为典型解决方案，研究者利用全局及局部上下文信息预测对话中每个语句（即话轮）的情感标签。具体而言，全局表征可通过建模对话层面的跨模态交互获得，而局部表征通常借助说话者时序信息或情感偏移进行推断，却忽略了话轮层面的关键因素。此外，现有方法多将多模态特征融合为统一输入，未能利用模态特异性表征。针对上述问题，本文提出基于辅助跨模态交互的关系时序图神经网络（CORECT）——一种以模态特异性方式有效捕捉对话层面跨模态交互与话轮层面时序依赖的新型神经网络框架。在IEMOCAP和CMU-MOSEI数据集上的多模态情绪识别任务实验中，CORECT通过其最先进的性能结果充分证明了有效性。代码地址：https://github.com/leson502/CORECT_EMNLP2023