Emotion recognition in conversations is challenging due to the multi-modal nature of the emotion expression. We propose a hierarchical cross-attention model (HCAM) approach to multi-modal emotion recognition using a combination of recurrent and co-attention neural network models. The input to the model consists of two modalities, i) audio data, processed through a learnable wav2vec approach and, ii) text data represented using a bidirectional encoder representations from transformers (BERT) model. The audio and text representations are processed using a set of bi-directional recurrent neural network layers with self-attention that converts each utterance in a given conversation to a fixed dimensional embedding. In order to incorporate contextual knowledge and the information across the two modalities, the audio and text embeddings are combined using a co-attention layer that attempts to weigh the utterance level embeddings relevant to the task of emotion recognition. The neural network parameters in the audio layers, text layers as well as the multi-modal co-attention layers, are hierarchically trained for the emotion classification task. We perform experiments on three established datasets namely, IEMOCAP, MELD and CMU-MOSI, where we illustrate that the proposed model improves significantly over other benchmarks and helps achieve state-of-art results on all these datasets.

翻译：对话中的情感识别因情感表达的多模态特性而颇具挑战。我们提出一种基于层次化交叉注意力模型（HCAM）的多模态情感识别方法，该方法融合了循环神经网络与共注意力神经网络模型。模型输入包含两种模态：i) 音频数据，通过可学习的wav2vec方法进行处理；ii) 文本数据，采用基于变换器的双向编码器表示（BERT）模型进行表征。音频与文本表征经由一组带有自注意力的双向循环神经网络层处理，将给定对话中的每一句话转换为固定维度的嵌入向量。为整合跨模态的上下文知识与信息，音频与文本嵌入通过共注意力层进行融合，该层旨在对与情感识别任务相关的语句级嵌入赋予权重。音频层、文本层以及多模态共注意力层中的神经网络参数均被层次化地训练以完成情感分类任务。我们在IEMOCAP、MELD与CMU-MOSI三个既有数据集上进行实验，结果表明所提方法相较其他基准模型有显著提升，并在所有数据集上均取得了最先进的性能。