There has been an encouraging progress in the affective states recognition models based on the single-modality signals as electroencephalogram (EEG) signals or peripheral physiological signals in recent years. However, multimodal physiological signals-based affective states recognition methods have not been thoroughly exploited yet. Here we propose Multiscale Convolutional Neural Networks (Multiscale CNNs) and a biologically inspired decision fusion model for multimodal affective states recognition. Firstly, the raw signals are pre-processed with baseline signals. Then, the High Scale CNN and Low Scale CNN in Multiscale CNNs are utilized to predict the probability of affective states output for EEG and each peripheral physiological signal respectively. Finally, the fusion model calculates the reliability of each single-modality signals by the Euclidean distance between various class labels and the classification probability from Multiscale CNNs, and the decision is made by the more reliable modality information while other modalities information is retained. We use this model to classify four affective states from the arousal valence plane in the DEAP and AMIGOS dataset. The results show that the fusion model improves the accuracy of affective states recognition significantly compared with the result on single-modality signals, and the recognition accuracy of the fusion result achieve 98.52% and 99.89% in the DEAP and AMIGOS dataset respectively.

翻译：近年来，基于单模态信号（如脑电图信号或外周生理信号）的情感状态识别模型取得了令人鼓舞的进展。然而，基于多模态生理信号的情感状态识别方法尚未得到充分开发。本文提出了一种多尺度卷积神经网络（Multiscale CNNs）及生物启发的决策融合模型，用于多模态情感状态识别。首先，使用基线信号对原始信号进行预处理。随后，利用多尺度CNN中的高尺度CNN和低尺度CNN分别对脑电图信号及各外周生理信号的情感状态输出概率进行预测。最终，融合模型通过计算各类别标签与多尺度CNN分类概率之间的欧氏距离，评估各单模态信号的可靠性，并基于更可靠的模态信息做出决策，同时保留其他模态的信息。我们使用该模型对DEAP和AMIGOS数据集中唤醒-效价平面的四种情感状态进行分类。结果表明，与单模态信号结果相比，融合模型显著提高了情感状态识别的准确性，在DEAP和AMIGOS数据集上的融合结果识别准确率分别达到98.52%和99.89%。