Automatic emotion recognition is a hot topic with a wide range of applications. Much work has been done in the area of automatic emotion recognition in recent years. The focus has been mainly on using the characteristics of a person such as speech, facial expression and pose for this purpose. However, the processing of scene and semantic features for emotion recognition has had limited exploration. In this paper, we propose to use combined scene and semantic features, along with personal features, for multi-modal emotion recognition. Scene features will describe the environment or context in which the target person is operating. The semantic feature can include objects that are present in the environment, as well as their attributes and relationships with the target person. In addition, we use a modified EmbraceNet to extract features from the images, which is trained to learn both the body and pose features simultaneously. By fusing both body and pose features, the EmbraceNet can improve the accuracy and robustness of the model, particularly when dealing with partially missing data. This is because having both body and pose features provides a more complete representation of the subject in the images, which can help the model to make more accurate predictions even when some parts of body are missing. We demonstrate the efficiency of our method on the benchmark EMOTIC dataset. We report an average precision of 40.39\% across the 26 emotion categories, which is a 5\% improvement over previous approaches.

翻译：自动情绪识别是一个具有广泛应用的热门课题。近年来，该领域已有大量研究，主要集中于利用人的语音、面部表情和姿态等个人特征进行情绪识别。然而，场景和语义特征在情绪识别中的处理尚未得到充分探索。本文提出结合场景特征、语义特征和个人特征进行多模态情绪识别。场景特征描述目标人物所处的环境或背景，语义特征则涵盖环境中存在的物体及其属性、以及这些物体与目标人物的关系。此外，我们采用改进的EmbraceNet网络从图像中提取特征，该网络同时训练以学习身体特征和姿态特征。通过融合身体和姿态特征，EmbraceNet能够提升模型的准确性和鲁棒性，特别是在处理部分数据缺失的情况下。这是因为同时拥有身体和姿态特征能为图像中的主体提供更完整的表征，即使人体部分区域缺失，模型也能做出更准确的预测。我们在基准数据集EMOTIC上验证了该方法的高效性。在26个情绪类别上，平均精度达到40.39%，较此前方法提升了5%。