The process of human affect understanding involves the ability to infer person specific emotional states from various sources including images, speech, and language. Affect perception from images has predominantly focused on expressions extracted from salient face crops. However, emotions perceived by humans rely on multiple contextual cues including social settings, foreground interactions, and ambient visual scenes. In this work, we leverage pretrained vision-language (VLN) models to extract descriptions of foreground context from images. Further, we propose a multimodal context fusion (MCF) module to combine foreground cues with the visual scene and person-based contextual information for emotion prediction. We show the effectiveness of our proposed modular design on two datasets associated with natural scenes and TV shows.

翻译：人类情感理解过程涉及从图像、语音和语言等多种来源推断个体特定情感状态的能力。基于图像的情感感知主要集中在从显著人脸区域提取的表情信息上。然而，人类所感知的情感依赖于多种上下文线索，包括社交环境、前景交互以及周围视觉场景。本研究中，我们利用预训练的视觉-语言模型（VLN）从图像中提取前景上下文的描述信息。进一步，我们提出了一种多模态上下文融合（MCF）模块，将前景线索与视觉场景及人物相关上下文信息相结合，用于情感预测。我们在两个分别包含自然场景和电视节目的数据集上验证了所提出模块化设计的有效性。