Understanding individual, group and event level emotions along with contextual information is crucial for analyzing a multi-person social situation. To achieve this, we frame emotion comprehension as the task of predicting fine-grained individual emotion to coarse grained group and event level emotion. We introduce GEMS that leverages a multimodal swin-transformer and S3Attention based architecture, which processes an input scene, group members, and context information to generate joint predictions. Existing multi-person emotion related benchmarks mainly focus on atomic interactions primarily based on emotion perception over time and group level. To this end, we extend and propose VGAF-GEMS to provide more fine grained and holistic analysis on top of existing group level annotation of VGAF dataset. GEMS aims to predict basic discrete and continuous emotions (including valence and arousal) as well as individual, group and event level perceived emotions. Our benchmarking effort links individual, group and situational emotional responses holistically. The quantitative and qualitative comparisons with adapted state-of-the-art models demonstrate the effectiveness of GEMS framework on VGAF-GEMS benchmarking. We believe that it will pave the way of further research. The code and data is available at: https://github.com/katariaak579/GEMS

翻译：理解个体、群体及事件层面的情感及其上下文信息对于分析多人社交情境至关重要。为实现这一目标，我们将情感理解构建为从细粒度个体情感预测到粗粒度群体及事件层面情感的任务。本文提出GEMS模型，该模型采用基于多模态Swin-Transformer与S3Attention的架构，通过处理输入场景、群体成员及上下文信息生成联合预测。现有多人情感相关基准数据集主要关注基于时序情感感知与群体层面的原子化交互。为此，我们在VGAF数据集现有群体标注基础上扩展并提出VGAF-GEMS基准，以提供更细粒度与整体性的分析。GEMS旨在预测基础离散/连续情感（包括效价与唤醒度）以及个体、群体与事件层面的感知情感。我们的基准构建工作将个体、群体与情境情感反应进行整体性关联。与改进的先进模型进行的定量与定性比较证明了GEMS框架在VGAF-GEMS基准上的有效性。我们相信这将为后续研究开辟道路。代码与数据详见：https://github.com/katariaak579/GEMS