Existing emotion prediction benchmarks contain coarse emotion labels which do not consider the diversity of emotions that an image and text can elicit in humans due to various reasons. Learning diverse reactions to multimodal content is important as intelligent machines take a central role in generating and delivering content to society. To address this gap, we propose Socratis, a societal reactions benchmark, where each image-caption (IC) pair is annotated with multiple emotions and the reasons for feeling them. Socratis contains 18K free-form reactions for 980 emotions on 2075 image-caption pairs from 5 widely-read news and image-caption (IC) datasets. We benchmark the capability of state-of-the-art multimodal large language models to generate the reasons for feeling an emotion given an IC pair. Based on a preliminary human study, we observe that humans prefer human-written reasons over 2 times more often than machine-generated ones. This shows our task is harder than standard generation tasks because it starkly contrasts recent findings where humans cannot tell apart machine vs human-written news articles, for instance. We further see that current captioning metrics based on large vision-language models also fail to correlate with human preferences. We hope that these findings and our benchmark will inspire further research on training emotionally aware models.

翻译：现有情感预测基准数据集中包含粗粒度的情感标签，未考虑图像与文本因多种因素在人类中引发的情感多样性。随着智能机器在内容生成与传播中扮演核心角色，理解多模态内容引发的多样化反应至关重要。为此，我们提出社会反应基准数据集Socratis，其每一对图像-文本(IC)标注了多种情感及产生相应情感的原因。该数据集包含来自5个主流新闻与IC数据集的2075对图像-文本，涵盖980种情感的18,000条自由形式反应。我们评估了最先进的多模态大语言模型在给定IC对时生成情感原因的能力。初步人类研究表明，人类对人类撰写原因的偏好程度是机器生成原因的2倍以上。这表明该任务比标准生成任务更具挑战性——例如与最新研究发现（人类无法区分机器撰写与人类撰写的新闻文章）形成鲜明对比。进一步发现，基于大型视觉语言模型的现有描述评价指标也无法与人类偏好保持一致。期待本研究发现与基准数据集能推动情感感知模型的进一步研究。