Existing emotion prediction benchmarks contain coarse emotion labels which do not consider the diversity of emotions that an image and text can elicit in humans due to various reasons. Learning diverse reactions to multimodal content is important as intelligent machines take a central role in generating and delivering content to society. To address this gap, we propose Socratis, a \underline{soc}ietal \underline{r}e\underline{a}c\underline{ti}on\underline{s} benchmark, where each image-caption (IC) pair is annotated with multiple emotions and the reasons for feeling them. Socratis contains 18K free-form reactions for 980 emotions on 2075 image-caption pairs from 5 widely-read news and image-caption (IC) datasets. We benchmark the capability of state-of-the-art multimodal large language models to generate the reasons for feeling an emotion given an IC pair. Based on a preliminary human study, we observe that humans prefer human-written reasons over 2 times more often than machine-generated ones. This shows our task is harder than standard generation tasks because it starkly contrasts recent findings where humans cannot tell apart machine vs human-written news articles, for instance. We further see that current captioning metrics based on large vision-language models also fail to correlate with human preferences. We hope that these findings and our benchmark will inspire further research on training emotionally aware models.

翻译：现有情感预测基准数据集包含粗粒度情感标签，未能考虑图像与文本因多重因素引发人类多样化情感反应的现象。学习多模态内容引发的差异化反应至关重要，因为智能机器正逐渐成为社会内容生成与传播的核心载体。为解决这一问题，我们提出Socratis——一个社会性反应基准数据集，其中每个图像-描述（IC）对均标注有多重情感及其产生原因。该数据集从5个广泛使用的新闻与图像描述数据集中采集2075个IC对，涵盖980种情感的18K条自由形式反应。我们评估了当前最先进的多模态大语言模型在给定IC对时生成情感原因的能力。基于初步人类研究，我们发现人类偏好人工撰写原因的次数是机器生成原因的2倍以上。这表明本任务比标准生成任务更具挑战性——例如，与近期研究中人类无法区分机器与人类撰写的新闻文章的现象形成鲜明对比。进一步发现，基于大型视觉-语言模型的现有描述指标也无法与人类偏好相关。我们期待这些发现与基准数据集能推动情感感知模型训练领域的进一步研究。