Memes can sway people's opinions over social media as they combine visual and textual information in an easy-to-consume manner. Since memes instantly turn viral, it becomes crucial to infer their intent and potentially associated harmfulness to take timely measures as needed. A common problem associated with meme comprehension lies in detecting the entities referenced and characterizing the role of each of these entities. Here, we aim to understand whether the meme glorifies, vilifies, or victimizes each entity it refers to. To this end, we address the task of role identification of entities in harmful memes, i.e., detecting who is the 'hero', the 'villain', and the 'victim' in the meme, if any. We utilize HVVMemes - a memes dataset on US Politics and Covid-19 memes, released recently as part of the CONSTRAINT@ACL-2022 shared-task. It contains memes, entities referenced, and their associated roles: hero, villain, victim, and other. We further design VECTOR (Visual-semantic role dEteCToR), a robust multi-modal framework for the task, which integrates entity-based contextual information in the multi-modal representation and compare it to several standard unimodal (text-only or image-only) or multi-modal (image+text) models. Our experimental results show that our proposed model achieves an improvement of 4% over the best baseline and 1% over the best competing stand-alone submission from the shared-task. Besides divulging an extensive experimental setup with comparative analyses, we finally highlight the challenges encountered in addressing the complex task of semantic role labeling within memes.

翻译：迷因因其图文并茂、易于消费的特性，能在社交媒体上左右公众舆论。由于迷因会迅速传播，及时推断其意图及潜在的危害性以采取必要措施变得至关重要。理解迷因的常见难点在于检测其所指实体并刻画每个实体的角色。本文旨在明确迷因对其所指实体是颂扬、诋毁还是伤害。为此，我们专注于有害迷因中的实体角色识别任务，即检测迷因中的“英雄”、“反派”与“受害者”（若存在）。我们采用HVVMemes数据集——该数据集专研美国政治与新冠疫情迷因，近期作为CONSTRAINT@ACL-2022共享任务的一部分发布。数据包含迷因、所指实体及其对应角色（英雄、反派、受害者及其他）。我们进一步设计了VECTOR（视觉语义角色检测器），一个针对该任务的鲁棒多模态框架。该框架在多模态表示中整合了基于实体的上下文信息，并与若干标准单模态（仅文本或仅图像）及多模态（图像+文本）模型进行了对比。实验结果表明，我们提出的模型相比最优基线提升了4%，相比共享任务中最佳独立提交方案提升了1%。除披露包含比较分析的广泛实验设置外，我们最后强调了解决迷因中复杂语义角色标注任务所面临的挑战。