Memes can sway people's opinions over social media as they combine visual and textual information in an easy-to-consume manner. Since memes instantly turn viral, it becomes crucial to infer their intent and potentially associated harmfulness to take timely measures as needed. A common problem associated with meme comprehension lies in detecting the entities referenced and characterizing the role of each of these entities. Here, we aim to understand whether the meme glorifies, vilifies, or victimizes each entity it refers to. To this end, we address the task of role identification of entities in harmful memes, i.e., detecting who is the 'hero', the 'villain', and the 'victim' in the meme, if any. We utilize HVVMemes - a memes dataset on US Politics and Covid-19 memes, released recently as part of the CONSTRAINT@ACL-2022 shared-task. It contains memes, entities referenced, and their associated roles: hero, villain, victim, and other. We further design VECTOR (Visual-semantic role dEteCToR), a robust multi-modal framework for the task, which integrates entity-based contextual information in the multi-modal representation and compare it to several standard unimodal (text-only or image-only) or multi-modal (image+text) models. Our experimental results show that our proposed model achieves an improvement of 4% over the best baseline and 1% over the best competing stand-alone submission from the shared-task. Besides divulging an extensive experimental setup with comparative analyses, we finally highlight the challenges encountered in addressing the complex task of semantic role labeling within memes.

翻译：迷因通过图文结合的易传播方式，能在社交媒体上影响公众观点。由于迷因极易引发病毒式传播，及时识别其意图及潜在的危害性对于采取必要措施至关重要。迷因理解中的一个常见难题在于检测其所指实体并刻画每个实体的角色。本研究旨在探究迷因对其所指涉的每个实体是否进行颂扬、污名化或加害。为此，我们聚焦有害迷因中的角色识别任务，即检测迷因中是否存在"英雄""反派"与"受害者"角色（如有）。我们采用HVVMemes数据集——该数据集近期作为CONSTRAINT@ACL-2022共享任务的一部分发布，涵盖美国政治与新冠疫情相关迷因，包含迷因图像、所指实体及其对应角色（英雄、反派、受害者及其他）。我们进一步设计VECTOR（视觉语义角色检测器），这是一个针对该任务的鲁棒多模态框架，通过将基于实体的上下文信息融入多模态表征，并将其与多个标准单模态（纯文本或纯图像）或多模态（图像+文本）模型进行对比。实验结果表明，我们的模型相较最佳基线提升4%，相较共享任务中最佳独立参赛方案提升1%。除披露包含比较分析的全面实验设置外，我们最终揭示了在解决迷因中语义角色标注这一复杂任务时面临的挑战。