Hate speech is a societal problem that has significantly grown through the Internet. New forms of digital content such as image memes have given rise to spread of hate using multimodal means, being far more difficult to analyse and detect compared to the unimodal case. Accurate automatic processing, analysis and understanding of this kind of content will facilitate the endeavor of hindering hate speech proliferation through the digital world. To this end, we propose MemeFier, a deep learning-based architecture for fine-grained classification of Internet image memes, utilizing a dual-stage modality fusion module. The first fusion stage produces feature vectors containing modality alignment information that captures non-trivial connections between the text and image of a meme. The second fusion stage leverages the power of a Transformer encoder to learn inter-modality correlations at the token level and yield an informative representation. Additionally, we consider external knowledge as an additional input, and background image caption supervision as a regularizing component. Extensive experiments on three widely adopted benchmarks, i.e., Facebook Hateful Memes, Memotion7k and MultiOFF, indicate that our approach competes and in some cases surpasses state-of-the-art. Our code is available on GitHub https://github.com/ckoutlis/memefier.

翻译：仇恨言论是一种通过互联网显著加剧的社会问题。图像模因等新型数字内容利用多模态手段传播仇恨，其分析与检测难度远超单模态情况。对此类内容的精确自动处理、分析与理解将有助于遏制数字世界中的仇恨言论扩散。为此，我们提出MemeFier——一种基于深度学习的互联网图像模因细粒度分类架构，其核心是双阶段模态融合模块。第一阶段融合生成包含模态对齐信息的特征向量，可捕获模因中文本与图像间的非平凡关联；第二阶段融合利用Transformer编码器的能力，在词元级别学习跨模态相关性并生成信息丰富的表征。此外，我们引入外部知识作为额外输入，并将背景图像字幕监督作为正则化组件。在Facebook Hateful Memes、Memotion7k和MultiOFF三个广泛采用的基准测试上的大量实验表明，我们的方法可与当前最优方法媲美，并在某些场景下超越它们。我们的代码已在GitHub上开源：https://github.com/ckoutlis/memefier。