Hate speech is a societal problem that has significantly grown through the Internet. New forms of digital content such as image memes have given rise to spread of hate using multimodal means, being far more difficult to analyse and detect compared to the unimodal case. Accurate automatic processing, analysis and understanding of this kind of content will facilitate the endeavor of hindering hate speech proliferation through the digital world. To this end, we propose MemeFier, a deep learning-based architecture for fine-grained classification of Internet image memes, utilizing a dual-stage modality fusion module. The first fusion stage produces feature vectors containing modality alignment information that captures non-trivial connections between the text and image of a meme. The second fusion stage leverages the power of a Transformer encoder to learn inter-modality correlations at the token level and yield an informative representation. Additionally, we consider external knowledge as an additional input, and background image caption supervision as a regularizing component. Extensive experiments on three widely adopted benchmarks, i.e., Facebook Hateful Memes, Memotion7k and MultiOFF, indicate that our approach competes and in some cases surpasses state-of-the-art. Our code is available on https://github.com/ckoutlis/memefier.

翻译：仇恨言论是一个通过互联网显著加剧的社会问题。图像模因等新型数字内容借助多模态手段传播仇恨，使其比单模态情况更难以分析与检测。对此类内容的精准自动处理、分析与理解，将有助于遏制仇恨言论在数字世界的扩散。为此，我们提出MemeFier——一种基于深度学习的架构，通过双阶段模态融合模块实现对互联网图像模因的细粒度分类。第一阶段融合生成包含模态对齐信息的特征向量，捕捉模因文本与图像间的非平凡关联；第二阶段融合利用Transformer编码器的能力，在词元级别学习模态间相关性，并生成信息丰富的表征。此外，我们将外部知识作为额外输入，并引入背景图像字幕监督作为正则化组件。在Facebook Hateful Memes、Memotion7k和MultiOFF三个广泛采用的基准测试上的大量实验表明，我们的方法在性能上可与现有最优模型竞争甚至超越。代码已开源至https://github.com/ckoutlis/memefier。