Moderation of social media content is currently a highly manual task, yet there is too much content posted daily to do so effectively. With the advent of a number of multimodal models, there is the potential to reduce the amount of manual labor for this task. In this work, we aim to explore different models and determine what is most effective for the Hateful Memes Challenge, a challenge by Meta designed to further machine learning research in content moderation. Specifically, we explore the differences between early fusion and late fusion models in classifying multimodal memes containing text and images. We first implement a baseline using unimodal models for text and images separately using BERT and ResNet-152, respectively. The outputs from these unimodal models were then concatenated together to create a late fusion model. In terms of early fusion models, we implement ConcatBERT, VisualBERT, ViLT, CLIP, and BridgeTower. It was found that late fusion performed significantly worse than early fusion models, with the best performing model being CLIP which achieved an AUROC of 70.06. The code for this work is available at https://github.com/bzhao18/CS-7643-Project.

翻译：社交媒体内容审核目前主要依赖人工操作，然而每日发布的内容量过大，难以高效完成。随着多种多模态模型的出现，减少该任务人工劳动量成为可能。本研究旨在探索不同模型，确定对Meta为推进内容审核机器学习研究而设立的"仇恨表情包挑战"（Hateful Memes Challenge）最有效的方案。具体而言，我们研究了早期融合与晚期融合模型在分类包含文本和图像的多模态表情包时的差异。首先，我们分别使用BERT和ResNet-152构建基于文本和图像单模态模型的基线方案，随后将这两个单模态模型的输出拼接形成晚期融合模型。在早期融合模型方面，我们实现了ConcatBERT、VisualBERT、ViLT、CLIP和BridgeTower。研究发现，晚期融合模型的性能显著低于早期融合模型，其中表现最佳的CLIP模型达到了70.06的AUROC值。本研究的代码已开源，详见https://github.com/bzhao18/CS-7643-Project。