We present the Multi-Modal Discussion Transformer (mDT), a novel multi-modal graph-based transformer model for detecting hate speech in online social networks. In contrast to traditional text-only methods, our approach to labelling a comment as hate speech centers around the holistic analysis of text and images. This is done by leveraging graph transformers to capture the contextual relationships in the entire discussion that surrounds a comment, with interwoven fusion layers to combine text and image embeddings instead of processing different modalities separately. We compare the performance of our model to baselines that only process text; we also conduct extensive ablation studies. We conclude with future work for multimodal solutions to deliver social value in online contexts, arguing that capturing a holistic view of a conversation greatly advances the effort to detect anti-social behavior.

翻译：我们提出多模态讨论Transformer（mDT），一种基于图的新型多模态Transformer模型，用于检测在线社交网络中的仇恨言论。与传统仅基于文本的方法不同，我们将评论标记为仇恨言论的核心在于对文本和图像进行整体分析。我们利用图Transformer捕捉评论所在完整讨论中的上下文关系，并通过交织融合层将文本与图像嵌入相结合，而非单独处理不同模态。我们将模型性能与仅处理文本的基线模型进行对比，并开展了广泛的消融研究。最后，我们探讨了多模态解决方案未来在在线环境中创造社会价值的工作方向，并认为整体把握对话内容能显著提升反社会行为检测的成效。