Nowadays, misinformation is widely spreading over various social media platforms and causes extremely negative impacts on society. To combat this issue, automatically identifying misinformation, especially those containing multimodal content, has attracted growing attention from the academic and industrial communities, and induced an active research topic named Multimodal Misinformation Detection (MMD). Typically, existing MMD methods capture the semantic correlation and inconsistency between multiple modalities, but neglect some potential clues in multimodal content. Recent studies suggest that manipulated traces of the images in articles are non-trivial clues for detecting misinformation. Meanwhile, we find that the underlying intentions behind the manipulation, e.g., harmful and harmless, also matter in MMD. Accordingly, in this work, we propose to detect misinformation by learning manipulation features that indicate whether the image has been manipulated, as well as intention features regarding the harmful and harmless intentions of the manipulation. Unfortunately, the manipulation and intention labels that make these features discriminative are unknown. To overcome the problem, we propose two weakly supervised signals as alternatives by introducing additional datasets on image manipulation detection and formulating two classification tasks as positive and unlabeled learning problems. Based on these ideas, we propose a novel MMD method, namely Harmfully Manipulated Images Matter in MMD (HAMI-M3D). Extensive experiments across three benchmark datasets can demonstrate that HAMI-M3D can consistently improve the performance of any MMD baselines.

翻译：当前，虚假信息在各类社交媒体平台上广泛传播，对社会造成极其负面的影响。为应对这一问题，自动识别虚假信息——特别是包含多模态内容的虚假信息——已日益受到学术界与工业界的关注，并催生了一个活跃的研究方向：多模态虚假信息检测。通常，现有的多模态虚假信息检测方法通过捕捉多模态间的语义关联与不一致性进行检测，但忽略了多模态内容中一些潜在的线索。近期研究表明，文章中图像的篡改痕迹是检测虚假信息的重要线索。同时，我们发现篡改行为背后的潜在意图（例如有害与无害）在多模态虚假信息检测中同样至关重要。为此，在本工作中，我们提出通过学习指示图像是否被篡改的篡改特征，以及反映篡改行为有害或无害意图的意图特征来检测虚假信息。然而，使这些特征具有区分性的篡改标签与意图标签通常是未知的。为克服这一问题，我们通过引入额外的图像篡改检测数据集，并将两个分类任务构建为正例与未标记学习问题，提出了两种弱监督信号作为替代方案。基于这些思路，我们提出了一种新颖的多模态虚假信息检测方法，即“有害篡改图像在多模态虚假信息检测中的重要性”。在三个基准数据集上的大量实验表明，该方法能够持续提升各类多模态虚假信息检测基线的性能。