Misinformation has become a pressing issue. Fake media, in both visual and textual forms, is widespread on the web. While various deepfake detection and text fake news detection methods have been proposed, they are only designed for single-modality forgery based on binary classification, let alone analyzing and reasoning subtle forgery traces across different modalities. In this paper, we highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM^4). DGM^4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content (i.e., image bounding boxes and text tokens), which requires deeper reasoning of multi-modal media manipulation. To support a large-scale investigation, we construct the first DGM^4 dataset, where image-text pairs are manipulated by various approaches, with rich annotation of diverse manipulations. Moreover, we propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities. HAMMER performs 1) manipulation-aware contrastive learning between two uni-modal encoders as shallow manipulation reasoning, and 2) modality-aware cross-attention by multi-modal aggregator as deep manipulation reasoning. Dedicated manipulation detection and grounding heads are integrated from shallow to deep levels based on the interacted multi-modal information. Finally, we build an extensive benchmark and set up rigorous evaluation metrics for this new research problem. Comprehensive experiments demonstrate the superiority of our model; several valuable observations are also revealed to facilitate future research in multi-modal media manipulation.

翻译：虚假信息已成为一个紧迫问题。网络上广泛存在视觉和文本形式的伪造媒体。尽管已提出多种深度伪造检测和文本虚假新闻检测方法，但这些方法仅针对基于二分类的单模态伪造设计，更无法跨不同模态分析和推理细微的伪造痕迹。本文提出一个新的多模态虚假媒体研究问题——检测与定位多模态媒体操纵（DGM^4）。DGM^4不仅需要检测多模态媒体的真实性，还需定位被操纵的内容（如图像边界框和文本标记），这需要对多模态媒体操纵进行更深层的推理。为支撑大规模研究，我们构建了首个DGM^4数据集，其中图像-文本对通过多种方式被操纵，并包含丰富的多样化操纵标注。此外，我们提出一种新颖的层次化多模态操纵推理Transformer（HAMMER），以全面捕获不同模态间的细粒度交互。HAMMER执行：1）两个单模态编码器间的操纵感知对比学习作为浅层操纵推理；2）多模态聚合器通过模态感知交叉注意力实现深层操纵推理。基于交互后的多模态信息，从浅层到深层集成了专用的操纵检测和定位头。最终，我们为该新研究问题建立了全面的基准测试并设置了严格的评估指标。大量实验证明了我们模型的优越性；同时揭示了若干有价值的发现，以促进多模态媒体操纵领域的未来研究。