Combating disinformation is one of the burning societal crises -- about 67% of the American population believes that disinformation produces a lot of uncertainty, and 10% of them knowingly propagate disinformation. Evidence shows that disinformation can manipulate democratic processes and public opinion, causing disruption in the share market, panic and anxiety in society, and even death during crises. Therefore, disinformation should be identified promptly and, if possible, mitigated. With approximately 3.2 billion images and 720,000 hours of video shared online daily on social media platforms, scalable detection of multimodal disinformation requires efficient fact verification. Despite progress in automatic text-based fact verification (e.g., FEVER, LIAR), the research community lacks substantial effort in multimodal fact verification. To address this gap, we introduce FACTIFY 3M, a dataset of 3 million samples that pushes the boundaries of the domain of fact verification via a multimodal fake news dataset, in addition to offering explainability through the concept of 5W question-answering. Salient features of the dataset include: (i) textual claims, (ii) ChatGPT-generated paraphrased claims, (iii) associated images, (iv) stable diffusion-generated additional images (i.e., visual paraphrases), (v) pixel-level image heatmap to foster image-text explainability of the claim, (vi) 5W QA pairs, and (vii) adversarial fake news stories.

翻译：打击虚假信息是当前最严峻的社会危机之一——约67%的美国民众认为虚假信息制造了大量不确定性，而其中10%的人知情传播虚假信息。证据表明，虚假信息可能操纵民主进程与公众舆论，引发股市动荡、社会恐慌与焦虑，甚至危机时期导致生命损失。因此，虚假信息需被及时发现，并尽可能予以缓解。鉴于社交媒体平台每日约32亿张图片和72万小时视频的共享规模，实现可扩展的多模态虚假信息检测亟需高效的事实核查。尽管基于文本的自动事实核查（如FEVER、LIAR）已取得进展，但研究界在多模态事实核查领域仍缺乏实质性努力。为填补这一空白，我们推出包含300万样本的FACTIFY 3M数据集，该数据集通过多模态假新闻数据集拓展了事实核查领域边界，同时通过5W问答概念提供可解释性。数据集的显著特征包括：(i)文本声明、(ii)ChatGPT生成的改写声明、(iii)关联图像、(iv)Stable Diffusion生成的增补图像（即视觉改写）、(v)像素级图像热力图以增强声明图文可解释性、(vi)5W问答对，以及(vii)对抗性假新闻故事。