The standard paradigm for fake news detection mainly utilizes text information to model the truthfulness of news. However, the discourse of online fake news is typically subtle and it requires expert knowledge to use textual information to debunk fake news. Recently, studies focusing on multimodal fake news detection have outperformed text-only methods. Recent approaches utilizing the pre-trained model to extract unimodal features, or fine-tuning the pre-trained model directly, have become a new paradigm for detecting fake news. Again, this paradigm either requires a large number of training instances, or updates the entire set of pre-trained model parameters, making real-world fake news detection impractical. Furthermore, traditional multimodal methods fuse the cross-modal features directly without considering that the uncorrelated semantic representation might inject noise into the multimodal features. This paper proposes a Similarity-Aware Multimodal Prompt Learning (SAMPLE) framework. First, we incorporate prompt learning into multimodal fake news detection. Prompt learning, which only tunes prompts with a frozen language model, can reduce memory usage significantly and achieve comparable performances, compared with fine-tuning. We analyse three prompt templates with a soft verbalizer to detect fake news. In addition, we introduce the similarity-aware fusing method to adaptively fuse the intensity of multimodal representation and mitigate the noise injection via uncorrelated cross-modal features. For evaluation, SAMPLE surpasses the F1 and the accuracies of previous works on two benchmark multimodal datasets, demonstrating the effectiveness of the proposed method in detecting fake news. In addition, SAMPLE also is superior to other approaches regardless of few-shot and data-rich settings.

翻译：假新闻检测的标准范式主要利用文本信息对新闻真实性进行建模。然而，网络假新闻的语篇通常十分微妙，需要专业知识才能通过文本信息揭穿假新闻。近年来，专注于多模态假新闻检测的研究已超越了纯文本方法。当前利用预训练模型提取单模态特征，或直接微调预训练模型的方法，已成为检测假新闻的新范式。然而，该范式要么需要大量训练样本，要么需要更新全部预训练模型参数，导致实际应用中的假新闻检测难以落地。此外，传统多模态方法直接融合跨模态特征，却未考虑不相关的语义表示可能向多模态特征中注入噪声。本文提出了一种相似性感知的多模态提示学习（SAMPLE）框架。首先，我们将提示学习引入多模态假新闻检测。与微调相比，提示学习仅需在冻结语言模型的情况下调整提示，即可显著降低内存占用并实现相当的性能。我们分析了三种使用软分类器的提示模板以检测假新闻。此外，我们引入了相似性感知融合方法，以自适应地调整多模态表示的融合强度，并缓解因不相关跨模态特征引入的噪声。在评估中，SAMPLE在两个基准多模态数据集上的F1值和准确率均超越了以往研究，证明了该方法在检测假新闻中的有效性。此外，无论在小样本还是数据充足场景下，SAMPLE均优于其他方法。