The standard paradigm for fake news detection mainly utilizes text information to model the truthfulness of news. However, the discourse of online fake news is typically subtle and it requires expert knowledge to use textual information to debunk fake news. Recently, studies focusing on multimodal fake news detection have outperformed text-only methods. Recent approaches utilizing the pre-trained model to extract unimodal features, or fine-tuning the pre-trained model directly, have become a new paradigm for detecting fake news. Again, this paradigm either requires a large number of training instances, or updates the entire set of pre-trained model parameters, making real-world fake news detection impractical. Furthermore, traditional multimodal methods fuse the cross-modal features directly without considering that the uncorrelated semantic representation might inject noise into the multimodal features. This paper proposes a Similarity-Aware Multimodal Prompt Learning (SAMPLE) framework. First, we incorporate prompt learning into multimodal fake news detection. Prompt learning, which only tunes prompts with a frozen language model, can reduce memory usage significantly and achieve comparable performances, compared with fine-tuning. We analyse three prompt templates with a soft verbalizer to detect fake news. In addition, we introduce the similarity-aware fusing method to adaptively fuse the intensity of multimodal representation and mitigate the noise injection via uncorrelated cross-modal features. For evaluation, SAMPLE surpasses the F1 and the accuracies of previous works on two benchmark multimodal datasets, demonstrating the effectiveness of the proposed method in detecting fake news. In addition, SAMPLE also is superior to other approaches regardless of few-shot and data-rich settings.

翻译：虚假新闻检测的标准范式主要利用文本信息对新闻真实性进行建模。然而，在线虚假新闻的论述通常较为隐晦，需要专业知识才能运用文本信息揭露虚假新闻。近期，聚焦于多模态虚假新闻检测的研究已超越仅依赖文本的方法。利用预训练模型提取单模态特征或直接微调预训练模型的最新方法，已成为检测虚假新闻的新范式。然而，这种范式要么需要大量训练样本，要么需要更新整个预训练模型的参数集，这使得实际场景中的虚假新闻检测难以实现。此外，传统多模态方法直接融合跨模态特征，而未考虑不相关的语义表示可能向多模态特征中注入噪声。本文提出一种相似性感知的多模态提示学习（SAMPLE）框架。首先，我们将提示学习融入多模态虚假新闻检测。与微调相比，提示学习仅需冻结语言模型并调整提示，能显著减少内存占用并实现同等性能。我们分析了三种结合软提示词的提示模板用于虚假新闻检测。其次，我们引入相似性感知融合方法，自适应地调整多模态表示的融合强度，并缓解不相关跨模态特征导致的噪声注入问题。实验评估表明，在两个基准多模态数据集上，SAMPLE在F1分数和准确率上均超越先前工作，证实了所提方法在虚假新闻检测中的有效性。此外，无论在小样本还是数据充足场景下，SAMPLE均优于其他方法。