The standard paradigm for fake news detection mainly utilizes text information to model the truthfulness of news. However, the discourse of online fake news is typically subtle and it requires expert knowledge to use textual information to debunk fake news. Recently, studies focusing on multimodal fake news detection have outperformed text-only methods. Recent approaches utilizing the pre-trained model to extract unimodal features, or fine-tuning the pre-trained model directly, have become a new paradigm for detecting fake news. Again, this paradigm either requires a large number of training instances, or updates the entire set of pre-trained model parameters, making real-world fake news detection impractical. Furthermore, traditional multimodal methods fuse the cross-modal features directly without considering that the uncorrelated semantic representation might inject noise into the multimodal features. This paper proposes a Similarity-Aware Multimodal Prompt Learning (SAMPLE) framework. First, we incorporate prompt learning into multimodal fake news detection. Prompt learning, which only tunes prompts with a frozen language model, can reduce memory usage significantly and achieve comparable performances, compared with fine-tuning. We analyse three prompt templates with a soft verbalizer to detect fake news. In addition, we introduce the similarity-aware fusing method to adaptively fuse the intensity of multimodal representation and mitigate the noise injection via uncorrelated cross-modal features. For evaluation, SAMPLE surpasses the F1 and the accuracies of previous works on two benchmark multimodal datasets, demonstrating the effectiveness of the proposed method in detecting fake news. In addition, SAMPLE also is superior to other approaches regardless of few-shot and data-rich settings.

翻译：假新闻检测的标准范式主要利用文本信息对新闻真实性进行建模。然而，在线假新闻的论述通常隐晦难辨，需要专业知识才能通过文本信息识破假新闻。近年来，针对多模态假新闻检测的研究已超越纯文本方法。当前利用预训练模型提取单模态特征或直接微调预训练模型的方法，已成为假新闻检测的新范式。然而，这类范式要么需要大量训练样本，要么需更新全部预训练模型参数，导致实际假新闻检测难以落地。此外，传统多模态方法直接融合跨模态特征，未考虑不相关语义表示可能向多模态特征注入噪声。本文提出相似性感知多模态提示学习（SAMPLE）框架。首先，我们将提示学习引入多模态假新闻检测。提示学习仅需在冻结语言模型状态下调整提示，与微调相比可显著降低内存占用且达到相当性能。我们设计了三种带软词表的提示模板进行假新闻检测。其次，引入相似性感知融合方法，自适应调整多模态表征强度，缓解不相关跨模态特征带来的噪声干扰。实验表明，在两个基准多模态数据集上，SAMPLE的F1值与准确率均超越先前工作，验证了该方法在假新闻检测中的有效性。此外，无论在小样本还是数据充足场景中，SAMPLE均优于其他方法。