This paper describes our approach to the multi-modal fact verification (FACTIFY) challenge at AAAI2023. In recent years, with the widespread use of social media, fake news can spread rapidly and negatively impact social security. Automatic claim verification becomes more and more crucial to combat fake news. In fact verification involving multiple modal data, there should be a structural coherence between claim and document. Therefore, we proposed a structure coherence-based multi-modal fact verification scheme to classify fake news. Our structure coherence includes the following four aspects: sentence length, vocabulary similarity, semantic similarity, and image similarity. Specifically, CLIP and Sentence BERT are combined to extract text features, and ResNet50 is used to extract image features. In addition, we also extract the length of the text as well as the lexical similarity. Then the features were concatenated and passed through the random forest classifier. Finally, our weighted average F1 score has reached 0.8079, achieving 2nd place in FACTIFY2.

翻译：本文描述了我们在AAAI2023多模态事实核查（FACTIFY）挑战赛中的方法。近年来，随着社交媒体的广泛使用，虚假新闻可能迅速传播并对社会安全产生负面影响。自动声明核查对于打击虚假新闻变得越来越重要。在涉及多模态数据的事实核查中，声明与文档之间应存在结构一致性。因此，我们提出了一种基于结构一致性的多模态事实核查方案来对虚假新闻进行分类。我们的结构一致性包括以下四个方面：句子长度、词汇相似性、语义相似性和图像相似性。具体而言，我们结合CLIP和Sentence BERT提取文本特征，并使用ResNet50提取图像特征。此外，我们还提取了文本长度及词汇相似性。随后，将特征拼接并输入随机森林分类器。最终，我们的加权平均F1得分达到0.8079，在FACTIFY2中取得第二名。