The explosive growth of rumors with text and images on social media platforms has drawn great attention. Existing studies have made significant contributions to cross-modal information interaction and fusion, but they fail to fully explore hierarchical and complex semantic correlation across different modality content, severely limiting their performance on detecting multi-modal rumor. In this work, we propose a novel knowledge-enhanced hierarchical information correlation learning approach (KhiCL) for multi-modal rumor detection by jointly modeling the basic semantic correlation and high-order knowledge-enhanced entity correlation. Specifically, KhiCL exploits cross-modal joint dictionary to transfer the heterogeneous unimodality features into the common feature space and captures the basic cross-modal semantic consistency and inconsistency by a cross-modal fusion layer. Moreover, considering the description of multi-modal content is narrated around entities, KhiCL extracts visual and textual entities from images and text, and designs a knowledge relevance reasoning strategy to find the shortest semantic relevant path between each pair of entities in external knowledge graph, and absorbs all complementary contextual knowledge of other connected entities in this path for learning knowledge-enhanced entity representations. Furthermore, KhiCL utilizes a signed attention mechanism to model the knowledge-enhanced entity consistency and inconsistency of intra-modality and inter-modality entity pairs by measuring their corresponding semantic relevant distance. Extensive experiments have demonstrated the effectiveness of the proposed method.

翻译：社交媒体平台上文本与图像谣言的爆炸式增长引起了广泛关注。现有研究在跨模态信息交互与融合方面取得了重要进展，但未能充分探索不同模态内容之间的层次化复杂语义关联，严重限制了多模态谣言检测的性能。本文提出一种新颖的知识增强层次化信息关联学习方法（KhiCL），通过联合建模基础语义关联与高阶知识增强实体关联，实现多模态谣言检测。具体而言，KhiCL利用跨模态联合词典将异构单模态特征映射至公共特征空间，并通过跨模态融合层捕获基础的跨模态语义一致性与不一致性。此外，考虑到多模态内容的叙述围绕实体展开，KhiCL从图像与文本中提取视觉实体与文本实体，设计知识关联推理策略以寻找外部知识图谱中每对实体间的最短语义关联路径，并吸收该路径中其他连接实体的所有互补上下文知识，用于学习知识增强的实体表征。进一步地，KhiCL采用符号注意力机制，通过度量模态内与模态间实体对的语义关联距离，建模知识增强的实体一致性与不一致性。大量实验验证了所提方法的有效性。