With the rapid development of online social media platforms, the spread of rumours has become a critical societal concern. Current methods for rumour detection can be categorized into image-text pair classification and source-reply graph classification. In this paper, we propose a novel approach that combines multimodal source and propagation graph features for rumour classification. We introduce the Unified Multimodal Graph Transformer Network (UMGTN) which integrates Transformer encoders to fuse these features. Given that not every message in social media is associated with an image and community responses in propagation graphs do not immediately follow source messages, our aim is to build a network architecture that handles missing features such as images or replies. To enhance the model's robustness to data with missing features, we adopt a multitask learning framework that simultaneously learns representations between samples with complete and missing features. We evaluate our proposed method on four real-world datasets, augmenting them by recovering images and replies from Twitter and Weibo. Experimental results demonstrate that our UMGTN with multitask learning achieves state-of-the-art performance, improving F1-score by 1.0% to 4.0%, while maintaining detection robustness to missing features within 2% accuracy and F1-score compared to models trained without the multitask learning framework. We have made our models and datasets publicly available at: https://thcheung.github.io/umgtn/.

翻译：随着在线社交媒体平台的快速发展，谣言传播已成为关键的社会问题。现有谣言检测方法可分为图文对分类和源-回复图分类两类。本文提出一种融合多模态源特征与传播图特征的新型谣言分类方法。我们设计了统一多模态图Transformer网络，通过集成Transformer编码器实现特征融合。考虑到社交媒体中并非每条消息都关联图像，且传播图中社区回复并非即时跟随源消息，我们的目标是构建能处理图像或回复等缺失特征的网络架构。为增强模型对缺失特征数据的鲁棒性，我们采用多任务学习框架，同步学习完整特征样本与缺失特征样本之间的表示。基于四个真实世界数据集进行实验评估，通过从Twitter和微博平台恢复图像与回复来增强数据。实验结果表明，本研究的UMGTN模型结合多任务学习达到了最先进的性能，F1-score提升1.0%-4.0%，同时与未使用多任务学习框架训练的模型相比，在2%准确率和F1-score范围内保持了对缺失特征的检测鲁棒性。我们已公开模型与数据集，访问地址为：https://thcheung.github.io/umgtn/。