Graph neural networks (GNNs) are widely used for the detection of fake news by modeling the content and propagation structure of news articles on social media. We show that two of the most commonly used benchmark data sets - GossipCop and PolitiFact - are poorly suited to evaluating the utility of models that use propagation structure. Specifically, these data sets exhibit shallow, ego-like graph topologies that provide little or no ability to differentiate among modeling methods. We systematically benchmark five GNN architectures against a structure-agnostic multilayer perceptron (MLP) that uses the same node features. We show that MLPs match or closely trail the performance of GNNs, with performance gaps often within 1-2% and overlapping confidence intervals. To isolate the contribution of structure in these datasets, we conduct controlled experiments where node features are shuffled or edge structures randomized. We find that performance collapses under feature shuffling but remains stable under edge randomization. This suggests that structure plays a negligible role in these benchmarks. Structural analysis further reveals that over 75% of nodes are only one hop from the root, exhibiting minimal structural diversity. In contrast, on synthetic datasets where node features are noisy and structure is informative, GNNs significantly outperform MLPs. These findings provide strong evidence that widely used benchmarks do not meaningfully test the utility of modeling structural features, and they motivate the development of datasets with richer, more diverse graph topologies.

翻译：图神经网络（GNNs）通过建模社交媒体上新闻文章的内容与传播结构，被广泛应用于虚假新闻检测。我们发现，两个最常用的基准数据集——GossipCop和PolitiFact——并不适合评估利用传播结构的模型效用。具体而言，这些数据集呈现浅层、自我中心式的图拓扑结构，几乎无法区分不同建模方法的性能。我们系统性地将五种GNN架构与使用相同节点特征的结构不可知多层感知机（MLP）进行基准测试。结果表明，MLP的性能与GNN相当或接近，性能差距通常在1-2%以内，且置信区间重叠。为分离结构在这些数据集中的贡献，我们进行了控制实验，包括打乱节点特征或随机化边结构。实验发现，打乱特征时性能显著下降，而随机化边结构时性能保持稳定，这暗示结构在这些基准测试中的作用可忽略不计。结构分析进一步显示，超过75%的节点距离根节点仅一跳，结构多样性极低。相比之下，在节点特征噪声较大且结构信息丰富的合成数据集上，GNN显著优于MLP。这些发现强有力地证明，广泛使用的基准数据集未能有效检验建模结构特征的效用，从而激励开发具有更丰富、更多样图拓扑结构的数据集。