Today we have a good theoretical understanding of the representational power of Graph Neural Networks (GNNs). For example, their limitations have been characterized in relation to a hierarchy of Weisfeiler-Lehman (WL) isomorphism tests. However, we do not know what is encoded in the learned representations. This is our main question. We answer it using a probing framework to quantify the amount of meaningful information captured in graph representations. Our findings on molecular datasets show the potential of probing for understanding the inductive biases of graph-based models. We compare different families of models and show that transformer-based models capture more chemically relevant information compared to models based on message passing. We also study the effect of different design choices such as skip connections and virtual nodes. We advocate for probing as a useful diagnostic tool for evaluating graph-based models.

翻译：当前，我们对图神经网络（GNNs）表示能力的理论基础已有深入理解。例如，其局限性已通过与韦费勒-莱曼（WL）同构测试层级体系的关系得到刻画。然而，我们尚不清楚这些学习到的表示中究竟编码了何种信息——这正是本文的核心研究问题。为解答该问题，我们采用探测框架量化图表示中蕴含的有意义信息量。在分子数据集上的实验表明，探测方法有助于理解基于图模型的归纳偏置。通过对比不同模型族，我们发现基于Transformer的模型比基于消息传递的模型能捕捉更多化学相关信息。我们还研究了跳跃连接、虚拟节点等不同设计选择的影响。我们倡导将探测作为评估图模型的有效诊断工具。