The message passing-based graph neural networks (GNNs) have achieved great success in many real-world applications. However, training GNNs on large-scale graphs suffers from the well-known neighbor explosion problem, i.e., the exponentially increasing dependencies of nodes with the number of message passing layers. Subgraph-wise sampling methods -- a promising class of mini-batch training techniques -- discard messages outside the mini-batches in backward passes to avoid the neighbor explosion problem at the expense of gradient estimation accuracy. This poses significant challenges to their convergence analysis and convergence speeds, which seriously limits their reliable real-world applications. To address this challenge, we propose a novel subgraph-wise sampling method with a convergence guarantee, namely Local Message Compensation (LMC). To the best of our knowledge, LMC is the {\it first} subgraph-wise sampling method with provable convergence. The key idea of LMC is to retrieve the discarded messages in backward passes based on a message passing formulation of backward passes. By efficient and effective compensations for the discarded messages in both forward and backward passes, LMC computes accurate mini-batch gradients and thus accelerates convergence. We further show that LMC converges to first-order stationary points of GNNs. Experiments on large-scale benchmark tasks demonstrate that LMC significantly outperforms state-of-the-art subgraph-wise sampling methods in terms of efficiency.

翻译：基于消息传递机制的图神经网络（GNN）在众多实际应用中取得了巨大成功。然而，大规模图上的GNN训练面临众所周知的邻域爆炸问题，即节点依赖关系随消息传递层数呈指数增长。子图采样方法作为一种有前景的mini-batch训练技术，通过在反向传播中丢弃mini-batch外的消息来避免邻域爆炸问题，但这导致梯度估计精度损失。该问题严重挑战了其收敛性分析与收敛速度，从而限制了其在真实场景中的可靠应用。为应对该挑战，我们提出一种具有收敛保证的新型子图采样方法——局部消息补偿（LMC）。据我们所知，LMC是首个具备可证明收敛性的子图采样方法。其核心思想是基于反向传播的消息传递公式，在反向过程中恢复被丢弃的消息。通过在前向与反向过程中对丢弃消息进行高效且有效的补偿，LMC能计算精确的mini-batch梯度，从而加速收敛。我们进一步证明LMC能收敛至GNN的一阶驻点。大规模基准实验表明，LMC在效率上显著优于当前最先进的子图采样方法。