Graph neural networks (GNNs), an emerging class of machine learning models for graphs, have gained popularity for their superior performance in various graph analytical tasks. Mini-batch training is commonly used to train GNNs on large graphs, and data parallelism is the standard approach to scale mini-batch training across multiple GPUs. One of the major performance costs in GNN training is the loading of input features, which prevents GPUs from being fully utilized. In this paper, we argue that this problem is exacerbated by redundancies that are inherent to the data parallel approach. To address this issue, we introduce a hybrid parallel mini-batch training paradigm called split parallelism. Split parallelism avoids redundant data loads and splits the sampling and training of each mini-batch across multiple GPUs online, at each iteration, using a lightweight splitting algorithm. We implement split parallelism in GSplit and show that it outperforms state-of-the-art mini-batch training systems like DGL, Quiver, and $P^3$.

翻译：图神经网络（GNNs）作为一类新兴的图机器学习模型，因其在各种图分析任务中的卓越性能而广受欢迎。在大规模图上训练GNN通常采用小批量训练方式，而数据并行是跨多个GPU扩展小批量训练的标准方法。GNN训练中的一个主要性能开销在于输入特征的加载，这阻碍了GPU的充分利用。本文认为，数据并行方法固有的冗余性加剧了这一问题。为解决此问题，我们提出了一种称为分割并行的混合并行小批量训练范式。分割并行避免了冗余数据加载，并在每个迭代中通过轻量级分割算法在线将每个小批量的采样与训练任务分配到多个GPU上。我们在GSplit中实现了分割并行，并证明其性能优于DGL、Quiver和$P^3$等先进的小批量训练系统。