When training a Neural Network, it is optimized using the available training data with the hope that it generalizes well to new or unseen testing data. At the same absolute value, a flat minimum in the loss landscape is presumed to generalize better than a sharp minimum. Methods for determining flat minima have been mostly researched for independent and identically distributed (i. i. d.) data such as images. Graphs are inherently non-i. i. d. since the vertices are edge-connected. We investigate flat minima methods and combinations of those methods for training graph neural networks (GNNs). We use GCN and GAT as well as extend Graph-MLP to work with more layers and larger graphs. We conduct experiments on small and large citation, co-purchase, and protein datasets with different train-test splits in both the transductive and inductive training procedure. Results show that flat minima methods can improve the performance of GNN models by over 2 points, if the train-test split is randomized. Following Shchur et al., randomized splits are essential for a fair evaluation of GNNs, as other (fixed) splits like 'Planetoid' are biased. Overall, we provide important insights for improving and fairly evaluating flat minima methods on GNNs. We recommend practitioners to always use weight averaging techniques, in particular EWA when using early stopping. While weight averaging techniques are only sometimes the best performing method, they are less sensitive to hyperparameters, need no additional training, and keep the original model unchanged. All source code is available in https://github.com/Foisunt/FMMs-in-GNNs.

翻译：训练神经网络时，模型利用现有训练数据进行优化，期望能良好泛化至未见过的测试数据。在绝对值相同的情况下，损失景观中的平坦极小值被认为比尖锐极小值具有更好的泛化能力。确定平坦极小值的方法主要针对独立同分布数据（如图像）展开研究。图数据本质上是非独立同分布的，因为顶点通过边相互连接。本文研究了平坦极小值方法及其组合在训练图神经网络中的效果。我们采用GCN和GAT模型，并扩展Graph-MLP以支持更多层和更大规模图数据。在引文、协同购买和蛋白质数据集上，我们针对不同训练-测试划分（包括直推式和归纳式训练流程）进行实验。结果表明，若训练-测试划分为随机划分，平坦极小值方法可使图神经网络模型性能提升超过2个百分点。遵循Shchur等人的建议，随机划分对于公平评估图神经网络至关重要，因为其他固定划分（如"Planetoid"）存在偏倚。总体而言，我们为改进和公平评估图神经网络上的平坦极小值方法提供了重要见解。建议实践者始终采用权重平均技术，尤其在使用早停法时推荐指数加权平均。虽然权重平均技术并非始终表现最优，但其对超参数敏感度较低、无需额外训练且保持原始模型不变。所有源代码已开源至https://github.com/Foisunt/FMMs-in-GNNs。