Fine-tuning pre-trained models has recently yielded remarkable performance gains in graph neural networks (GNNs). In addition to pre-training techniques, inspired by the latest work in the natural language fields, more recent work has shifted towards applying effective fine-tuning approaches, such as parameter-efficient tuning (delta tuning). However, given the substantial differences between GNNs and transformer-based models, applying such approaches directly to GNNs proved to be less effective. In this paper, we present a comprehensive comparison of delta tuning techniques for GNNs and propose a novel delta tuning method specifically designed for GNNs, called AdapterGNN. AdapterGNN preserves the knowledge of the large pre-trained model and leverages highly expressive adapters for GNNs, which can adapt to downstream tasks effectively with only a few parameters, while also improving the model's generalization ability on the downstream tasks. Extensive experiments show that AdapterGNN achieves higher evaluation performance (outperforming full fine-tuning by 1.4% and 5.5% in the chemistry and biology domains respectively, with only 5% of its parameters tuned) and lower generalization gaps compared to full fine-tuning. Moreover, we empirically show that a larger GNN model can have a worse generalization ability, which differs from the trend observed in large language models. We have also provided a theoretical justification for delta tuning can improve the generalization ability of GNNs by applying generalization bounds.

翻译：近期，对预训练模型进行微调在图神经网络（GNNs）中取得了显著性能提升。除预训练技术外，受自然语言领域最新研究的启发，更多近期工作转向应用高效微调方法（如参数高效微调/增量调优）。然而，鉴于GNNs与基于Transformer的模型之间存在本质差异，直接将此类方法应用于GNNs被证实效果不佳。本文对GNNs的增量调优技术进行了全面比较，并提出了一种专为GNNs设计的新型增量调优方法——AdapterGNN。AdapterGNN在保留大型预训练模型知识的同时，为GNNs引入了高表达力的适配器，仅需少量参数便能有效适应下游任务，并进一步提升模型在下游任务中的泛化能力。大量实验表明，AdapterGNN实现了更高的评估性能（在化学和生物领域分别仅调优5%的参数，即超过全参数微调1.4%和5.5%），且相比全参数微调具有更低的泛化差距。此外，我们通过实证发现，大型GNN模型的泛化能力可能更差，这与大型语言模型中观察到的趋势不同。同时，我们通过应用泛化界从理论上证明了增量调优能够提升GNNs的泛化能力。