Recent research has demonstrated the efficacy of pre-training graph neural networks (GNNs) to capture the transferable graph semantics and enhance the performance of various downstream tasks. However, the semantic knowledge learned from pretext tasks might be unrelated to the downstream task, leading to a semantic gap that limits the application of graph pre-training. To reduce this gap, traditional approaches propose hybrid pre-training to combine various pretext tasks together in a multi-task learning fashion and learn multi-grained knowledge, which, however, cannot distinguish tasks and results in some transferable task-specific knowledge distortion by each other. Moreover, most GNNs cannot distinguish nodes located in different parts of the graph, making them fail to learn position-specific knowledge and lead to suboptimal performance. In this work, inspired by the prompt-based tuning in natural language processing, we propose a unified framework for graph hybrid pre-training which injects the task identification and position identification into GNNs through a prompt mechanism, namely multi-task graph dual prompt (ULTRA-DP). Based on this framework, we propose a prompt-based transferability test to find the most relevant pretext task in order to reduce the semantic gap. To implement the hybrid pre-training tasks, beyond the classical edge prediction task (node-node level), we further propose a novel pre-training paradigm based on a group of $k$-nearest neighbors (node-group level). The combination of them across different scales is able to comprehensively express more structural semantics and derive richer multi-grained knowledge. Extensive experiments show that our proposed ULTRA-DP can significantly enhance the performance of hybrid pre-training methods and show the generalizability to other pre-training tasks and backbone architectures.

翻译：近期研究证明了预训练图神经网络（GNNs）能够捕获可迁移的图语义并提升下游任务表现。然而，预训练任务习得的语义知识可能与下游任务无关，导致语义鸿沟限制了图预训练的应用。为缩小这一鸿沟，传统方法提出混合预训练，以多任务学习方式组合多种预文本任务并学习多粒度知识，但该方法无法区分任务，导致部分可迁移的任务特异性知识相互扭曲。此外，多数GNNs难以区分图中不同位置的节点，使其无法学习位置特异性知识，从而影响性能。受自然语言处理中基于提示调优的启发，本文提出统一的图混合预训练框架，通过提示机制将任务识别和位置识别注入GNNs，即多任务图双提示（ULTRA-DP）。基于该框架，我们提出基于提示的可迁移性测试以寻找最相关的预文本任务，从而缩小语义鸿沟。为实施混合预训练任务，除经典的边预测任务（节点-节点级别）外，我们进一步提出基于$k$-近邻组的全新预训练范式（节点-组级别）。二者跨不同尺度的组合能全面表达更多结构语义并获取更丰富的多粒度知识。大量实验表明，所提出的ULTRA-DP能显著增强混合预训练方法的性能，并展现出对其他预训练任务和骨干架构的泛化能力。