Recent research has demonstrated the efficacy of pre-training graph neural networks (GNNs) to capture the transferable graph semantics and enhance the performance of various downstream tasks. However, the semantic knowledge learned from pretext tasks might be unrelated to the downstream task, leading to a semantic gap that limits the application of graph pre-training. To reduce this gap, traditional approaches propose hybrid pre-training to combine various pretext tasks together in a multi-task learning fashion and learn multi-grained knowledge, which, however, cannot distinguish tasks and results in some transferable task-specific knowledge distortion by each other. Moreover, most GNNs cannot distinguish nodes located in different parts of the graph, making them fail to learn position-specific knowledge and lead to suboptimal performance. In this work, inspired by the prompt-based tuning in natural language processing, we propose a unified framework for graph hybrid pre-training which injects the task identification and position identification into GNNs through a prompt mechanism, namely multi-task graph dual prompt (ULTRA-DP). Based on this framework, we propose a prompt-based transferability test to find the most relevant pretext task in order to reduce the semantic gap. To implement the hybrid pre-training tasks, beyond the classical edge prediction task (node-node level), we further propose a novel pre-training paradigm based on a group of $k$-nearest neighbors (node-group level). The combination of them across different scales is able to comprehensively express more structural semantics and derive richer multi-grained knowledge. Extensive experiments show that our proposed ULTRA-DP can significantly enhance the performance of hybrid pre-training methods and show the generalizability to other pre-training tasks and backbone architectures.

翻译：近期研究已证明，通过预训练图神经网络（GNNs）捕获可迁移的图语义并提升各类下游任务性能的有效性。然而，从前置任务中习得的语义知识可能与下游任务无关，导致语义鸿沟限制了图预训练的应用。为缩小这一差距，传统方法提出混合预训练策略，以多任务学习方式组合多种前置任务以获取多粒度知识，但此类方法无法区分任务，导致部分可迁移的任务特定知识相互扭曲。此外，多数GNN无法区分位于图不同区域的节点，使其难以学习位置特定知识，导致性能次优。受自然语言处理中提示调优技术的启发，本文提出一种统一的图混合预训练框架，通过提示机制将任务识别与位置识别注入GNN，即多任务图双提示（ULTRA-DP）。基于该框架，我们提出基于提示的可迁移性测试以定位最相关的前置任务，从而缩小语义鸿沟。为实施混合预训练任务，除经典边预测任务（节点-节点层级）外，本文进一步提出基于$k$-近邻集合的全新预训练范式（节点-群组层级）。跨尺度的任务组合能更全面地表达结构语义，并获取更丰富的多粒度知识。大量实验表明，所提出的ULTRA-DP可显著增强混合预训练方法的性能，并展现出对其他预训练任务和主干架构的泛化能力。