Despite recent community revelations about the advancements and potential applications of Large Language Models (LLMs) in understanding Text-Attributed Graph (TAG), the deployment of LLMs for production is hindered by its high computational and storage requirements, as well as long latencies during model inference. Simultaneously, although traditional Graph Neural Networks (GNNs) are light weight and adept at learning structural features of graphs, their ability to grasp the complex semantics in TAG is somewhat constrained for real applications. To address these limitations, we concentrate on the downstream task of node classification in TAG and propose a novel graph knowledge distillation framework, termed Linguistic Graph Knowledge Distillation (LinguGKD), using LLMs as teacher models and GNNs as student models for knowledge distillation. It involves TAG-oriented instruction tuning of LLM on designed tailored prompts, followed by propagating knowledge and aligning the hierarchically learned node features from the teacher LLM to the student GNN in latent space, employing a layer-adaptive contrastive learning strategy. Through extensive experiments on a variety of LLM and GNN models and multiple benchmark datasets, the proposed LinguGKD significantly boosts the student GNN's predictive accuracy and convergence rate, without the need of extra data or model parameters. Compared to teacher LLM, distilled GNN achieves superior inference speed equipped with much fewer computing and storage demands, when surpassing the teacher LLM's classification accuracy on some of benchmark datasets.

翻译：尽管近期学界揭示了大型语言模型（LLMs）在理解文本属性图（TAG）方面的进展与潜在应用，但在实际生产中部署LLMs仍受限于其高昂的计算与存储成本，以及模型推理过程中的长延迟。与此同时，传统图神经网络（GNNs）虽轻量且擅长学习图的结构特征，但在实际应用中，它们对TAG中复杂语义的捕获能力仍存在一定局限。为应对这些挑战，我们聚焦于TAG中的节点分类下游任务，提出了一种新颖的图知识蒸馏框架——语言化图知识蒸馏（LinguGKD），该框架以LLMs作为教师模型、GNNs作为学生模型进行知识蒸馏。该方法首先针对TAG特性，基于设计的定制化提示对LLM进行指令微调；随后通过层级自适应对比学习策略，将教师LLM分层学习的节点特征在隐空间中传播并对齐至学生GNN。通过在多种LLM与GNN模型及多个基准数据集上的广泛实验，所提出的LinguGKD显著提升了学生GNN的预测精度与收敛速度，且无需额外数据或模型参数。与教师LLM相比，蒸馏后的GNN在部分基准数据集上超越教师LLM分类精度的同时，实现了更优的推理速度，且大幅降低了计算与存储需求。