Recent studies attempted to utilize multilayer perceptrons (MLPs) to solve semi-supervised node classification on graphs, by training a student MLP by knowledge distillation (KD) from a teacher graph neural network (GNN). While previous studies have focused mostly on training the student MLP by matching the output probability distributions between the teacher and student models during KD, it has not been systematically studied how to inject the structural information in an explicit and interpretable manner. Inspired by GNNs that separate feature transformation $T$ and propagation $\Pi$, we re-frame the KD process as enabling the student MLP to explicitly learn both $T$ and $\Pi$. Although this can be achieved by applying the inverse propagation $\Pi^{-1}$ before distillation from the teacher GNN, it still comes with a high computational cost from large matrix multiplications during training. To solve this problem, we propose Propagate & Distill (P&D), which propagates the output of the teacher GNN before KD and can be interpreted as an approximate process of the inverse propagation $\Pi^{-1}$. Through comprehensive evaluations using real-world benchmark datasets, we demonstrate the effectiveness of P&D by showing further performance boost of the student MLP.

翻译：近期研究尝试利用多层感知机（MLPs）解决图上的半监督节点分类问题，通过知识蒸馏（KD）从教师图神经网络（GNN）训练学生MLP。尽管先前研究主要关注在KD过程中通过匹配教师模型与学生模型的输出概率分布来训练学生MLP，但如何以显式且可解释的方式注入结构信息尚未得到系统研究。受分离特征变换$T$与传播$\Pi$的GNN启发，我们将KD过程重新定义为使学生MLP能够显式学习$T$与$\Pi$。虽然这可以通过在从教师GNN蒸馏前应用逆传播$\Pi^{-1}$实现，但训练过程中大矩阵乘法仍会带来高昂计算成本。为解决该问题，我们提出传播与蒸馏（P&D），该方法在KD前传播教师GNN的输出，可被解释为逆传播$\Pi^{-1}$的近似过程。通过在真实世界基准数据集上的全面评估，我们展示了P&D的有效性，表明其能进一步提升学生MLP的性能。