Rich textual and topological information of textual graphs need to be modeled in real-world applications such as webpages, e-commerce, and academic articles. Practitioners have been long following the path of adopting a shallow text encoder and a subsequent graph neural network (GNN) to solve this problem. In light of recent advancements in large language models (LLMs), it is apparent that integrating LLMs for enhanced textual encoding can substantially improve the performance of textual graphs. Nevertheless, the efficiency of these methods poses a significant challenge. In this paper, we propose ENGINE, a parameter- and memory-efficient fine-tuning method for textual graphs with an LLM encoder. The key insight is to combine the LLMs and GNNs through a tunable side structure, which significantly reduces the training complexity without impairing the joint model's capacity. Extensive experiments on textual graphs demonstrate our method's effectiveness by achieving the best model performance, meanwhile having the lowest training cost compared to previous methods. Moreover, we introduce two variants with caching and dynamic early exit to further enhance training and inference speed. Specifically, caching accelerates ENGINE's training by 12x, and dynamic early exit achieves up to 5x faster inference with a negligible performance drop (at maximum 1.17% relevant drop across 7 datasets).

翻译：丰富的文本与拓扑信息在网页、电子商务和学术论文等实际应用场景中需要被建模。长期以来，研究者们采用浅层文本编码器结合图神经网络（GNN）的方式来解决这一问题。随着大语言模型（LLM）的最新进展，整合LLM以增强文本编码能力明显可以显著提升文本图的性能。然而，这类方法的效率问题构成了重大挑战。本文提出ENGINE——一种面向文本图的参数与内存高效的LLM编码器微调方法。其核心思想是通过可调侧结构结合LLM与GNN，在保持联合模型能力的同时大幅降低训练复杂度。在多个文本图上的大量实验表明，该方法在实现最优模型性能的同时，相比先前方法具有最低的训练成本。此外，我们引入了缓存与动态早停两种变体，分别进一步加速训练与推理速度。具体而言，缓存机制使ENGINE训练速度提升12倍，动态早停则实现最高5倍的推理加速，且性能下降可忽略（在7个数据集上的最大性能降幅为1.17%）。