The emergence of large-scale pre-trained language models, such as ChatGPT, has revolutionized various research fields in artificial intelligence. Transformers-based large language models (LLMs) have gradually replaced CNNs and RNNs to unify fields of computer vision and natural language processing. Compared with the data that exists relatively independently such as images, videos or texts, graph is a type of data that contains rich structural and relational information. Meanwhile, natural language, as one of the most expressive mediums, excels in describing complex structures. However, existing work on incorporating graph learning problems into the generative language modeling framework remains very limited. As the importance of language models continues to grow, it becomes essential to explore whether LLMs can also replace GNNs as the foundational model for graphs. In this paper, we propose InstructGLM (Instruction-finetuned Graph Language Model), systematically design highly scalable prompts based on natural language instructions, and use natural language to describe the geometric structure and node features of the graph for instruction tuning an LLMs to perform learning and inference on graphs in a generative manner. Our method exceeds all competitive GNN baselines on ogbn-arxiv, Cora and PubMed datasets, which demonstrates the effectiveness of our method and sheds light on generative language models replacing GNNs as the foundation model for graph machine learning.

翻译：大规模预训练语言模型（如ChatGPT）的出现，已彻底革新了人工智能领域的多个研究方向。基于Transformer的大语言模型（LLMs）逐渐取代CNN和RNN，统一了计算机视觉与自然语言处理领域。与图像、视频或文本等相对独立存在的数据不同，图是一种包含丰富结构与关系信息的数据类型。同时，自然语言作为最具表现力的媒介之一，在描述复杂结构方面尤为擅长。然而，现有将图学习问题整合到生成式语言建模框架中的工作仍然十分有限。随着语言模型重要性的持续增长，探究LLMs能否取代GNNs成为图的基础模型变得至关重要。本文提出InstructGLM（指令微调图语言模型），系统性地设计了基于自然语言指令的高度可扩展提示，利用自然语言描述图的几何结构及节点特征，通过指令微调LLMs实现以生成式方式对图进行学习与推理。我们的方法在ogbn-arxiv、Cora和PubMed数据集上超越了所有具有竞争力的GNN基线，这不仅证明了该方法的有效性，也揭示了生成式语言模型取代GNNs成为图机器学习基础模型的潜力。