The emergence of large-scale pre-trained language models, such as ChatGPT, has revolutionized various research fields in artificial intelligence. Transformers-based large language models (LLMs) have gradually replaced CNNs and RNNs to unify fields of computer vision and natural language processing. Compared with the data that exists relatively independently such as images, videos or texts, graph is a type of data that contains rich structural and relational information. Meanwhile, natural language, as one of the most expressive mediums, excels in describing complex structures. However, existing work on incorporating graph learning problems into the generative language modeling framework remains very limited. As the importance of large language models continues to grow, it becomes essential to explore whether LLMs can also replace GNNs as the foundation model for graphs. In this paper, we propose InstructGLM (Instruction-finetuned Graph Language Model), systematically design highly scalable prompts based on natural language instructions, and use natural language to describe the geometric structure and node features of the graph for instruction tuning an LLM to perform learning and inference on graphs in a generative manner. Our method exceeds all competitive GNN baselines on ogbn-arxiv, Cora and PubMed datasets, which demonstrates the effectiveness of our method and sheds light on generative large language models as the foundation model for graph machine learning.

翻译：大规模预训练语言模型（如ChatGPT）的出现，彻底改变了人工智能领域的多个研究方向。基于Transformer的大语言模型（LLMs）逐渐取代CNN和RNN，统一了计算机视觉和自然语言处理领域。与图像、视频或文本等相对独立存在的数据相比，图是一种包含丰富结构和关系信息的数据类型。同时，自然语言作为最具表现力的媒介之一，擅长描述复杂结构。然而，现有将图学习问题融入生成式语言建模框架的工作仍然非常有限。随着大语言模型重要性不断提升，探究LLMs能否取代GNN作为图的基座模型变得至关重要。本文提出InstructGLM（指令微调图语言模型），系统性地设计了基于自然语言指令的高度可扩展提示，并利用自然语言描述图的几何结构和节点特征，通过指令微调使LLM以生成式方式对图进行学习和推理。我们的方法在ogbn-arxiv、Cora和PubMed数据集上超越了所有具有竞争力的GNN基线，证明了该方法的有效性，并为生成式大语言模型作为图机器学习的基座模型提供了新思路。