Large Language Models (LLMs) have achieved impressive performance in text understanding and have become an essential tool for building smart assistants. Originally focusing on text, they have been enhanced with multimodal capabilities in recent works that successfully built visual instruction following assistants. As far as the graph modality goes, however, no such assistants have yet been developed. Graph structures are complex in that they represent relation between different features and are permutation invariant. Moreover, representing them in purely textual form does not always lead to good LLM performance even for finetuned models. As a result, there is a need to develop a new method to integrate graphs in LLMs for general graph understanding. This work explores the integration of the graph modality in LLM for general graph instruction following tasks. It aims at producing a deep learning model that enhances an underlying LLM with graph embeddings and trains it to understand them and to produce, given an instruction, an answer grounded in the graph representation. The approach performs significantly better than a graph to text approach and remains consistent even for larger graphs.

翻译：大型语言模型（LLM）在文本理解方面已取得令人瞩目的性能，并成为构建智能助手的重要工具。早期研究主要聚焦于文本，而近期工作通过增强多模态能力，已成功构建出遵循视觉指令的助手。然而，针对图结构模态，目前尚未开发出类似的助手。图结构具有复杂性，既能表示不同特征间的关系，又具备排列不变性。此外，即使对微调后的模型而言，仅以纯文本形式表示图结构也未必能带来良好的LLM性能。因此，有必要开发一种新方法，将图结构整合到LLM中以实现通用图理解。本研究探索了在LLM中集成图模态以完成通用图指令遵循任务。其目标是构建一个深度学习模型，通过图嵌入增强底层LLM，并训练其理解图结构，使其能够根据给定指令生成基于图表示的答案。该方法性能显著优于图到文本的转换方法，且在处理更大规模图时仍保持稳定性。