While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs -- which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure -- but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility.

翻译：尽管语言模型（LM）是自然语言处理的主力工具，但其与结构化知识图谱（KG）的交互仍处于活跃研究阶段。当前编码此类图谱的方法通常要么（i）将其线性化后通过LM进行嵌入——这会导致结构信息利用不足，要么（ii）使用图神经网络（GNN）保留图谱结构——但GNN在文本特征表示上无法媲美预训练LM。本研究提出一种新型LM——图语言模型（GLM），该模型融合了两种方法的优势并弥补了各自的缺陷。GLM参数从预训练LM初始化，以增强对单个图谱概念与三元组的理解能力；同时，我们通过架构设计引入图偏置，促进图谱内知识的有效分布。这使得GLM能够处理图、文本及两者交织的输入。在关系分类任务上的实验表明，无论是有监督还是零样本场景，GLM嵌入均超越基于LM和GNN的基线方法，充分证明了其通用性。