While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs -- which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure -- but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility.

翻译：尽管语言模型（LMs）是自然语言处理领域的核心工具，但其与结构化知识图谱（KGs）的交互机制仍是当前研究热点。现有编码此类图谱的方法通常分为两类：（i）将其线性化后交由语言模型嵌入——这未能充分利用结构信息；或（ii）使用图神经网络（GNNs）保持图结构——但GNNs在文本特征表示方面不及预训练语言模型。本研究提出一种新型语言模型——图语言模型（GLM），它融合了两种方法的优势并弥补了其缺陷。GLM参数通过预训练语言模型初始化，以增强对图中独立概念与三元组的理解。同时，我们设计了具有图结构偏置的GLM架构，从而促进知识在图中有效传播。这使得GLM能够处理图结构数据、文本数据以及两者交织的混合输入。在关系分类任务上的实证评估表明，GLM嵌入表示在监督学习与零样本场景下均超越基于语言模型和图神经网络的基线方法，展现了其卓越的通用性。