Question Answering over Knowledge Graph (KGQA) aims to seek answer entities for the natural language question from a large-scale Knowledge Graph~(KG). To better perform reasoning on KG, recent work typically adopts a pre-trained language model~(PLM) to model the question, and a graph neural network~(GNN) based module to perform multi-hop reasoning on the KG. Despite the effectiveness, due to the divergence in model architecture, the PLM and GNN are not closely integrated, limiting the knowledge sharing and fine-grained feature interactions. To solve it, we aim to simplify the above two-module approach, and develop a more capable PLM that can directly support subgraph reasoning for KGQA, namely ReasoningLM. In our approach, we propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning, and also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions. After adaptation, the PLM can be parameter-efficient fine-tuned on downstream tasks. Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data. Our codes and data are publicly available at~\url{https://github.com/RUCAIBox/ReasoningLM}.

翻译：知识图谱问答（KGQA）旨在从大规模知识图谱中为自然语言问题寻找答案实体。为了更好地在知识图谱上进行推理，近期工作通常采用预训练语言模型对问题建模，并基于图神经网络模块在知识图谱上进行多跳推理。尽管效果显著，但由于模型架构的差异性，PLM与GNN并未紧密集成，限制了知识共享与细粒度特征交互。为解决该问题，我们旨在简化上述双模块方法，开发一种能够直接支持KGQA子图推理的更强大PLM——ReasoningLM。在方法中，我们提出子图感知的自注意力机制以模拟GNN执行结构化推理，并采用适配调优策略，通过20,000个子图及合成问题对模型参数进行适配。完成适配后，PLM可通过参数高效微调应用于下游任务。实验表明，ReasoningLM以更少的参数更新量和训练数据量，大幅超越现有最优模型。我们的代码与数据已开源至\url{https://github.com/RUCAIBox/ReasoningLM}。