Recent knowledge enhanced pre-trained language models have shown remarkable performance on downstream tasks by incorporating structured knowledge from external sources into language models. However, they usually suffer from a heterogeneous information alignment problem and a noisy knowledge injection problem. For complex reasoning, the contexts contain rich knowledge that typically exists in complex and sparse forms. In order to model structured knowledge in the context and avoid these two problems, we propose to unify structure reasoning and language model pre-training. It identifies four types of elementary knowledge structures from contexts to construct structured queries, and utilizes the box embedding method to conduct explicit structure reasoning along queries during language modeling. To fuse textual and structured semantics, we utilize contextual language representations of knowledge structures to initialize their box embeddings for structure reasoning. We conduct experiments on complex language reasoning and knowledge graph (KG) reasoning tasks. The results show that our model can effectively enhance the performance of complex reasoning of both language and KG modalities.

翻译：最近的知识强化了培训前语言模式,通过将外部来源的结构化知识纳入语言模式,在下游任务中表现出了显著的成绩,但是,它们通常会遇到信息协调问题和知识注入的烦琐问题。对于复杂的推理,背景中包含着通常以复杂和稀少的形式存在的丰富知识。为了在背景中建模结构化知识,避免这两个问题,我们建议统一结构推理和语言模式的预培训。它确定了四种基本知识结构,从背景到结构化查询,并利用嵌入方法在语言模型的查询中进行明确的结构推理。为了整合文本和结构化的语义学,我们利用知识结构的背景语言表达来启动结构推理的嵌入盒。我们进行了关于复杂的语言推理和知识图(KG)推理任务的实验。结果表明,我们的模型能够有效地提高语言和KG模式复杂推理的绩效。