Recent pre-trained language models (PLMs) equipped with foundation reasoning skills have shown remarkable performance on downstream complex tasks. However, the significant structure reasoning skill has been rarely studied, which involves modeling implicit structure information within the text and performing explicit logical reasoning over them to deduce the conclusion. This paper proposes a unified learning framework that combines explicit structure reasoning and language pre-training to endow PLMs with the structure reasoning skill. It first identifies several elementary structures within contexts to construct structured queries and performs step-by-step reasoning along the queries to identify the answer entity. The fusion of textual semantics and structure reasoning is achieved by using contextual representations learned by PLMs to initialize the representation space of structures, and performing stepwise reasoning on this semantic representation space. Experimental results on four datasets demonstrate that the proposed model achieves significant improvements in complex reasoning tasks involving diverse structures, and shows transferability to downstream tasks with limited training data and effectiveness for complex reasoning of KGs modality.

翻译：近期，具备基础推理能力的预训练语言模型（PLMs）在复杂下游任务中展现出显著性能。然而，涉及文本隐含结构信息建模并在此基础上进行显式逻辑推理以得出结论的结构推理能力却鲜有研究。本文提出一种融合显式结构推理与语言预训练的统一学习框架，旨在赋予PLMs结构推理能力。该框架首先识别文本中的若干基础结构以构建结构化查询，并沿查询逐步推理以定位答案实体。通过利用PLMs学习到的上下文表示初始化结构表示空间，并在该语义表示空间上执行逐步推理，实现文本语义与结构推理的融合。在四个数据集上的实验结果表明，所提模型在处理包含多种结构的复杂推理任务时取得显著提升，在训练数据受限的下游任务中展现出可迁移性，并有效支持知识图谱模态的复杂推理。