Recent research on transformer-based language models investigates their reasoning ability over logical rules expressed in natural language text. However, their logic is not yet well-understood as we cannot explain the abstractions made by the models that help them in reasoning. These models are criticized for merely memorizing complex patterns in the data, which often creates issues for their generalizability in unobserved situations. In this work, we analyze the use of probabilistic logical rules in transformer-based language models. In particular, we propose a new approach, Probabilistic Constraint Training (PCT), that explicitly models probabilistic logical reasoning by imposing the rules of reasoning as constraints during training. We create a new QA benchmark for evaluating probabilistic reasoning over uncertain textual rules, which creates instance-specific rules, unlike the only existing relevant benchmark. Experimental results show that our proposed technique improves the base language models' accuracy and explainability when probabilistic logical reasoning is required for question answering. Moreover, we show that the learned probabilistic reasoning abilities are transferable to novel situations.

翻译：近期针对基于Transformer的语言模型的研究，探讨了它们在自然语言文本表达的逻辑规则上进行推理的能力。然而，由于我们无法解释这些模型在推理中所形成的抽象概念，其逻辑机制尚未得到充分理解。这些模型被批评为仅仅记忆数据中的复杂模式，这常常导致其在未见情景中泛化能力不足。在本研究中，我们分析了概率逻辑规则在基于Transformer的语言模型中的应用。具体而言，我们提出了一种新方法——概率约束训练（PCT），该方法通过在训练过程中将推理规则作为约束条件，显式建模概率逻辑推理。我们创建了一个新的问答基准，用于评估在不确定文本规则下的概率推理能力，该基准生成了实例特定的规则，不同于现有唯相关基准。实验结果表明，当问题回答需借助概率逻辑推理时，我们的技术提升了基础语言模型的准确性与可解释性。此外，我们证明所习得的概率推理能力可迁移至新颖情景。