In this paper, we evaluate the capability of transformer-based language models in making inferences over uncertain text that includes uncertain rules of reasoning. We cover both Pre-trained Language Models (PLMs) and generative Large Language Models (LLMs). Our evaluation results show that both generations of language models struggle with reasoning over uncertain text. We propose a novel end-to-end fine-tuning approach, Probabilistic Constraint Training (PCT), that utilizes probabilistic logical rules as constraints in the fine-tuning phase without relying on these rules in the inference stage. To assess the effectiveness of PCT, we utilize the related corpora and, additionally, create a new and more challenging benchmark that, unlike the previous ones, uses instance-specific rules. Our study demonstrates that PCT improves the transformer-based language model's intrinsic reasoning and makes their probabilistic logical reasoning process more explicit and explainable. Furthermore, PCT equips these models to effectively handle novel situations, including higher reasoning depth, new domains, and complex probabilistic structures.

翻译：本文评估了基于Transformer的语言模型在包含不确定性推理规则的文本中进行推理的能力。我们涵盖了预训练语言模型（PLMs）和生成式大语言模型（LLMs）。评估结果表明，这两代语言模型在处理不确定性文本的推理时均存在困难。我们提出了一种新颖的端到端微调方法——概率约束训练（PCT），该方法在微调阶段利用概率逻辑规则作为约束，而无需在推理阶段依赖这些规则。为评估PCT的有效性，我们使用了相关语料库，并额外创建了一个更具挑战性的新基准，该基准与以往不同，采用了实例特定的规则。研究表明，PCT提升了基于Transformer的语言模型的内在推理能力，并使其概率逻辑推理过程更加明确和可解释。此外，PCT使这些模型能够有效应对新情境，包括更深的推理层次、新领域及复杂的概率结构。