We study a synthetic corpus based approach for language models (LMs) to acquire logical deductive reasoning ability. The previous studies generated deduction examples using specific sets of deduction rules. However, these rules were limited or otherwise arbitrary, limiting the generalizability of acquired reasoning ability. We rethink this and adopt a well-grounded set of deduction rules based on formal logic theory, which can derive any other deduction rules when combined in a multistep way. Then, using the proposed corpora, which we name FLD (Formal Logic Deduction), we first evaluate and analyze the logical reasoning ability of the latest LLMs. Even GPT-4 can solve only half of the problems, suggesting that pure logical reasoning isolated from knowledge is still challenging for the LLMs, and additional training specialized in logical reasoning is indeed essential. We next empirically verify that LMs trained on FLD corpora acquire more generalizable reasoning ability. Furthermore, we identify the aspects of reasoning ability on which deduction corpora can enhance LMs and those on which they cannot, and discuss future directions on each aspect. The released corpora serve both as learning resources and as challenging benchmarks.

翻译：我们研究了一种基于合成语料库的方法，使语言模型（LMs）获得逻辑演绎推理能力。先前的研究使用特定的演绎规则集生成推理示例。然而，这些规则要么受限，要么具有随意性，限制了所获推理能力的泛化性。我们对此进行反思，采用基于形式逻辑理论的一套根基扎实的演绎规则，这些规则在多步组合时能够推导出任何其他演绎规则。然后，利用所提出的语料库——我们将其命名为FLD（形式逻辑演绎），我们首先评估并分析了最新大型语言模型（LLMs）的逻辑推理能力。即便是GPT-4也只能解决一半的问题，这表明脱离知识的纯逻辑推理对LLMs而言仍然具有挑战性，专门针对逻辑推理的额外训练确实至关重要。接下来，我们通过实验验证了在FLD语料库上训练的LMs获得了更具泛化性的推理能力。此外，我们明确了演绎语料库能够提升LMs的推理能力方面以及无法提升的方面，并针对每个方面讨论了未来方向。所发布的语料库既可作为学习资源，也可作为具有挑战性的基准测试。