Existing efforts to improve logical reasoning ability of language models have predominantly relied on supervised fine-tuning, hindering generalization to new domains and/or tasks. The development of Large Langauge Models (LLMs) has demonstrated the capacity of compressing abundant knowledge into a single proxy, enabling them to tackle multiple tasks effectively. Our preliminary experiments, nevertheless, show that LLMs do not show capability on logical reasoning. The performance of LLMs on logical reasoning benchmarks is far behind the existing state-of-the-art baselines. In this paper, we make the first attempt to investigate the feasibility of incorporating logical knowledge through self-supervised post-training, and activating it via in-context learning, which we termed as LogicLLM. Specifically, we devise an auto-regressive objective variant of MERIt and integrate it with two LLM series, i.e., FLAN-T5 and LLaMA, with parameter size ranging from 3 billion to 13 billion. The results on two challenging logical reasoning benchmarks demonstrate the effectiveness of LogicLLM. Besides, we conduct extensive ablation studies to analyze the key factors in designing logic-oriented proxy tasks.

翻译：现有提升语言模型逻辑推理能力的研究主要依赖监督微调，这限制了其对新领域和/或任务的泛化能力。大型语言模型（LLMs）的发展表明，将海量知识压缩至单一代理中可实现多任务有效处理。然而，我们的初步实验显示，LLMs在逻辑推理方面并未展现出相应能力。在逻辑推理基准测试中，LLMs的性能远落后于现有最优基线。本文首次尝试探究通过自监督后训练融入逻辑知识的可行性，并借助上下文学习激活该能力，我们将此方法命名为LogicLLM。具体而言，我们设计了MERIt的自回归目标变体，并将其与参数规模从30亿到130亿的两种LLM系列（即FLAN-T5和LLaMA）集成。在两个具有挑战性的逻辑推理基准上的实验结果验证了LogicLLM的有效性。此外，我们通过大量消融实验分析了设计逻辑导向代理任务的关键因素。