Beyond the great cognitive powers showcased by language models, it is crucial to scrutinize whether their reasoning capabilities stem from strong generalization or merely exposure to relevant data. As opposed to constructing increasingly complex logic, this paper probes into the boolean logic, the root capability of a logical reasoner. We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested boolean logic, a task that humans can handle with ease. To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method \textit{Curriculum Logical Reasoning} (\textsc{Clr}), where we augment the training data with nested boolean logic chain step-by-step, and program the training from simpler logical patterns gradually to harder ones. This new training paradigm allows language models to effectively generalize to much harder and longer-hop logic, which can hardly be learned through naive training. Furthermore, we show that boolean logic is a great foundation for improving the subsequent general logical tasks.

翻译：超越语言模型展现的强大认知能力，关键需审视其推理能力究竟是源于强泛化还是仅因接触相关数据。不同于构建日益复杂的逻辑，本文聚焦于逻辑推理器的根基能力——布尔逻辑。我们发现任何预训练语言模型（包括大语言模型）在面对人类能轻松处理的多重嵌套布尔逻辑任务时，仅表现出随机选择的行为。为赋能语言模型这种基础能力，本文提出一种新的自监督学习方法——课程逻辑推理（\textsc{Clr}），通过逐步增广含嵌套布尔逻辑链的训练数据，并设计从简单逻辑模式渐进至困难模式的训练流程。这种新训练范式使语言模型能够有效泛化至更难、更长跳数的逻辑，而这在朴素训练中几乎无法习得。此外，我们证明布尔逻辑是提升后续通用逻辑任务的重要基础。