The reasoning capabilities of LLMs are currently hotly debated. We examine the issue from the perspective of claim/rumour verification. We propose the first logical reasoning framework designed to break down any claim or rumor paired with evidence into the atomic reasoning steps necessary for verification. Based on our framework, we curate two annotated collections of such claim/evidence pairs: a synthetic dataset from Wikipedia and a real-world set stemming from rumours circulating on Twitter. We use them to evaluate the reasoning capabilities of GPT-3.5-Turbo and GPT-4 (hereinafter referred to as ChatGPT) within the context of our framework, providing a thorough analysis. Our results show that ChatGPT struggles in abductive reasoning, although this can be somewhat mitigated by using manual Chain of Thought (CoT) as opposed to Zero Shot (ZS) and ZS CoT approaches. Our study contributes to the growing body of research suggesting that ChatGPT's reasoning processes are unlikely to mirror human-like reasoning, and that LLMs need to be more rigorously evaluated in order to distinguish between hype and actual capabilities, especially in high stake real-world tasks such as claim verification.

翻译：大型语言模型（LLM）的推理能力目前备受争议。我们从声明/谣言验证的视角探讨这一问题。本文首次提出逻辑推理框架，旨在将任意声明或谣言及其配套证据分解为验证所需的原子化推理步骤。基于这一框架，我们构建了两个带标注的声明-证据对数据集：一个源自维基百科的合成数据集，以及一个基于Twitter传播谣言的真实世界数据集。我们利用这些数据集，在该框架下系统性评估了GPT-3.5-Turbo与GPT-4（统称ChatGPT）的推理能力，并进行了深入分析。结果表明，ChatGPT在溯因推理方面表现不佳，尽管通过人工链式思维（CoT）提示方法相比零样本（ZS）和零样本链式思维（ZS CoT）方法能部分缓解这一缺陷。本研究进一步印证了相关研究结论：ChatGPT的推理过程难以模拟人类思维模式，且针对LLM的评估需要更为严谨，以区分其实际能力与宣传夸大成分——特别是在事实验证这类高风险现实任务中。