Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.

翻译：具备逻辑推理能力是一项综合性的自然语言理解任务。随着被标榜为“先进”推理能力的生成式预训练Transformer 4（GPT-4）的发布，我们迫切希望了解GPT-4在各种逻辑推理任务上的表现。本报告分析了多个逻辑推理数据集，包括LogiQA和ReClor等主流基准，以及AR-LSAT等新近发布的数据集。我们利用需要逻辑推理能力的基准测试，检验了多项选择阅读理解与自然语言推理任务。此外，我们构建了一个逻辑推理分布外数据集，以探究ChatGPT与GPT-4的鲁棒性。我们还对ChatGPT与GPT-4的性能进行了对比。实验结果表明，在大多数逻辑推理基准上，ChatGPT的表现显著优于基于RoBERTa的微调方法。借助对GPT-4 API的早期访问权限，我们对GPT-4模型进行了密集实验。结果显示，GPT-4在大多数逻辑推理数据集上取得了更高的性能。在基准测试中，ChatGPT与GPT-4在LogiQA和ReClor等知名数据集上表现相对良好。然而，在处理新发布及分布外数据集时，其性能显著下降。逻辑推理对ChatGPT与GPT-4而言仍具挑战性，尤其是在分布外和自然语言推理数据集上。我们将基于提示风格的逻辑推理数据集作为基准套件发布，并命名为LogiEval。