Recently, large language models (LLMs), including notable models such as GPT-4 and burgeoning community models, have showcased significant general language understanding abilities. However, there has been a scarcity of attempts to assess the logical reasoning capacities of these LLMs, an essential facet of natural language understanding. To encourage further investigation in this area, we introduce GLoRE, a meticulously assembled General Logical Reasoning Evaluation benchmark comprised of 12 datasets that span three different types of tasks. Our experimental results show that compared to the performance of human and supervised fine-tuning, the logical reasoning capabilities of open LLM models necessitate additional improvement; ChatGPT and GPT-4 show a strong capability of logical reasoning, with GPT-4 surpassing ChatGPT by a large margin. We propose a self-consistency probing method to enhance the accuracy of ChatGPT and a fine-tuned method to boost the performance of an open LLM. We release the datasets and evaluation programs to facilitate future research.

翻译：近期，包括GPT-4等知名模型和新兴社区模型在内的大型语言模型（LLMs）展现了显著的通用语言理解能力。然而，针对这些LLMs逻辑推理能力（自然语言理解的关键维度）的评估尝试仍较为稀缺。为促进该领域的进一步研究，我们提出了GLoRE——一个精心整合的通用逻辑推理评估基准，包含12个数据集，覆盖三种不同类型的任务。实验结果表明，与人类表现和监督微调相比，开放LLM模型的逻辑推理能力仍需额外提升；ChatGPT和GPT-4展现出强大的逻辑推理能力，其中GPT-4以显著优势超越ChatGPT。我们提出了一种自一致性探测方法以提高ChatGPT的准确性，并采用微调方法提升开放LLM的性能。我们已公开数据集和评估程序，以推动未来研究进展。