The existing methods for evaluating the inference abilities of Large Language Models (LLMs) have been results-centric, making it difficult to assess the inference process. We introduce a new approach using the Abstract and Reasoning Corpus (ARC) dataset to evaluate the inference and contextual understanding abilities of large language models in a process-centric manner. ARC demands rigorous logical structures for problem-solving, making it a benchmark that facilitates the comparison of model inference abilities with humans. Experimental results confirm that while large language models possess weak inference abilities, they still lag in terms of logical coherence, compositionality, and productivity. Our experiments highlight the reasoning capabilities of LLMs, proposing development paths for achieving human-level reasoning.

翻译：现有评估大型语言模型推理能力的方法以结果为中心，难以评估推理过程。我们提出一种新方法，利用抽象与推理语料库数据集，以过程为中心的方式评估大型语言模型的推理能力和语境理解能力。该数据集要求问题解决具备严谨的逻辑结构，因此成为促进模型推理能力与人类推理能力比较的基准。实验结果表明，尽管大型语言模型具备较弱的推理能力，但在逻辑连贯性、组合性和生产性方面仍存在差距。我们的实验凸显了大型语言模型的推理潜力，并提出了实现人类水平推理的发展路径。