Recent advances in large language models (LLMs) have transformed the field of natural language processing (NLP). From GPT-3 to PaLM, the state-of-the-art performance on natural language tasks is being pushed forward with every new large language model. Along with natural language abilities, there has been a significant interest in understanding whether such models exhibit reasoning capabilities with the use of reasoning benchmarks. However, even though results are seemingly positive, these benchmarks prove to be simplistic in nature and the performance of LLMs on these benchmarks cannot be used as evidence to support, many a times outlandish, claims being made about LLMs' reasoning capabilities. Further, these only represent a very limited set of simple reasoning tasks and we need to look at more sophisticated reasoning problems if we are to measure the true limits of such LLM-based systems. Motivated by this, we propose an extensible assessment framework to test the capabilities of LLMs on reasoning about actions and change, a central aspect of human intelligence. We provide multiple test cases that are more involved than any of the previously established benchmarks and each test case evaluates a different aspect of reasoning about actions and change. Results on GPT-3 (davinci), Instruct-GPT3 (text-davinci-002) and BLOOM (176B), showcase subpar performance on such reasoning tasks.

翻译：近期大型语言模型（LLM）的进步已彻底改变了自然语言处理（NLP）领域。从GPT-3到PaLM，每个新的大型语言模型都在推动自然语言任务的最优性能。除自然语言能力外，学界对这类模型是否具备推理能力产生了浓厚兴趣，并采用推理基准进行评估。然而，尽管结果看似积极，这些基准实际上过于简单，LLM在此类基准上的表现并不能作为支持（往往言过其实的）LLM推理能力论断的证据。更关键的是，这类基准仅代表极有限的简单推理任务，若要衡量此类基于LLM系统的真实极限，必须考察更复杂的推理问题。基于此，我们提出一个可扩展的评估框架，专门测试LLM对人类智能核心要素——行动与变化推理——的能力。我们设计了多个比现有基准更复杂的测试案例，每个案例针对行动与变化推理的不同方面进行评估。针对GPT-3（davinci）、Instruct-GPT3（text-davinci-002）和BLOOM（176B）的测试结果显示，这些模型在此类推理任务上的表现欠佳。