We study syllogistic reasoning in LLMs from the logical and natural language perspectives. In process, we explore fundamental reasoning capabilities of the LLMs and the direction this research is moving forward. To aid in our studies, we use 14 large language models and investigate their syllogistic reasoning capabilities in terms of symbolic inferences as well as natural language understanding. Even though this reasoning mechanism is not a uniform emergent property across LLMs, the perfect symbolic performances in certain models make us wonder whether LLMs are becoming more and more formal reasoning mechanisms, rather than making explicit the nuances of human reasoning.

翻译：本研究从逻辑学与自然语言处理的双重视角，探讨大语言模型中的三段论推理机制。在此过程中，我们系统考察了大语言模型的基础推理能力及其研究发展趋势。为支撑研究，我们选取了14个大型语言模型，从符号推理与自然语言理解两个维度评估其三段论推理性能。尽管该推理机制并非所有大语言模型普遍涌现的特性，但部分模型在符号推理任务上的完美表现促使我们思考：大语言模型是否正逐渐演化为形式化推理机制，而非精确复现人类推理的微妙特征。