As Large Language Models (LLMs) become increasingly integrated into our everyday lives, understanding their ability to comprehend human mental states becomes critical for ensuring effective interactions. However, despite the recent attempts to assess the Theory-of-Mind (ToM) reasoning capabilities of LLMs, the degree to which these models can align with human ToM remains a nuanced topic of exploration. This is primarily due to two distinct challenges: (1) the presence of inconsistent results from previous evaluations, and (2) concerns surrounding the validity of existing evaluation methodologies. To address these challenges, we present a novel framework for procedurally generating evaluations with LLMs by populating causal templates. Using our framework, we create a new social reasoning benchmark (BigToM) for LLMs which consists of 25 controls and 5,000 model-written evaluations. We find that human participants rate the quality of our benchmark higher than previous crowd-sourced evaluations and comparable to expert-written evaluations. Using BigToM, we evaluate the social reasoning capabilities of a variety of LLMs and compare model performances with human performance. Our results suggest that GPT4 has ToM capabilities that mirror human inference patterns, though less reliable, while other LLMs struggle.

翻译：随着大型语言模型日益融入日常生活，理解它们对人类心理状态的认知能力，对于确保有效交互至关重要。然而，尽管近期有研究尝试评估大语言模型的“心智理论”推理能力，但这些模型能在多大程度上与人类心智理论对齐，仍是一个需要精细探索的议题。这主要源于两个不同的挑战：（1）以往评估结果存在不一致性；（2）现有评估方法的有效性存疑。为应对这些挑战，我们提出了一种新型框架，通过填充因果模板，利用大语言模型程序化生成评估任务。基于该框架，我们构建了一个名为BigToM的新社交推理基准，包含25个控制任务和5000个模型生成的评估任务。我们发现，人类参与者对该基准质量的评价高于以往众包评估，且与专家撰写的评估相当。利用BigToM，我们评估了多种大语言模型的社交推理能力，并将其性能与人类表现对比。研究结果表明，GPT4具备与人类推理模式相似的心智理论能力，尽管可靠性稍低，而其他大语言模型在此任务上表现不佳。