Eleven Large Language Models (LLMs) were assessed using a custom-made battery of false-belief tasks, considered a gold standard in testing Theory of Mind (ToM) in humans. The battery included 640 prompts spread across 40 diverse tasks, each one including a false-belief scenario, three closely matched true-belief control scenarios, and the reversed versions of all four. To solve a single task, a model needed to correctly answer 16 prompts across all eight scenarios. Smaller and older models solved no tasks; GPT-3-davinci-003 (from November 2022) and ChatGPT-3.5-turbo (from March 2023) solved 20% of the tasks; ChatGPT-4 (from June 2023) solved 75% of the tasks, matching the performance of six-year-old children observed in past studies. We explore the potential interpretation of these findings, including the intriguing possibility that ToM, previously considered exclusive to humans, may have spontaneously emerged as a byproduct of LLMs' improving language skills.

翻译：采用定制化的错误信念任务（公认的人类心理理论测试金标准）对十一个大语言模型进行评估。实验包含640个提示词，分布在40项多样化任务中，每项任务包含一个错误信念场景、三个高度匹配的真实信念对照场景，以及所有四个场景的反向版本。模型需正确回答全部八个场景中的16个提示词方能完成单任务。小型及过时模型未能完成任何任务；GPT-3-davinci-003（2022年11月版）与ChatGPT-3.5-turbo（2023年3月版）完成20%任务；ChatGPT-4（2023年6月版）完成75%任务，达到过往研究中六岁儿童的表现水平。我们探讨这些发现的潜在解释，包括一个引人深思的可能性：曾被认为人类独有的心理理论，或已作为大语言模型语言能力提升的副产品而自发涌现。