Intuitive psychology is a pillar of common-sense reasoning. The replication of this reasoning in machine intelligence is an important stepping-stone on the way to human-like artificial intelligence. Several recent tasks and benchmarks for examining this reasoning in Large-Large Models have focused in particular on belief attribution in Theory-of-Mind tasks. These tasks have shown both successes and failures. We consider in particular a recent purported success case, and show that small variations that maintain the principles of ToM turn the results on their head. We argue that in general, the zero-hypothesis for model evaluation in intuitive psychology should be skeptical, and that outlying failure cases should outweigh average success rates. We also consider what possible future successes on Theory-of-Mind tasks by more powerful LLMs would mean for ToM tasks with people.

翻译：直觉心理学是常识推理的支柱之一。在机器智能中复现这种推理能力，是实现类人人工智能的重要基石。最近针对大型语言模型（LLM）的此类推理能力开展的若干任务和基准测试，特别聚焦于心理理论（Theory-of-Mind, ToM）任务中的信念归因。这些任务既展现了成功案例，也暴露了失败情形。我们着重考察了一个近期声称成功的案例，并表明在保持ToM原则的前提下，微小变动便会导致结果彻底逆转。我们认为，总体而言，在直觉心理学中，模型评估的零假设应持审慎态度，且离群的失败案例比平均成功率更具权重。同时，我们也思考了未来更强大的LLM在心理理论任务上可能取得的成功，对于涉及人类受试者的ToM任务意味着什么。