Large language models (LLMs) are increasingly deployed to simulate human collective behaviors, yet the methodological rigor of these "AI societies" remains under-explored. Through a systematic audit of 39 recent studies, we identify six pervasive flaws-spanning agent profiles, interaction, memory, control, unawareness, and realism (PIMMUR). Our analysis reveals that 89.7% of studies violate at least one principle, undermining simulation validity. We demonstrate that frontier LLMs correctly identify the underlying social experiment in 50.8% of cases, while 61.0% of prompts exert excessive control that pre-determines outcomes. By reproducing five representative experiments (e.g., telephone game), we show that reported collective phenomena often vanish or reverse when PIMMUR principles are enforced, suggesting that many "emergent" behaviors are methodological artifacts rather than genuine social dynamics. Our findings suggest that current AI simulations may capture model-specific biases rather than universal human social behaviors, raising critical concerns about the use of LLMs as scientific proxies for human society.

翻译：大型语言模型（LLM）被越来越多地用于模拟人类集体行为，然而这些“AI社会”的方法论严谨性仍未得到充分探索。通过对39项近期研究的系统审计，我们识别出六大普遍缺陷——涵盖智能体画像、交互、记忆、控制、无意识性和现实性（PIMMUR）。分析表明，89.7%的研究至少违反一项原则，损害了模拟有效性。我们证明，前沿LLM在50.8%的案例中正确识别了潜在的社会实验，而61.0%的提示词施加了过度控制，预先决定了结果。通过复现五个代表性实验（例如电话游戏），我们发现当PIMMUR原则被强制执行时，所报告的集体现象往往消失或逆转，这表明许多“涌现”行为是方法论伪影而非真实社会动态。我们的发现表明，当前AI模拟可能捕捉到的是模型特定偏差，而非普遍的人类社会行为，这引发了对LLM作为人类社会科学代理的关键性质疑。