Large language model (LLM) simulations of human behavior have the potential to revolutionize the social and behavioral sciences, if and only if they faithfully reflect real human behaviors. Current evaluations of simulation fidelity are fragmented, based on bespoke tasks and metrics, creating a patchwork of incomparable results. To address this, we introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation. By unifying 20 diverse datasets covering tasks from moral decision-making to economic choice across a large global participant pool, SimBench provides the necessary foundation to ask fundamental questions about when, how, and why LLM simulations succeed or fail. We show that the best LLMs today achieve meaningful but modest simulation fidelity (score: 40.80/100), with performance scaling log-linearly with model size but not with increased inference-time compute. We discover an alignment-simulation tradeoff: instruction tuning improves performance on low-entropy (consensus) questions but degrades it on high-entropy (diverse) ones. Models particularly struggle when simulating specific demographic groups. Finally, we demonstrate that simulation ability correlates most strongly with knowledge-intensive reasoning (MMLU-Pro, r = 0.939). By making progress measurable, we aim to accelerate the development of more faithful LLM simulators.

翻译：大语言模型（LLM）对人类行为的模拟有望彻底改变社会与行为科学，但前提是其必须真实反映实际人类行为。当前对模拟保真度的评估存在碎片化问题，依赖定制化任务与指标，导致结果缺乏可比性。为此，我们提出SimBench——首个大规模、标准化的LLM模拟科学鲁棒可重复性基准。通过整合涵盖道德决策到经济选择等任务的20个多样化数据集（参与者遍布全球），SimBench为探索LLM模拟何时、如何及为何成功或失败等根本性问题奠定了必要基础。研究表明，当前最优LLM仅能达到中等水平的模拟保真度（得分：40.80/100），其性能随模型规模呈对数线性增长，但不受推理时算力增强的影响。我们还发现一种对齐-模拟权衡：指令微调可提升低熵（共识性）问题的性能，却会降低高熵（多样性）问题的表现。模型在模拟特定人口群体时尤为困难。最后，模拟能力与知识密集型推理（MMLU-Pro，r=0.939）呈现最强相关性。通过使进展可量化，我们旨在加速更逼真的LLM模拟器研发进程。