As Large Language Model (LLM)-based agents increasingly undertake real-world tasks and engage with human society, how well do we understand their behaviors? We (1) investigate how LLM agents' prosocial behaviors -- a fundamental social norm -- can be induced by different personas and benchmarked against human behaviors; and (2) introduce a behavioral and social science approach to evaluate LLM agents' decision-making. We explored how different personas and experimental framings affect these AI agents' altruistic behavior in dictator games and compared their behaviors within the same LLM family, across various families, and with human behaviors. The findings reveal substantial variations and inconsistencies among LLMs and notable differences compared to human behaviors. Merely assigning a human-like identity to LLMs does not produce human-like behaviors. Despite being trained on extensive human-generated data, these AI agents are unable to capture the internal processes of human decision-making. Their alignment with human is highly variable and dependent on specific model architectures and prompt formulations; even worse, such dependence does not follow a clear pattern. LLMs can be useful task-specific tools but are not yet intelligent human-like agents.

翻译：随着基于大语言模型（LLM）的智能体越来越多地承担现实世界任务并融入人类社会，我们对其行为的理解程度如何？本研究（1）探讨了LLM智能体的亲社会行为——一种基本社会规范——如何被不同角色设定所诱导，并以人类行为为基准进行衡量；（2）引入行为与社会科学的研究方法来评估LLM智能体的决策机制。我们通过独裁者博弈探究了不同角色设定和实验框架如何影响这些AI智能体的利他行为，并在同一LLM家族内部、不同家族之间以及与人类行为进行了系统比较。研究结果显示：不同LLM之间存在显著差异与不一致性，与人类行为相比也存在明显区别。仅赋予LLM类人身份并不能产生类人行为。尽管这些AI智能体接受了海量人类生成数据的训练，它们仍无法捕捉人类决策的内在过程。其与人类行为的对齐度具有高度可变性，且严重依赖于特定模型架构与提示词设计；更严重的是，这种依赖性并未呈现清晰规律。LLM可作为特定任务的有效工具，但尚未成为具有类人智能的智能体。