Can Large Language Models (LLMs) simulate human behavior in complex environments? LLMs have recently been shown to exhibit advanced reasoning skills but much of NLP evaluation still relies on static benchmarks. Answering this requires evaluation environments that probe strategic reasoning in competitive, dynamic scenarios that involve long-term planning. We introduce AucArena, a novel simulation environment for evaluating LLMs within auctions, a setting chosen for being highly unpredictable and involving many skills related to resource and risk management, while also being easy to evaluate. We conduct several controlled simulations using state-of-the-art LLMs as bidding agents. We find that through simple prompting, LLMs do indeed demonstrate many of the skills needed for effectively engaging in auctions (e.g., managing budget, adhering to long-term goals and priorities), skills that we find can be sharpened by explicitly encouraging models to be adaptive and observe strategies in past auctions. These results are significant as they show the potential of using LLM agents to model intricate social dynamics, especially in competitive settings. However, we also observe considerable variability in the capabilities of individual LLMs. Notably, even our most advanced models (GPT-4) are occasionally surpassed by heuristic baselines and human agents, highlighting the potential for further improvements in the design of LLM agents and the important role that our simulation environment can play in further testing and refining agent architectures.

翻译：大型语言模型能否在复杂环境中模拟人类行为？尽管近期研究表明大型语言模型展现出高级推理能力，但自然语言处理领域的多数评估仍依赖静态基准。要解答这一问题，需要构建能够探测模型在涉及长期规划的竞争性动态场景中战略推理能力的评估环境。我们提出AucArena——一个专为评估拍卖场景下大型语言模型表现而设计的新型模拟环境。该场景因其高度不可预测性、涉及资源与风险管理等多重技能且易于评估而被选用。我们采用当前最先进的大型语言模型作为竞价代理进行了多组受控模拟实验。研究发现，通过简单提示工程，大型语言模型确实展现出参与拍卖所需的多项关键能力（如预算管理、长期目标与优先级遵循），而这些能力可通过明确鼓励模型保持适应性并观察历史拍卖策略得到强化。这一发现意义重大，表明利用大型语言模型代理建模复杂社会动态（尤其在竞争性场景中）具有可行性。然而，我们也观察到不同大型语言模型个体能力存在显著差异。值得注意的是，即便是最先进的模型（如GPT-4）有时也会被启发式基线方法和人类代理超越，这凸显了大型语言模型代理架构的改进空间，以及我们的模拟环境在进一步测试与优化代理架构方面的重要作用。