As autonomous agents powered by LLM are increasingly deployed in society, understanding their collective behaviour in social dilemmas becomes critical. We introduce an evaluation framework where LLMs generate strategies encoded as algorithms, enabling inspection prior to deployment and scaling to populations of hundreds of agents -- substantially larger than in previous work. We find that more recent models tend to produce worse societal outcomes compared to older models when agents prioritise individual gain over collective benefits. Using cultural evolution to model user selection of agents, our simulations reveal a significant risk of convergence to poor societal equilibria, particularly when the relative benefit of cooperation diminishes and population sizes increase. We release our code as an evaluation suite for developers to assess the emergent collective behaviour of their models.

翻译：随着由LLM驱动的自主智能体在社会中日益广泛地部署，理解其在社会困境中的集体行为变得至关重要。我们引入了一个评估框架，其中LLM生成编码为算法的策略，从而能够在部署前进行检查，并扩展到数百个智能体的群体规模——这比之前的研究规模大得多。我们发现，当智能体优先考虑个人收益而非集体利益时，较新的模型往往会产生更糟糕的社会结果。通过使用文化演化来模拟用户对智能体的选择，我们的模拟揭示了一个显著的风险：系统可能收敛到不良的社会均衡，尤其是在合作的相对收益减少且群体规模增加的情况下。我们发布了我们的代码，作为一个评估套件，供开发者评估其模型涌现的集体行为。