Large language models (LLMs) have been extensively used as the backbones for general-purpose agents, and some economics literature suggest that LLMs are capable of playing various types of economics games. Following these works, to overcome the limitation of evaluating LLMs using static benchmarks, we propose to explore competitive games as an evaluation for LLMs to incorporate multi-players and dynamicise the environment. By varying the game history revealed to LLMs-based players, we find that most of LLMs are rational in that they play strategies that can increase their payoffs, but not as rational as indicated by Nash Equilibria (NEs). Moreover, when game history are available, certain types of LLMs, such as GPT-4, can converge faster to the NE strategies, which suggests higher rationality level in comparison to other models. In the meantime, certain types of LLMs can win more often when game history are available, and we argue that the winning rate reflects the reasoning ability with respect to the strategies of other players. Throughout all our experiments, we observe that the ability to strictly follow the game rules described by natural languages also vary among the LLMs we tested. In this work, we provide an economics arena for the LLMs research community as a dynamic simulation to test the above-mentioned abilities of LLMs, i.e. rationality, strategic reasoning ability, and instruction-following capability.

翻译：大型语言模型（LLMs）已被广泛用作通用智能体的骨干，一些经济学文献表明，LLMs能够进行多种类型的经济博弈。基于这些工作，为了克服使用静态基准评估LLMs的局限性，我们提出将竞争性博弈作为LLMs的评估手段，以引入多智能体并动态化环境。通过改变向基于LLMs的玩家揭示的博弈历史，我们发现大多数LLMs是理性的，它们会选择能增加自身收益的策略，但其理性程度并未达到纳什均衡（NEs）所指示的水平。此外，当博弈历史可用时，某些类型的LLMs（如GPT-4）能更快地收敛至NE策略，这表明其理性水平高于其他模型。同时，某些类型的LLMs在博弈历史可用时胜率更高，我们认为胜率反映了其针对其他玩家策略的推理能力。在所有实验中，我们观察到，严格遵循自然语言描述的博弈规则的能力在测试的LLMs间也存在差异。在这项工作中，我们为LLMs研究社区提供了一个作为动态模拟的经济竞技场，用于测试LLMs的上述能力，即理性、策略推理能力和指令遵循能力。