The growing capabilities and increasingly widespread deployment of AI systems necessitate robust benchmarks for measuring their cooperative capabilities. Unfortunately, most multi-agent benchmarks are either zero-sum or purely cooperative, providing limited opportunities for such measurements. We introduce a general-sum variant of the zero-sum board game Diplomacy -- called Welfare Diplomacy -- in which players must balance investing in military conquest and domestic welfare. We argue that Welfare Diplomacy facilitates both a clearer assessment of and stronger training incentives for cooperative capabilities. Our contributions are: (1) proposing the Welfare Diplomacy rules and implementing them via an open-source Diplomacy engine; (2) constructing baseline agents using zero-shot prompted language models; and (3) conducting experiments where we find that baselines using state-of-the-art models attain high social welfare but are exploitable. Our work aims to promote societal safety by aiding researchers in developing and assessing multi-agent AI systems. Code to evaluate Welfare Diplomacy and reproduce our experiments is available at https://github.com/mukobi/welfare-diplomacy.

翻译：随着AI系统能力的不断增强和日益广泛的部署，亟需稳健的基准来衡量其合作能力。然而，大多数多智能体基准要么是零和博弈，要么是完全合作博弈，为这类测量提供的空间有限。我们引入了一种零和棋盘游戏《外交》的广义和变体——称为“福利外交”——在该游戏中，玩家必须平衡对军事征服和国内福利的投入。我们认为，“福利外交”既有助于更清晰地评估合作能力，也能提供更强的训练激励。我们的贡献包括：(1) 提出“福利外交”规则并通过开源《外交》引擎实现；(2) 使用零样本提示的语言模型构建基线智能体；(3) 开展实验，发现使用最先进模型的基线智能体虽然实现了较高的社会福利，但容易被利用。我们的工作旨在通过帮助研究人员开发和评估多智能体AI系统，促进社会安全。评估“福利外交”及复现我们实验的代码可在 https://github.com/mukobi/welfare-diplomacy 获取。