Solving the problem of cooperation is fundamentally important for the creation and maintenance of functional societies. Problems of cooperation are omnipresent within human society, with examples ranging from navigating busy road junctions to negotiating treaties. As the use of AI becomes more pervasive throughout society, the need for socially intelligent agents capable of navigating these complex cooperative dilemmas is becoming increasingly evident. Direct punishment is a ubiquitous social mechanism that has been shown to foster the emergence of cooperation in both humans and non-humans. In the natural world, direct punishment is often strongly coupled with partner selection and reputation and used in conjunction with third-party punishment. The interactions between these mechanisms could potentially enhance the emergence of cooperation within populations. However, no previous work has evaluated the learning dynamics and outcomes emerging from Multi-Agent Reinforcement Learning (MARL) populations that combine these mechanisms. This paper addresses this gap. It presents a comprehensive analysis and evaluation of the behaviors and learning dynamics associated with direct punishment, third-party punishment, partner selection, and reputation. Finally, we discuss the implications of using these mechanisms on the design of cooperative AI systems.

翻译：解决合作问题对于创建和维护功能健全的社会至关重要。合作问题在人类社会中无处不在，从繁忙道路交叉口的通行到条约谈判皆是其例。随着人工智能在社会各领域的日益普及，对具备社会智能、能够应对这些复杂合作困境的智能体的需求愈发凸显。直接惩罚作为一种普遍存在的社会机制，已被证明能够促进人类及非人类群体中合作的涌现。在自然界中，直接惩罚常与伙伴选择及声誉机制紧密耦合，并与第三方惩罚结合使用。这些机制间的相互作用可能潜在地增强群体内合作的涌现。然而，此前尚无研究评估结合这些机制的多智能体强化学习（MARL）群体所产生的学习动态与结果。本文填补了这一空白，对直接惩罚、第三方惩罚、伙伴选择及声誉相关的行为与学习动态进行了全面分析与评估。最后，我们探讨了运用这些机制对设计合作型人工智能系统的启示。