Multi-agent reinforcement learning (MARL) is a powerful tool for training automated systems acting independently in a common environment. However, it can lead to sub-optimal behavior when individual incentives and group incentives diverge. Humans are remarkably capable at solving these social dilemmas. It is an open problem in MARL to replicate such cooperative behaviors in selfish agents. In this work, we draw upon the idea of formal contracting from economics to overcome diverging incentives between agents in MARL. We propose an augmentation to a Markov game where agents voluntarily agree to binding state-dependent transfers of reward, under pre-specified conditions. Our contributions are theoretical and empirical. First, we show that this augmentation makes all subgame-perfect equilibria of all fully observed Markov games exhibit socially optimal behavior, given a sufficiently rich space of contracts. Next, we complement our game-theoretic analysis by showing that state-of-the-art RL algorithms learn socially optimal policies given our augmentation. Our experiments include classic static dilemmas like Stag Hunt, Prisoner's Dilemma and a public goods game, as well as dynamic interactions that simulate traffic, pollution management and common pool resource management.

翻译：多智能体强化学习（Multi-agent Reinforcement Learning，MARL）是训练自主系统在共同环境中独立行动的有力工具。然而，当个体激励与群体激励出现分歧时，该方法可能导致次优行为。人类在解决这些社会困境方面具有显著能力。如何在MARL中让自私智能体复现此类合作行为仍是一个开放问题。本文借鉴经济学中的正式契约思想，旨在克服MARL中智能体间的激励分歧。我们提出一种对马尔可夫博弈的增强方法：智能体在预设条件下自愿同意具有状态依赖性的有约束奖励转移。我们的贡献兼具理论性与实证性。首先，我们证明：在合同空间足够丰富的前提下，该增强方法能使所有完全可观测马尔可夫博弈的子博弈完美均衡均呈现社会最优行为。其次，我们通过博弈论分析补充说明，采用本增强方法后，最先进的强化学习算法能学习到社会最优策略。实验涵盖经典静态困境（如猎鹿博弈、囚徒困境及公共品博弈），以及模拟交通、污染管理和公共池塘资源管理的动态交互场景。