Multi-agent reinforcement learning (MARL) has been shown effective for cooperative games in recent years. However, existing state-of-the-art methods face challenges related to sample inefficiency, brittleness regarding hyperparameters, and the risk of converging to a suboptimal Nash Equilibrium. To resolve these issues, in this paper, we propose a novel theoretical framework, named Maximum Entropy Heterogeneous-Agent Mirror Learning (MEHAML), that leverages the maximum entropy principle to design maximum entropy MARL actor-critic algorithms. We prove that algorithms derived from the MEHAML framework enjoy the desired properties of the monotonic improvement of the joint maximum entropy objective and the convergence to quantal response equilibrium (QRE). The practicality of MEHAML is demonstrated by developing a MEHAML extension of the widely used RL algorithm, HASAC (for soft actor-critic), which shows significant improvements in exploration and robustness on three challenging benchmarks: Multi-Agent MuJoCo, StarCraftII, and Google Research Football. Our results show that HASAC outperforms strong baseline methods such as HATD3, HAPPO, QMIX, and MAPPO, thereby establishing the new state of the art. See our project page at https://sites.google.com/view/mehaml.

翻译：近年来，多智能体强化学习（MARL）在合作博弈中表现出有效性。然而，现有最先进方法面临样本效率低下、超参数脆弱性以及收敛至次优纳什均衡的风险等挑战。为解决这些问题，本文提出了一种名为最大熵异质智能体镜像学习（MEHAML）的新型理论框架，该框架利用最大熵原理设计最大熵MARL演员-评论家算法。我们证明了从MEHAML框架推导出的算法具有联合最大熵目标的单调改进以及收敛至量化响应均衡（QRE）的理想性质。通过开发广泛使用的RL算法HASAC（软演员-评论家）的MEHAML扩展，我们展示了MEHAML的实用性，该算法在三个具有挑战性的基准测试（多智能体MuJoCo、星际争霸II和谷歌研究足球）中在探索性和鲁棒性方面表现出显著改进。我们的结果表明，HASAC优于HATD3、HAPPO、QMIX和MAPPO等强基线方法，从而确立了新的最先进水平。项目页面详见https://sites.google.com/view/mehaml。