Multi-agent reinforcement learning (MARL) addresses sequential decision-making problems with multiple agents, where each agent optimizes its own objective. In many real-world instances, the agents may not only want to optimize their objectives, but also ensure safe behavior. For example, in traffic routing, each car (agent) aims to reach its destination quickly (objective) while avoiding collisions (safety). Constrained Markov Games (CMGs) are a natural formalism for safe MARL problems, though generally intractable. In this work, we introduce and study Constrained Markov Potential Games (CMPGs), an important class of CMGs. We first show that a Nash policy for CMPGs can be found via constrained optimization. One tempting approach is to solve it by Lagrangian-based primal-dual methods. As we show, in contrast to the single-agent setting, however, CMPGs do not satisfy strong duality, rendering such approaches inapplicable and potentially unsafe. To solve the CMPG problem, we propose our algorithm Coordinate-Ascent for CMPGs (CA-CMPG), which provably converges to a Nash policy in tabular, finite-horizon CMPGs. Furthermore, we provide the first sample complexity bounds for learning Nash policies in unknown CMPGs, and, which under additional assumptions, guarantee safe exploration.

翻译：多智能体强化学习（MARL）解决涉及多个智能体的序贯决策问题，其中每个智能体优化自身目标。在许多现实场景中，智能体不仅需要优化目标，还需确保行为的安全性。例如，在交通路径规划中，每辆汽车（智能体）既要快速到达目的地（目标），又要避免碰撞（安全）。约束马尔可夫博弈（CMG）是安全MARL问题的自然形式化框架，但通常难以处理。本文引入并研究了一类重要的CMG——约束马尔可夫势博弈（CMPG）。我们首先证明，CMPG的纳什策略可通过约束优化求得。一种颇具吸引力的解法是基于拉格朗日法的原始-对偶方法。然而，与单智能体场景不同，CMPG不满足强对偶性，导致此类方法不可行且可能不安全。为解决CMPG问题，我们提出CMPG坐标上升算法（CA-CMPG），该算法在表格型有限时域CMPG中可证明收敛到纳什策略。此外，我们首次给出了在未知CMPG中学习纳什策略的样本复杂度界，并在额外假设下保障安全探索。