We propose a multi-agent reinforcement learning dynamics, and analyze its convergence in infinite-horizon discounted Markov potential games. We focus on the independent and decentralized setting, where players do not have knowledge of the game model and cannot coordinate. In each stage, players update their estimate of Q-function that evaluates their total contingent payoff based on the realized one-stage reward in an asynchronous manner. Then, players independently update their policies by incorporating an optimal one-stage deviation strategy based on the estimated Q-function. A key feature of the learning dynamics is that the Q-function estimates are updated at a faster timescale than the policies. We prove that the policies induced by our learning dynamics converge to the set of stationary Nash equilibria in Markov potential games with probability 1. Our results highlight the efficacy of simple learning dynamics in reaching to the set of stationary Nash equilibrium even in environments with minimal information available.

翻译：我们提出了一种多智能体强化学习动力学，并分析了其在无限时域折扣马尔可夫势博弈中的收敛性。聚焦于独立与分散设定，其中智能体不知晓博弈模型且无法协调。在每个阶段，智能体以异步方式基于实现的一阶段奖励更新评估其总或有收益的Q函数估计。随后，智能体根据估计的Q函数融入最优一阶段偏离策略，独立更新其策略。该学习动力学的关键特征在于Q函数估计的更新速度快于策略更新。我们证明，该学习动力学诱导的策略以概率1收敛至马尔可夫势博弈的平稳纳什均衡集。我们的结果凸显了即使在信息极度匮乏的环境中，简单学习动力学在达成平稳纳什均衡集方面的有效性。