We propose a multi-agent reinforcement learning dynamics, and analyze its convergence in infinite-horizon discounted Markov potential games. We focus on the independent and decentralized setting, where players do not have knowledge of the game model and cannot coordinate. In each stage, players update their estimate of a perturbed Q-function that evaluates their total contingent payoff based on the realized one-stage reward in an asynchronous manner. Then, players independently update their policies by incorporating a smoothed optimal one-stage deviation strategy based on the estimated Q-function. A key feature of the learning dynamics is that the Q-function estimates are updated at a faster timescale than the policies. We prove that the policies induced by our learning dynamics converge to a stationary Nash equilibrium in Markov potential games with probability 1. Our results highlight the efficacy of simple learning dynamics in reaching a stationary Nash equilibrium even in environments with minimal information available.

翻译：我们提出了一种多智能体强化学习动力学，并分析了其在无限时域折扣马尔可夫势博弈中的收敛性。我们聚焦于独立且去中心化的环境，其中智能体不了解博弈模型且无法进行协调。在每个阶段，智能体以异步方式基于实现的单阶段回报更新其对扰动Q函数的估计，该Q函数评估其总条件收益。随后，智能体通过基于估计的Q函数融入平滑的最优单阶段偏差策略，独立更新其策略。该学习动力学的一个关键特征是Q函数估计的更新速度比策略更新更快。我们证明了由我们的学习动力学诱导的策略以概率1收敛到马尔可夫势博弈中的平稳纳什均衡。我们的结果凸显了即使在信息最匮乏的环境中，简单的学习动力学也能有效收敛至平稳纳什均衡。