We propose a multi-agent reinforcement learning dynamics, and analyze its convergence in infinite-horizon discounted Markov potential games. We focus on the independent and decentralized setting, where players do not have knowledge of the game model and cannot coordinate. In each stage, players update their estimate of Q-function that evaluates their total contingent payoff based on the realized one-stage reward in an asynchronous manner. Then, players independently update their policies by incorporating an optimal one-stage deviation strategy based on the estimated Q-function. A key feature of the learning dynamics is that the Q-function estimates are updated at a faster timescale than the policies. We prove that the policies induced by our learning dynamics converge to the set of stationary Nash equilibria in Markov potential games with probability 1. Our results highlight the efficacy of simple learning dynamics in reaching to the set of stationary Nash equilibrium even in environments with minimal information available.

翻译：我们提出一种多智能体强化学习动力学方法，并分析其在无限时域折扣马尔可夫势博弈中的收敛性。研究聚焦于独立与分散式场景，其中智能体既无博弈模型知识也无法进行协调。在每个阶段，智能体以异步方式基于实际实现的单阶段奖励更新评估其总或有收益的Q函数估计值，随后通过结合基于估计Q函数的最优单阶段偏离策略独立更新其策略。该学习动力学的核心特征在于Q函数估计值以快于策略更新的时间尺度进行迭代。我们证明，该学习动力学诱导的策略以概率1收敛至马尔可夫势博弈的平稳纳什均衡集。研究结果表明，即使在信息极其有限的环境下，简单的学习动力学仍能有效收敛至平稳纳什均衡集。