We study Markov potential games under the infinite horizon average reward criterion. Most previous studies have been for discounted rewards. We prove that both algorithms based on independent policy gradient and independent natural policy gradient converge globally to a Nash equilibrium for the average reward criterion. To set the stage for gradient-based methods, we first establish that the average reward is a smooth function of policies and provide sensitivity bounds for the differential value functions, under certain conditions on ergodicity and the second largest eigenvalue of the underlying Markov decision process (MDP). We prove that three algorithms, policy gradient, proximal-Q, and natural policy gradient (NPG), converge to an $\epsilon$-Nash equilibrium with time complexity $O(\frac{1}{\epsilon^2})$, given a gradient/differential Q function oracle. When policy gradients have to be estimated, we propose an algorithm with $\tilde{O}(\frac{1}{\min_{s,a}\pi(a|s)\delta})$ sample complexity to achieve $\delta$ approximation error w.r.t~the $\ell_2$ norm. Equipped with the estimator, we derive the first sample complexity analysis for a policy gradient ascent algorithm, featuring a sample complexity of $\tilde{O}(1/\epsilon^5)$. Simulation studies are presented.

翻译：我们研究了无限时域平均奖励准则下的马尔可夫势博弈。以往研究多聚焦于折扣奖励情形。我们证明，基于独立策略梯度和独立自然策略梯度的两种算法在平均奖励准则下均能全局收敛至纳什均衡。为奠定梯度类方法的基础，我们首先在遍历性及底层马尔可夫决策过程（MDP）第二最大特征值的特定条件下，证明了平均奖励是策略的平滑函数，并给出了微分值函数的敏感性界。我们证明，策略梯度、近端Q学习和自然策略梯度（NPG）这三种算法在给定梯度/微分Q函数预言机的情况下，以时间复杂度$O(\frac{1}{\epsilon^2})$收敛至$\epsilon$-纳什均衡。当策略梯度需通过估计获得时，我们提出一种具有$\tilde{O}(\frac{1}{\min_{s,a}\pi(a|s)\delta})$样本复杂度的算法，可实现关于$\ell_2$范数的$\delta$近似误差。借助该估计器，我们首次推导了策略梯度上升算法的样本复杂度分析，其样本复杂度为$\tilde{O}(1/\epsilon^5)$。最后给出了仿真研究。