We study policy optimization algorithms for computing correlated equilibria in multi-player general-sum Markov Games. Previous results achieve $O(T^{-1/2})$ convergence rate to a correlated equilibrium and an accelerated $O(T^{-3/4})$ convergence rate to the weaker notion of coarse correlated equilibrium. In this paper, we improve both results significantly by providing an uncoupled policy optimization algorithm that attains a near-optimal $\tilde{O}(T^{-1})$ convergence rate for computing a correlated equilibrium. Our algorithm is constructed by combining two main elements (i) smooth value updates and (ii) the optimistic-follow-the-regularized-leader algorithm with the log barrier regularizer.

翻译：我们研究了用于多玩家一般和马尔可夫博弈中计算相关均衡的策略优化算法。先前的研究结果实现了以$O(T^{-1/2})$的收敛速率达到相关均衡，并以加速后的$O(T^{-3/4})$的收敛速率达到较弱的粗相关均衡。本文通过提出一种非耦合策略优化算法，在计算相关均衡时实现了近优的$\tilde{O}(T^{-1})$收敛速率，显著改进了上述两项结果。我们的算法由两个主要元素组合构建而成：(i) 平滑值更新；(ii) 采用对数障碍正则化项的乐观跟随正则化领导者算法。