We propose a new model, independent linear Markov game, for multi-agent reinforcement learning with a large state space and a large number of agents. This is a class of Markov games with independent linear function approximation, where each agent has its own function approximation for the state-action value functions that are marginalized by other players' policies. We design new algorithms for learning the Markov coarse correlated equilibria (CCE) and Markov correlated equilibria (CE) with sample complexity bounds that only scale polynomially with each agent's own function class complexity, thus breaking the curse of multiagents. In contrast, existing works for Markov games with function approximation have sample complexity bounds scale with the size of the \emph{joint action space} when specialized to the canonical tabular Markov game setting, which is exponentially large in the number of agents. Our algorithms rely on two key technical innovations: (1) utilizing policy replay to tackle non-stationarity incurred by multiple agents and the use of function approximation; (2) separating learning Markov equilibria and exploration in the Markov games, which allows us to use the full-information no-regret learning oracle instead of the stronger bandit-feedback no-regret learning oracle used in the tabular setting. Furthermore, we propose an iterative-best-response type algorithm that can learn pure Markov Nash equilibria in independent linear Markov potential games. In the tabular case, by adapting the policy replay mechanism for independent linear Markov games, we propose an algorithm with $\widetilde{O}(\epsilon^{-2})$ sample complexity to learn Markov CCE, which improves the state-of-the-art result $\widetilde{O}(\epsilon^{-3})$ in Daskalakis et al. 2022, where $\epsilon$ is the desired accuracy, and also significantly improves other problem parameters.

翻译：我们提出了一个针对大规模状态空间和大量智能体的多智能体强化学习新模型——独立线性马尔可夫博弈。这是一类具有独立线性函数逼近的马尔可夫博弈，其中每个智能体拥有自己的状态-动作价值函数逼近器，该函数被其他智能体的策略边缘化。我们设计了新的算法来学习马尔可夫粗相关均衡和马尔可夫相关均衡，其样本复杂度仅随每个智能体自身函数类复杂度多项式增长，从而打破了多智能体魔咒。相比之下，现有关于函数逼近马尔可夫博弈的工作在标准表格型马尔可夫博弈设定下，样本复杂度与联合动作空间大小呈正比，而该空间随智能体数量呈指数级增长。我们的算法依赖于两项关键技术革新：（1）利用策略重放应对多智能体带来的非平稳性及函数逼近的使用；（2）将马尔可夫均衡学习与马尔可夫博弈中的探索分离，这使得我们能够使用全信息无遗憾学习预言机，而非表格型设定中使用的更严格的带反馈的无遗憾学习预言机。此外，我们提出了一种迭代最优响应型算法，可在独立线性马尔可夫势博弈中学习纯策略马尔可夫纳什均衡。在表格型情况下，通过为独立线性马尔可夫博弈调整策略重放机制，我们提出了一种样本复杂度为$\widetilde{O}(\epsilon^{-2})$的算法来学习马尔可夫粗相关均衡，这改进了Daskalakis等人2022年工作中的最新结果$\widetilde{O}(\epsilon^{-3})$（其中$\epsilon$为期望精度），并显著改善了其他问题参数。