This paper investigates posterior sampling algorithms for competitive reinforcement learning (RL) in the context of general function approximations. Focusing on zero-sum Markov games (MGs) under two critical settings, namely self-play and adversarial learning, we first propose the self-play and adversarial generalized eluder coefficient (GEC) as complexity measures for function approximation, capturing the exploration-exploitation trade-off in MGs. Based on self-play GEC, we propose a model-based self-play posterior sampling method to control both players to learn Nash equilibrium, which can successfully handle the partial observability of states. Furthermore, we identify a set of partially observable MG models fitting MG learning with the adversarial policies of the opponent. Incorporating the adversarial GEC, we propose a model-based posterior sampling method for learning adversarial MG with potential partial observability. We further provide low regret bounds for proposed algorithms that can scale sublinearly with the proposed GEC and the number of episodes $T$. To the best of our knowledge, we for the first time develop generic model-based posterior sampling algorithms for competitive RL that can be applied to a majority of tractable zero-sum MG classes in both fully observable and partially observable MGs with self-play and adversarial learning.

翻译：本文研究了竞争性强化学习在通用函数逼近背景下的后验采样算法。聚焦于零和马尔可夫博弈在两种关键设置（即自博弈与对抗学习）下的情形，我们首先提出自博弈与对抗广义埃路德系数作为函数逼近的复杂度指标，以刻画马尔可夫博弈中的探索-利用权衡。基于自博弈广义埃路德系数，我们提出一种基于模型的自博弈后验采样方法，用于控制双方玩家学习纳什均衡，该方法可有效处理状态的部分可观测性。进一步地，我们识别出一类与对手对抗策略相符的部分可观测马尔可夫博弈模型。结合对抗广义埃路德系数，我们提出一种基于模型的后验采样方法，用于学习具有潜在部分可观测性的对抗马尔可夫博弈。我们还为所提出的算法提供了低遗憾界，该界限可随所提出的广义埃路德系数及回合数$T$呈次线性增长。据我们所知，本文首次为竞争性强化学习开发出通用的基于模型的后验采样算法，可应用于完全可观测与部分可观测马尔可夫博弈中自博弈与对抗学习的大多数可解零和马尔可夫博弈类别。