Policy Mirror Descent (PMD) is a general family of algorithms that covers a wide range of novel and fundamental methods in reinforcement learning. Motivated by the instability of policy iteration (PI) with inexact policy evaluation, PMD algorithmically regularises the policy improvement step of PI. With exact policy evaluation, PI is known to converge linearly with a rate given by the discount factor $\gamma$ of a Markov Decision Process. In this work, we bridge the gap between PI and PMD with exact policy evaluation and show that the dimension-free $\gamma$-rate of PI can be achieved by the general family of unregularised PMD algorithms under an adaptive step-size. We show that both the rate and step-size are unimprovable for PMD: we provide matching lower bounds that demonstrate that the $\gamma$-rate is optimal for PMD methods as well as PI, and that the adaptive step-size is necessary for PMD to achieve it. Our work is the first to relate PMD to rate-optimality and step-size necessity. Our study of the convergence of PMD avoids the use of the performance difference lemma, which leads to a direct analysis of independent interest. We also extend the analysis to the inexact setting and establish the first dimension-optimal sample complexity for unregularised PMD under a generative model, improving upon the best-known result.

翻译：策略镜像下降（PMD）是一类通用算法家族，涵盖了强化学习中众多新颖且基础的方法。受非精确策略评估下策略迭代（PI）不稳定性问题的启发，PMD通过算法方式对PI的策略改进步骤进行正则化处理。在精确策略评估条件下，已知PI以折扣因子γ确定的线性速率收敛于马尔可夫决策过程。本研究弥合了PI与精确策略评估下PMD之间的鸿沟，证明在自适应步长条件下，无正则化的通用PMD算法家族能够实现PI的无维度依赖γ速率收敛。我们证明该速率与步长对PMD而言均不可改进：通过给出匹配下界，证实γ速率对PMD方法与PI均为最优，且自适应步长是PMD实现该速率的必要条件。本研究首次将PMD与速率最优性及步长必要性建立关联。我们在PMD收敛性分析中规避了性能差异引理的使用，从而形成了具有独立价值的直接分析方法。此外，研究还将分析扩展至非精确设定，在生成模型下建立了首个无正则化PMD的维度最优样本复杂度，改进了现有最优结果。