While policy optimization algorithms have played an important role in recent empirical success of Reinforcement Learning (RL), the existing theoretical understanding of policy optimization remains rather limited -- they are either restricted to tabular MDPs or suffer from highly suboptimal sample complexity, especial in online RL where exploration is necessary. This paper proposes a simple efficient policy optimization framework -- Optimistic NPG for online RL. Optimistic NPG can be viewed as a simple combination of the classic natural policy gradient (NPG) algorithm [Kakade, 2001] with optimistic policy evaluation subroutines to encourage exploration. For $d$-dimensional linear MDPs, Optimistic NPG is computationally efficient, and learns an $\varepsilon$-optimal policy within $\tilde{O}(d^2/\varepsilon^3)$ samples, which is the first computationally efficient algorithm whose sample complexity has the optimal dimension dependence $\tilde{\Theta}(d^2)$. It also improves over state-of-the-art results of policy optimization algorithms [Zanette et al., 2021] by a factor of $d$. In the realm of general function approximation, which subsumes linear MDPs, Optimistic NPG, to our best knowledge, stands as the first policy optimization algorithm that achieves polynomial sample complexity for learning near-optimal policies.

翻译：尽管政策优化算法在近期强化学习（RL）的实际成功中发挥了重要作用，但现有对政策优化的理论理解仍相当有限——它们要么局限于表格型马尔可夫决策过程（MDP），要么样本复杂度严重欠优，尤其是在需要探索的在线RL场景中。本文提出了一种简单高效的政策优化框架——用于在线RL的乐观NPG（Optimistic NPG）。乐观NPG可被视为经典自然策略梯度（NPG）算法[Kakade, 2001]与促进探索的乐观策略评估子程序的简单结合。对于$d$维线性MDP，乐观NPG在计算上高效，且能在$\tilde{O}(d^2/\varepsilon^3)$个样本内学习到一个$\varepsilon$-最优策略，这是首个样本复杂度具有最优维度依赖性$\tilde{\Theta}(d^2)$的计算高效算法。此外，它还将政策优化算法的现有最先进结果[Zanette et al., 2021]提升了$d$倍。在涵盖线性MDP的一般函数逼近领域，据我们所知，乐观NPG是首个能在学习近优策略时实现多项式样本复杂度的政策优化算法。