Policy optimization methods are popular reinforcement learning algorithms in practice. Recent works have built theoretical foundation for them by proving $\sqrt{T}$ regret bounds even when the losses are adversarial. Such bounds are tight in the worst case but often overly pessimistic. In this work, we show that in tabular Markov decision processes (MDPs), by properly designing the regularizer, the exploration bonus and the learning rates, one can achieve a more favorable polylog$(T)$ regret when the losses are stochastic, without sacrificing the worst-case guarantee in the adversarial regime. To our knowledge, this is also the first time a gap-dependent polylog$(T)$ regret bound is shown for policy optimization. Specifically, we achieve this by leveraging a Tsallis entropy or a Shannon entropy regularizer in the policy update. Then we show that under known transitions, we can further obtain a first-order regret bound in the adversarial regime by leveraging the log-barrier regularizer.

翻译：策略优化方法是实践中流行的强化学习算法。近期研究通过证明即使在损失具有对抗性的情况下也能获得$\sqrt{T}$的遗憾界，为这些方法奠定了理论基础。这类界在最坏情况下是紧致的，但往往过于悲观。本文表明，在表格型马尔可夫决策过程中，通过适当设计正则化器、探索奖励和学习率，可以在损失具有随机性时实现更优的polylog$(T)$遗憾界，同时不牺牲对抗性场景下的最坏情况保证。据我们所知，这也是首次为策略优化证明间隙相关的polylog$(T)$遗憾界。具体而言，我们通过在策略更新中利用Tsallis熵或Shannon熵正则化器实现这一目标。进一步地，我们证明在已知状态转移矩阵的情况下，借助对数障碍正则化器可在对抗性场景中获得一阶遗憾界。