Direct policy optimization in reinforcement learning is usually solved with policy-gradient algorithms, which optimize policy parameters via stochastic gradient ascent. This paper provides a new theoretical interpretation and justification of these algorithms. First, we formulate direct policy optimization in the optimization by continuation framework. The latter is a framework for optimizing nonconvex functions where a sequence of surrogate objective functions, called continuations, are locally optimized. Second, we show that optimizing affine Gaussian policies and performing entropy regularization can be interpreted as implicitly optimizing deterministic policies by continuation. Based on these theoretical results, we argue that exploration in policy-gradient algorithms consists in computing a continuation of the return of the policy at hand, and that the variance of policies should be history-dependent functions adapted to avoid local extrema rather than to maximize the return of the policy.

翻译：强化学习中的直接策略优化通常通过策略梯度算法求解，这类算法利用随机梯度上升优化策略参数。本文为这些算法提供了新的理论诠释与验证。首先，我们在优化延拓框架下形式化直接策略优化问题——该框架通过局部优化一系列称为"延拓"的代理目标函数来处理非凸函数优化。其次，我们证明优化仿射高斯策略与执行熵正则化可被解释为通过延拓隐式优化确定性策略。基于这些理论成果，我们指出策略梯度算法中的探索机制本质上是对当前策略回报的延拓计算，且策略方差应被设计为依赖于历史轨迹的函数，其作用在于规避局部极值而非最大化策略回报。