Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. They learn stochastic parametric (hyper)policies by either exploring in the space of actions or in the space of parameters. Stochastic controllers, however, are often undesirable from a practical perspective because of their lack of robustness, safety, and traceability. In common practice, stochastic (hyper)policies are learned only to deploy their deterministic version. In this paper, we make a step towards the theoretical understanding of this practice. After introducing a novel framework for modeling this scenario, we study the global convergence to the best deterministic policy, under (weak) gradient domination assumptions. Then, we illustrate how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy. Finally, we quantitatively compare action-based and parameter-based exploration, giving a formal guise to intuitive results.

翻译：策略梯度（PG）方法是处理连续强化学习（RL）问题的有效途径。它们通过在动作空间或参数空间中探索来学习随机参数化（超）策略。然而，从实践角度看，随机控制器往往因其缺乏鲁棒性、安全性和可追溯性而不可取。常见做法是先学习随机（超）策略，随后部署其确定性版本。本文旨在从理论上理解这一实践。在引入一个用于建模该场景的新框架后，我们研究了在（弱）梯度主导假设下全局收敛到最优确定性策略的性质。接着，我们阐述了如何调整用于学习的探索水平，以在样本复杂度与部署的确定性策略性能之间实现最优权衡。最后，我们定量比较了基于动作的探索与基于参数的探索，为直观结果提供了形式化表述。