Many Deep Reinforcement Learning (D-RL) algorithms rely on simple forms of exploration such as the additive action noise often used in continuous control domains. Typically, the scaling factor of this action noise is chosen as a hyper-parameter and is kept constant during training. In this paper, we focus on action noise in off-policy deep reinforcement learning for continuous control. We analyze how the learned policy is impacted by the noise type, noise scale, and impact scaling factor reduction schedule. We consider the two most prominent types of action noise, Gaussian and Ornstein-Uhlenbeck noise, and perform a vast experimental campaign by systematically varying the noise type and scale parameter, and by measuring variables of interest like the expected return of the policy and the state-space coverage during exploration. For the latter, we propose a novel state-space coverage measure $\operatorname{X}_{\mathcal{U}\text{rel}}$ that is more robust to estimation artifacts caused by points close to the state-space boundary than previously-proposed measures. Larger noise scales generally increase state-space coverage. However, we found that increasing the space coverage using a larger noise scale is often not beneficial. On the contrary, reducing the noise scale over the training process reduces the variance and generally improves the learning performance. We conclude that the best noise type and scale are environment dependent, and based on our observations derive heuristic rules for guiding the choice of the action noise as a starting point for further optimization.

翻译：许多深度强化学习算法依赖于简单的探索方式，例如连续控制领域中常用的加性动作噪声。通常，此类动作噪声的缩放因子作为超参数在训练过程中保持恒定。本文聚焦离线深度强化学习中连续控制任务的动作噪声问题，分析噪声类型、噪声尺度及缩放因子衰减策略对学习策略的影响。我们考虑两类主流动作噪声——高斯噪声和奥恩斯坦-乌伦贝克噪声，通过系统化改变噪声类型与尺度参数，并测量策略期望回报与探索过程中的状态空间覆盖率等关键变量，开展大规模实验研究。针对状态空间覆盖率，我们提出一种新型度量指标$\operatorname{X}_{\mathcal{U}\text{rel}}$，相较于现有方法，该指标对接近状态空间边界点引起的估计伪影具有更强的鲁棒性。实验表明，增大噪声尺度通常能提升状态空间覆盖率，但通过加大噪声尺度来提高空间覆盖率往往无益于性能提升。相反，在训练过程中逐步降低噪声尺度可减少方差，并普遍改善学习性能。我们得出结论：最优噪声类型与尺度取决于具体环境，并基于观测结果推导出启发式规则，以指导动作噪声的初始选择，为后续优化提供起点。