While reinforcement learning has shown experimental success in a number of applications, it is known to be sensitive to noise and perturbations in the parameters of the system, leading to high variance in the total reward amongst different episodes in slightly different environments. To introduce robustness, as well as sample efficiency, risk-sensitive reinforcement learning methods are being thoroughly studied. In this work, we provide a definition of robust reinforcement learning policies and formulate a risk-sensitive reinforcement learning problem to approximate them, by solving an optimization problem with respect to a modified objective based on exponential criteria. In particular, we study a model-free risk-sensitive variation of the widely-used Monte Carlo Policy Gradient algorithm and introduce a novel risk-sensitive online Actor-Critic algorithm based on solving a multiplicative Bellman equation using stochastic approximation updates. Analytical results suggest that the use of exponential criteria generalizes commonly used ad-hoc regularization approaches, improves sample efficiency, and introduces robustness with respect to perturbations in the model parameters and the environment. The implementation, performance, and robustness properties of the proposed methods are evaluated in simulated experiments.

翻译：虽然强化学习在多项应用中的实验已取得成功，但已知其对系统参数中的噪声和扰动敏感，导致在略有差异的环境下不同回合的总奖励差异较大。为提升鲁棒性和样本效率，风险敏感强化学习方法正被深入研究。本文定义了鲁棒强化学习策略，并通过求解基于指数准则的修正目标优化问题，提出了近似这些策略的风险敏感强化学习框架。具体而言，我们研究了广泛使用的蒙特卡洛策略梯度算法的无模型风险敏感变体，并基于随机逼近更新求解乘法贝尔曼方程，提出了一种新型风险敏感在线Actor-Critic算法。理论分析表明，采用指数准则可泛化常见的启发式正则化方法，提升样本效率，并增强对模型参数和环境扰动的鲁棒性。通过仿真实验评估了所提方法的实现、性能及鲁棒性特性。