Robust Markov decision processes (RMDPs) provide a promising framework for computing reliable policies in the face of model errors. Many successful reinforcement learning algorithms build on variations of policy-gradient methods, but adapting these methods to RMDPs has been challenging. As a result, the applicability of RMDPs to large, practical domains remains limited. This paper proposes a new Double-Loop Robust Policy Gradient (DRPG), the first generic policy gradient method for RMDPs. In contrast with prior robust policy gradient algorithms, DRPG monotonically reduces approximation errors to guarantee convergence to a globally optimal policy in tabular RMDPs. We introduce a novel parametric transition kernel and solve the inner loop robust policy via a gradient-based method. Finally, our numerical results demonstrate the utility of our new algorithm and confirm its global convergence properties.

翻译：鲁棒马尔可夫决策过程（RMDPs）为应对模型误差下计算可靠策略提供了有前景的框架。许多成功的强化学习算法基于策略梯度方法的变体，但将这些方法适配到RMDPs颇具挑战性。因此，RMDPs在大型实际应用场景中的适用性仍受到限制。本文提出了一种新的双循环鲁棒策略梯度（DRPG）算法——首个专为RMDPs设计的通用策略梯度方法。与先前的鲁棒策略梯度算法不同，DRPG通过单调减小近似误差，确保在表格型RMDPs中收敛到全局最优策略。我们引入了一种新颖的参数化转移核，并采用基于梯度的方法求解内部循环的鲁棒策略。最后，数值实验结果验证了新算法的实用性，并证实了其全局收敛特性。