Robust Markov decision processes (MDPs) aim to handle changing or partially known system dynamics. To solve them, one typically resorts to robust optimization methods. However, this significantly increases computational complexity and limits scalability in both learning and planning. On the other hand, regularized MDPs show more stability in policy learning without impairing time complexity. Yet, they generally do not encompass uncertainty in the model dynamics. In this work, we aim to learn robust MDPs using regularization. We first show that regularized MDPs are a particular instance of robust MDPs with uncertain reward. We thus establish that policy iteration on reward-robust MDPs can have the same time complexity as on regularized MDPs. We further extend this relationship to MDPs with uncertain transitions: this leads to a regularization term with an additional dependence on the value function. We then generalize regularized MDPs to twice regularized MDPs ($\text{R}^2$ MDPs), i.e., MDPs with $\textit{both}$ value and policy regularization. The corresponding Bellman operators enable us to derive planning and learning schemes with convergence and generalization guarantees, thus reducing robustness to regularization. We numerically show this two-fold advantage on tabular and physical domains, highlighting the fact that $\text{R}^2$ preserves its efficacy in continuous environments.

翻译：鲁棒马尔可夫决策过程（MDPs）旨在处理动态系统变化或部分已知的情况。为了求解此类问题，通常需要采用鲁棒优化方法。然而，这会显著增加计算复杂度，并限制其在学习和规划中的可扩展性。另一方面，正则化MDPs在不影响时间复杂度的前提下，在策略学习方面表现出更强的稳定性。然而，这类方法通常无法涵盖模型动态中的不确定性。在本工作中，我们旨在利用正则化学习鲁棒MDPs。首先，我们证明正则化MDPs是具有不确定奖励的鲁棒MDPs的一个特例，进而确立奖励鲁棒MDPs上的策略迭代可具有与正则化MDPs相同的时间复杂度。我们进一步将此关系扩展到具有不确定转移的MDPs：这会导致正则化项额外依赖于值函数。随后，我们将正则化MDPs推广至双重正则化马尔可夫决策过程（$\text{R}^2$ MDPs），即同时包含值函数和策略正则化的MDPs。相应的贝尔曼算子使我们能够推导出具有收敛性和泛化保证的规划与学习方案，从而将鲁棒性简化为正则化。我们在表格型和物理领域通过数值实验展示了这种双重优势，并强调$\text{R}^2$在连续环境中仍能保持其有效性。