Robust Markov Decision Processes (RMDPs) are a widely used framework for sequential decision-making under parameter uncertainty. RMDPs have been extensively studied when the objective is to maximize the discounted return, but little is known for average optimality (optimizing the long-run average of the rewards obtained over time) and Blackwell optimality (remaining discount optimal for all discount factors sufficiently close to 1). In this paper, we prove several foundational results for RMDPs beyond the discounted return. We show that average optimal policies can be chosen stationary and deterministic for sa-rectangular RMDPs but, perhaps surprisingly, that history-dependent (Markovian) policies strictly outperform stationary policies for average optimality in s-rectangular RMDPs. We also study Blackwell optimality for sa-rectangular RMDPs, where we show that {\em approximate} Blackwell optimal policies always exist, although Blackwell optimal policies may not exist. We also provide a sufficient condition for their existence, which encompasses virtually any examples from the literature. We then discuss the connection between average and Blackwell optimality, and we describe several algorithms to compute the optimal average return. Interestingly, our approach leverages the connections between RMDPs and stochastic games.

翻译：鲁棒马尔可夫决策过程（RMDPs）是在参数不确定性下进行序贯决策的广泛使用的框架。当目标为最大化折扣回报时，RMDPs已得到广泛研究，但对于平均最优性（优化长期平均累积奖励）和Blackwell最优性（在所有足够接近1的折扣因子下保持折扣最优）的了解却很少。本文证明了超越折扣回报的RMDPs的若干基础性结果。我们证明：对于sa-矩形RMDPs，平均最优策略可选取为平稳确定性策略；但令人惊讶的是，对于s-矩形RMDPs，平均最优性中历史依赖（马尔可夫）策略严格优于平稳策略。我们还研究了sa-矩形RMDPs的Blackwell最优性，表明尽管Blackwell最优策略可能不存在，但近似Blackwell最优策略始终存在。我们给出了其存在的一个充分条件，该条件涵盖了文献中的几乎所有实例。随后讨论了平均最优性与Blackwell最优性之间的关联，并描述了多种计算最优平均回报的算法。有趣的是，我们的方法利用了RMDPs与随机博弈之间的联系。