We study the sample complexity of obtaining an $\epsilon$-optimal policy in \emph{Robust} discounted Markov Decision Processes (RMDPs), given only access to a generative model of the nominal kernel. This problem is widely studied in the non-robust case, and it is known that any planning approach applied to an empirical MDP estimated with $\tilde{\mathcal{O}}(\frac{H^3 \mid S \mid\mid A \mid}{\epsilon^2})$ samples provides an $\epsilon$-optimal policy, which is minimax optimal. Results in the robust case are much more scarce. For $sa$- (resp $s$-)rectangular uncertainty sets, the best known sample complexity is $\tilde{\mathcal{O}}(\frac{H^4 \mid S \mid^2\mid A \mid}{\epsilon^2})$ (resp. $\tilde{\mathcal{O}}(\frac{H^4 \mid S \mid^2\mid A \mid^2}{\epsilon^2})$), for specific algorithms and when the uncertainty set is based on the total variation (TV), the KL or the Chi-square divergences. In this paper, we consider uncertainty sets defined with an $L_p$-ball (recovering the TV case), and study the sample complexity of \emph{any} planning algorithm (with high accuracy guarantee on the solution) applied to an empirical RMDP estimated using the generative model. In the general case, we prove a sample complexity of $\tilde{\mathcal{O}}(\frac{H^4 \mid S \mid\mid A \mid}{\epsilon^2})$ for both the $sa$- and $s$-rectangular cases (improvements of $\mid S \mid$ and $\mid S \mid\mid A \mid$ respectively). When the size of the uncertainty is small enough, we improve the sample complexity to $\tilde{\mathcal{O}}(\frac{H^3 \mid S \mid\mid A \mid }{\epsilon^2})$, recovering the lower-bound for the non-robust case for the first time and a robust lower-bound when the size of the uncertainty is small enough.

翻译：我们研究了在仅能访问名义核生成模型的情况下，获得稳健折扣马尔可夫决策过程（RMDP）中$\epsilon$-最优策略的样本复杂度。该问题在非稳健情形下已被广泛研究，已知任何基于经验MDP（通过$\tilde{\mathcal{O}}(\frac{H^3 |S||A|}{\epsilon^2})$样本估计）的规划方法均可提供$\epsilon$-最优策略，且该结果已达极小极大最优。但稳健情形下的研究结果则稀缺得多。对于$sa$-矩形（或$s$-矩形）不确定性集，当不确定性集基于全变差（TV）、KL散度或卡方散度时，已知的最佳样本复杂度分别为$\tilde{\mathcal{O}}(\frac{H^4 |S|^2|A|}{\epsilon^2})$（或$\tilde{\mathcal{O}}(\frac{H^4 |S|^2|A|^2}{\epsilon^2})$），且仅适用于特定算法。本文考虑基于$L_p$球（可还原TV情形）定义的不确定性集，并研究任何规划算法（要求解具有高精度保证）应用于基于生成模型估计的经验RMDP时的样本复杂度。在一般情况下，我们证明$sa$-矩形和$s$-矩形情形下的样本复杂度均为$\tilde{\mathcal{O}}(\frac{H^4 |S||A|}{\epsilon^2})$（分别较先前结果改进$|S|$和$|S||A|$因子）。当不确定性规模足够小时，我们将样本复杂度提升至$\tilde{\mathcal{O}}(\frac{H^3 |S||A|}{\epsilon^2})$，首次恢复了非稳健情形下的下界，并在不确定性规模足够小时实现了稳健情形下的下界。