This study tackles the challenges of adversarial corruption in model-based reinforcement learning (RL), where the transition dynamics can be corrupted by an adversary. Existing studies on corruption-robust RL mostly focus on the setting of model-free RL, where robust least-square regression is often employed for value function estimation. However, these techniques cannot be directly applied to model-based RL. In this paper, we focus on model-based RL and take the maximum likelihood estimation (MLE) approach to learn transition model. Our work encompasses both online and offline settings. In the online setting, we introduce an algorithm called corruption-robust optimistic MLE (CR-OMLE), which leverages total-variation (TV)-based information ratios as uncertainty weights for MLE. We prove that CR-OMLE achieves a regret of $\tilde{\mathcal{O}}(\sqrt{T} + C)$, where $C$ denotes the cumulative corruption level after $T$ episodes. We also prove a lower bound to show that the additive dependence on $C$ is optimal. We extend our weighting technique to the offline setting, and propose an algorithm named corruption-robust pessimistic MLE (CR-PMLE). Under a uniform coverage condition, CR-PMLE exhibits suboptimality worsened by $\mathcal{O}(C/n)$, nearly matching the lower bound. To the best of our knowledge, this is the first work on corruption-robust model-based RL algorithms with provable guarantees.

翻译：本研究探讨了模型强化学习（RL）中对抗性扰动带来的挑战，其中状态转移动态可能受到对手的破坏。现有关于抗扰动鲁棒RL的研究主要集中于无模型RL场景，通常采用鲁棒最小二乘回归进行价值函数估计。然而，这些技术无法直接应用于模型强化学习。本文聚焦于模型强化学习，采用最大似然估计（MLE）方法学习转移模型。我们的工作涵盖在线与离线两种设置。在线设置中，我们提出了一种称为抗扰动乐观MLE（CR-OMLE）的算法，该算法利用基于总变差（TV）的信息比率作为MLE的不确定性权重。我们证明CR-OMLE能够实现$\tilde{\mathcal{O}}(\sqrt{T} + C)$的遗憾值，其中$C$表示$T$个回合后的累积扰动水平。同时我们通过下界证明表明对$C$的加法依赖是最优的。我们将权重技术扩展至离线设置，提出了名为抗扰动悲观MLE（CR-PMLE）的算法。在均匀覆盖条件下，CR-PMLE表现出$\mathcal{O}(C/n)$的次优性恶化，几乎匹配理论下界。据我们所知，这是首个具有可证明保证的抗扰动模型强化学习算法研究。