This study tackles the challenges of adversarial corruption in model-based reinforcement learning (RL), where the transition dynamics can be corrupted by an adversary. Existing studies on corruption-robust RL mostly focus on the setting of model-free RL, where robust least-square regression is often employed for value function estimation. However, these techniques cannot be directly applied to model-based RL. In this paper, we focus on model-based RL and take the maximum likelihood estimation (MLE) approach to learn transition model. Our work encompasses both online and offline settings. In the online setting, we introduce an algorithm called corruption-robust optimistic MLE (CR-OMLE), which leverages total-variation (TV)-based information ratios as uncertainty weights for MLE. We prove that CR-OMLE achieves a regret of $\tilde{\mathcal{O}}(\sqrt{T} + C)$, where $C$ denotes the cumulative corruption level after $T$ episodes. We also prove a lower bound to show that the additive dependence on $C$ is optimal. We extend our weighting technique to the offline setting, and propose an algorithm named corruption-robust pessimistic MLE (CR-PMLE). Under a uniform coverage condition, CR-PMLE exhibits suboptimality worsened by $\mathcal{O}(C/n)$, nearly matching the lower bound. To the best of our knowledge, this is the first work on corruption-robust model-based RL algorithms with provable guarantees.

翻译：本研究旨在解决模型强化学习中面临的对抗性扰动挑战，其中转移动力学可能遭受来自对抗方的破坏。现有关于抗扰动鲁棒RL的研究主要聚焦于无模型RL设定，常采用稳健最小二乘回归进行值函数估计。然而，这些技术无法直接应用于基于模型的RL。本文聚焦于模型RL，采用最大似然估计方法学习转移模型，同时涵盖在线与离线两种设定。在线设定下，我们提出名为抗扰动乐观MLE的算法，该算法利用基于全变差的信息比作为MLE的不确定性权重。我们证明CR-OMLE可达到$\tilde{\mathcal{O}}(\sqrt{T} + C)$的遗憾界，其中$C$表示$T$轮次后的累积扰动水平。进一步通过下界证明表明对$C$的加性依赖是最优的。我们将加权技术扩展至离线设定，提出抗扰动悲观MLE算法。在均匀覆盖条件下，CR-PMLE的次优性被$\mathcal{O}(C/n)$恶化，该结果近乎匹配下界。据我们所知，这是首个具有可证明保证的抗扰动鲁棒模型RL算法研究。