We investigate robustness to strong data corruption in offline sparse reinforcement learning (RL). In our setting, an adversary may arbitrarily perturb a fraction of the collected trajectories from a high-dimensional but sparse Markov decision process, and our goal is to estimate a near optimal policy. The main challenge is that, in the high-dimensional regime where the number of samples $N$ is smaller than the feature dimension $d$, exploiting sparsity is essential for obtaining non-vacuous guarantees but has not been systematically studied in offline RL. We analyse the problem under uniform coverage and sparse single-concentrability assumptions. While Least Square Value Iteration (LSVI), a standard approach for robust offline RL, performs well under uniform coverage, we show that integrating sparsity into LSVI is unnatural, and its analysis may break down due to overly pessimistic bonuses. To overcome this, we propose actor-critic methods with sparse robust estimator oracles, which avoid the use of pointwise pessimistic bonuses and provide the first non-vacuous guarantees for sparse offline RL under single-policy concentrability coverage. Moreover, we extend our results to the contaminated setting and show that our algorithm remains robust under strong contamination. Our results provide the first non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage and corruption, showing that learning a near-optimal policy remains possible in regimes where traditional robust offline RL techniques may fail.

翻译：本文研究了稀疏离线强化学习中对强数据污染的鲁棒性问题。在我们的设定中，对手可以任意扰动从高维稀疏马尔可夫决策过程中收集的部分轨迹，我们的目标是估计一个接近最优的策略。主要挑战在于，在样本数量$N$小于特征维度$d$的高维场景下，利用稀疏性对于获得非平凡保证至关重要，但这一特性尚未在离线强化学习中得到系统研究。我们在均匀覆盖和稀疏单集中性假设下分析了该问题。虽然最小二乘值迭代作为鲁棒离线强化学习的标准方法在均匀覆盖下表现良好，但我们发现将稀疏性整合到LSVI中并不自然，且其分析可能因过于悲观的奖励修正项而失效。为解决这一问题，我们提出了基于稀疏鲁棒估计器预言机的演员-批评家方法，该方法避免了逐点悲观奖励修正项的使用，并为单策略集中性覆盖下的稀疏离线强化学习提供了首个非平凡保证。此外，我们将结果扩展到污染场景，证明我们的算法在强污染条件下仍保持鲁棒性。本研究首次为具有单策略集中性覆盖和污染的高维稀疏MDP提供了非平凡保证，表明在传统鲁棒离线强化学习技术可能失效的场景中，学习接近最优策略仍然是可行的。