This paper is concerned with offline reinforcement learning (RL), which learns using pre-collected data without further exploration. Effective offline RL would be able to accommodate distribution shift and limited data coverage. However, prior algorithms or analyses either suffer from suboptimal sample complexities or incur high burn-in cost to reach sample optimality, thus posing an impediment to efficient offline RL in sample-starved applications. We demonstrate that the model-based (or "plug-in") approach achieves minimax-optimal sample complexity without burn-in cost for tabular Markov decision processes (MDPs). Concretely, consider a finite-horizon (resp. $\gamma$-discounted infinite-horizon) MDP with $S$ states and horizon $H$ (resp. effective horizon $\frac{1}{1-\gamma}$), and suppose the distribution shift of data is reflected by some single-policy clipped concentrability coefficient $C^{\star}_{\text{clipped}}$. We prove that model-based offline RL yields $\varepsilon$-accuracy with a sample complexity of \[ \begin{cases} \frac{H^{4}SC_{\text{clipped}}^{\star}}{\varepsilon^{2}} & (\text{finite-horizon MDPs}) \frac{SC_{\text{clipped}}^{\star}}{(1-\gamma)^{3}\varepsilon^{2}} & (\text{infinite-horizon MDPs}) \end{cases} \] up to log factor, which is minimax optimal for the entire $\varepsilon$-range. The proposed algorithms are "pessimistic" variants of value iteration with Bernstein-style penalties, and do not require sophisticated variance reduction. Our analysis framework is established upon delicate leave-one-out decoupling arguments in conjunction with careful self-bounding techniques tailored to MDPs.

翻译：本文研究离线强化学习（Offline Reinforcement Learning, RL），即利用预先收集的数据进行学习而无需进一步探索。有效的离线强化学习应能适应分布偏移和有限的数据覆盖。然而，现有算法或分析要么样本复杂度欠优，要么为达到样本最优性而付出高昂的启动代价，从而阻碍了样本匮乏应用中高效离线强化学习的实现。我们证明，对于表格型马尔可夫决策过程（Markov Decision Processes, MDPs），基于模型（或称"插件式"）方法能在无启动代价的情况下达到极小化最优样本复杂度。具体而言，考虑一个有限时域（或$\gamma$折扣无限时域）MDP，其状态数为$S$，时域长度为$H$（或有效时域$\frac{1}{1-\gamma}$），并假设数据的分布偏移由某个单策略截断集中系数$C^{\star}_{\text{clipped}}$刻画。我们证明，基于模型的离线强化学习在样本复杂度上能达到$\varepsilon$精度：\[ \begin{cases} \frac{H^{4}SC_{\text{clipped}}^{\star}}{\varepsilon^{2}} & (\text{有限时域MDPs}) \\ \frac{SC_{\text{clipped}}^{\star}}{(1-\gamma)^{3}\varepsilon^{2}} & (\text{无限时域MDPs}) \end{cases} \]（忽略对数因子），这在整个$\varepsilon$范围内是极小化最优的。所提出的算法是带有伯恩斯坦惩罚项的值迭代的"悲观"变体，无需复杂的方差缩减技术。我们的分析框架建立在精细的留一法解耦论证基础上，并结合专为MDP设计的自约束技术。