This paper is concerned with offline reinforcement learning (RL), which learns using pre-collected data without further exploration. Effective offline RL would be able to accommodate distribution shift and limited data coverage. However, prior algorithms or analyses either suffer from suboptimal sample complexities or incur high burn-in cost to reach sample optimality, thus posing an impediment to efficient offline RL in sample-starved applications. We demonstrate that the model-based (or "plug-in") approach achieves minimax-optimal sample complexity without burn-in cost for tabular Markov decision processes (MDPs). Concretely, consider a finite-horizon (resp. $\gamma$-discounted infinite-horizon) MDP with $S$ states and horizon $H$ (resp. effective horizon $\frac{1}{1-\gamma}$), and suppose the distribution shift of data is reflected by some single-policy clipped concentrability coefficient $C^{\star}_{\text{clipped}}$. We prove that model-based offline RL yields $\varepsilon$-accuracy with a sample complexity of \[ \begin{cases} \frac{H^{4}SC_{\text{clipped}}^{\star}}{\varepsilon^{2}} & (\text{finite-horizon MDPs}) \frac{SC_{\text{clipped}}^{\star}}{(1-\gamma)^{3}\varepsilon^{2}} & (\text{infinite-horizon MDPs}) \end{cases} \] up to log factor, which is minimax optimal for the entire $\varepsilon$-range. The proposed algorithms are ``pessimistic'' variants of value iteration with Bernstein-style penalties, and do not require sophisticated variance reduction. Our analysis framework is established upon delicate leave-one-out decoupling arguments in conjunction with careful self-bounding techniques tailored to MDPs.

翻译：本文关注离线强化学习（Offline RL），即利用预先收集的数据进行学习而无需进一步探索。有效的离线学习需能应对分布偏移和有限的数据覆盖。然而，现有算法或分析要么样本复杂度次优，要么需通过高昂的“预热”成本才能达到样本最优性，这制约了样本匮乏场景中离线强化学习的效率。我们证明：对于表格型马尔可夫决策过程（MDP），基于模型（或称“插件式”）方法可在无需预热成本的前提下实现极小极大最优样本复杂度。具体而言，考虑一个有限时域（或γ折扣无限时域）MDP，其状态数为S，时域为H（或有效时域为1/(1-γ)），并假设数据分布偏移由某个单策略截断集中系数C*_{clipped}刻画。我们证明，基于模型的离线强化学习可达到ε精度，其样本复杂度为（公式略），在对数因子范围内达到整个ε范围内的极小极大最优。所提出的算法是带有Bernstein型惩罚项的“悲观”值迭代变体，且无需复杂方差缩减技术。我们的分析框架基于精巧的逐次去耦论证及针对MDP设计的自约束技术。