This paper is concerned with the sample efficiency of reinforcement learning, assuming access to a generative model (or simulator). We first consider $\gamma$-discounted infinite-horizon Markov decision processes (MDPs) with state space $\mathcal{S}$ and action space $\mathcal{A}$. Despite a number of prior works tackling this problem, a complete picture of the trade-offs between sample complexity and statistical accuracy is yet to be determined. In particular, all prior results suffer from a severe sample size barrier, in the sense that their claimed statistical guarantees hold only when the sample size exceeds at least $\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^2}$. The current paper overcomes this barrier by certifying the minimax optimality of two algorithms -- a perturbed model-based algorithm and a conservative model-based algorithm -- as soon as the sample size exceeds the order of $\frac{|\mathcal{S}||\mathcal{A}|}{1-\gamma}$ (modulo some log factor). Moving beyond infinite-horizon MDPs, we further study time-inhomogeneous finite-horizon MDPs, and prove that a plain model-based planning algorithm suffices to achieve minimax-optimal sample complexity given any target accuracy level. To the best of our knowledge, this work delivers the first minimax-optimal guarantees that accommodate the entire range of sample sizes (beyond which finding a meaningful policy is information theoretically infeasible).

翻译：本文关注强化学习的样本效率问题，假设可访问生成模型（或模拟器）。我们首先考虑具有状态空间$\mathcal{S}$和动作空间$\mathcal{A}$的$\gamma$折扣无限时域马尔可夫决策过程（MDP）。尽管已有大量研究解决该问题，但样本复杂度与统计精度之间的权衡关系尚未完全明确。具体而言，所有先前的研究结果都受到严重的样本量壁垒制约——其声称的统计保证仅在样本量至少达到$\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^2}$时才成立。本文通过证明两种算法（一种基于扰动的模型算法和一种保守的模型算法）在样本量超过$\frac{|\mathcal{S}||\mathcal{A}|}{1-\gamma}$量级（忽略对数因子）后即达到极小化最优性，从而突破了这一壁垒。超越无限时域MDP，我们进一步研究时间非齐次有限时域MDP，并证明简单的基于模型的规划算法即可在任意目标精度下实现极小化最优样本复杂度。据我们所知，本文首次提供了覆盖完整样本量范围（超出该范围后寻找有意义策略在信息论上不可行）的极小化最优性保证。