We consider the optimal sample complexity theory of tabular reinforcement learning (RL) for controlling the infinite horizon discounted reward in a Markov decision process (MDP). Optimal min-max complexity results have been developed for tabular RL in this setting, leading to a sample complexity dependence on $\gamma$ and $\epsilon$ of the form $\tilde \Theta((1-\gamma)^{-3}\epsilon^{-2})$, where $\gamma$ is the discount factor and $\epsilon$ is the tolerance solution error. However, in many applications of interest, the optimal policy (or all policies) will induce mixing. We show that in these settings the optimal min-max complexity is $\tilde \Theta(t_{\text{minorize}}(1-\gamma)^{-2}\epsilon^{-2})$, where $t_{\text{minorize}}$ is a measure of mixing that is within an equivalent factor of the total variation mixing time. Our analysis is based on regeneration-type ideas, that, we believe are of independent interest since they can be used to study related problems for general state space MDPs.

翻译：我们研究了表格型强化学习在控制无限时域折扣回报马尔可夫决策过程中的最优样本复杂度理论。针对该场景下的表格型强化学习，已有研究建立了最优最小最大复杂度结果，其样本复杂度对折扣因子γ和容差解误差ε的依赖关系为$\tilde \Theta((1-\gamma)^{-3}\epsilon^{-2})$。然而，在许多实际应用中，最优策略（或所有策略）会引发混合效应。我们证明，在这些场景下，最优最小最大复杂度为$\tilde \Theta(t_{\text{minorize}}(1-\gamma)^{-2}\epsilon^{-2})$，其中$t_{\text{minorize}}$是与全变差混合时间等价因子内的一种混合度量。我们的分析基于再生型方法，该方法对一般状态空间马尔可夫决策过程的相关问题研究具有独立参考价值。