We consider the optimal sample complexity theory of tabular reinforcement learning (RL) for controlling the infinite horizon discounted reward in a Markov decision process (MDP). Optimal min-max complexity results have been developed for tabular RL in this setting, leading to a sample complexity dependence on $\gamma$ and $\epsilon$ of the form $\tilde \Theta((1-\gamma)^{-3}\epsilon^{-2})$, where $\gamma$ is the discount factor and $\epsilon$ is the tolerance solution error. However, in many applications of interest, the optimal policy (or all policies) will induce mixing. We show that in these settings the optimal min-max complexity is $\tilde \Theta(t_{\text{minorize}}(1-\gamma)^{-2}\epsilon^{-2})$, where $t_{\text{minorize}}$ is a measure of mixing that is within an equivalent factor of the total variation mixing time. Our analysis is based on regeneration-type ideas, that, we believe are of independent interest since they can be used to study related problems for general state space MDPs.

翻译：我们研究表格型强化学习（RL）在无限水平折扣奖励马尔可夫决策过程（MDP）控制中的最优样本复杂度理论。针对该场景下的表格型RL，已建立起最优极小极大复杂度结果，其样本复杂度对折扣因子$\gamma$和容差求解误差$\epsilon$的依赖形式为$\tilde \Theta((1-\gamma)^{-3}\epsilon^{-2})$。然而在许多实际应用中，最优策略（或所有策略）均会诱导混合特性。我们证明在这些场景下，最优极小极大复杂度为$\tilde \Theta(t_{\text{minorize}}(1-\gamma)^{-2}\epsilon^{-2})$，其中$t_{\text{minorize}}$是混合程度的度量，其量级与总变差混合时间等价。我们的分析基于再生型思想，该思想具有独立研究价值，可推广至一般状态空间MDP相关问题的研究。