We study the sample complexity of learning an $\varepsilon$-optimal policy in an average-reward Markov decision process (MDP) under a generative model. We establish the complexity bound $\widetilde{O}\left(SA\frac{H}{\varepsilon^2} \right)$, where $H$ is the span of the bias function of the optimal policy and $SA$ is the cardinality of the state-action space. Our result is the first that is minimax optimal (up to log factors) in all parameters $S,A,H$ and $\varepsilon$, improving on existing work that either assumes uniformly bounded mixing times for all policies or has suboptimal dependence on the parameters. Our result is based on reducing the average-reward MDP to a discounted MDP. To establish the optimality of this reduction, we develop improved bounds for $\gamma$-discounted MDPs, showing that $\widetilde{O}\left(SA\frac{H}{(1-\gamma)^2\varepsilon^2} \right)$ samples suffice to learn a $\varepsilon$-optimal policy in weakly communicating MDPs under the regime that $\gamma \geq 1 - \frac{1}{H}$, circumventing the well-known lower bound of $\widetilde{\Omega}\left(SA\frac{1}{(1-\gamma)^3\varepsilon^2} \right)$ for general $\gamma$-discounted MDPs. Our analysis develops upper bounds on certain instance-dependent variance parameters in terms of the span parameter. These bounds are tighter than those based on the mixing time or diameter of the MDP and may be of broader use.

翻译：我们研究了在生成模型下学习平均奖励马尔可夫决策过程（MDP）中$\varepsilon$-最优策略的样本复杂度。我们建立了复杂度边界$\widetilde{O}\left(SA\frac{H}{\varepsilon^2} \right)$，其中$H$为最优策略偏置函数的跨度，$SA$为状态-动作空间的基数。该结果是首个在所有参数$S,A,H$和$\varepsilon$上达到极小极大最优（忽略对数因子）的成果，改进了现有工作要么假设所有策略具有一致有界混合时间、要么参数依赖关系次优的局限。我们的结果基于将平均奖励MDP简化为折扣MDP。为验证该约化的最优性，我们发展了$\gamma$-折扣MDP的改进边界，证明在$\gamma \geq 1 - \frac{1}{H}$的弱连通MDP中，$\widetilde{O}\left(SA\frac{H}{(1-\gamma)^2\varepsilon^2} \right)$个样本足以学习$\varepsilon$-最优策略，从而规避了通用$\gamma$-折扣MDP中已知的$\widetilde{\Omega}\left(SA\frac{1}{(1-\gamma)^3\varepsilon^2} \right)$下界。我们的分析通过跨度参数建立了某些实例相关方差参数的上界，这些边界比基于MDP混合时间或直径的边界更紧，并可能具有更广泛的应用价值。