This paper considers the best policy identification (BPI) problem in online Constrained Markov Decision Processes (CMDPs). We are interested in algorithms that are model-free, have low regret, and identify an optimal policy with a high probability. Existing model-free algorithms for online CMDPs with sublinear regret and constraint violation do not provide any convergence guarantee to an optimal policy and provide only average performance guarantees when a policy is uniformly sampled at random from all previously used policies. In this paper, we develop a new algorithm, named Pruning-Refinement-Identification (PRI), based on a fundamental structural property of CMDPs proved in Koole(1988); Ross(1989), which we call limited stochasticity. The property says for a CMDP with $N$ constraints, there exists an optimal policy with at most $N$ stochastic decisions. The proposed algorithm first identifies at which step and in which state a stochastic decision has to be taken and then fine-tunes the distributions of these stochastic decisions. PRI achieves trio objectives: (i) PRI is a model-free algorithm; and (ii) it outputs a near-optimal policy with a high probability at the end of learning; and (iii) in the tabular setting, PRI guarantees $\tilde{\mathcal{O}}(\sqrt{K})$ regret and constraint violation, which significantly improves the best existing regret bound $\tilde{\mathcal{O}}(K^{\frac{4}{5}})$ under a model-free algorithm, where $K$ is the total number of episodes.

翻译：本文考虑在线约束马尔可夫决策过程（CMDPs）中的最佳策略识别（BPI）问题。我们关注那些无需模型、具有低遗憾并能以高概率识别最优策略的算法。现有针对在线CMDPs的无需模型算法虽能实现次线性遗憾与约束违反，但无法保证收敛到最优策略，且当从所有历史策略中均匀随机采样时仅能提供平均性能保证。本文基于Koole(1988)与Ross(1989)证明的CMDPs基本结构性质（我们称之为有限随机性），提出一种名为"剪枝-精炼-识别（PRI）"的新算法。该性质表明：对于含$N$个约束的CMDP，存在一个最多包含$N$个随机决策的最优策略。所提算法首先识别需在哪个步骤及状态下采取随机决策，随后对这些随机决策的分布进行微调。PRI实现了三重目标：（i）PRI是一种无需模型算法；（ii）在学习结束时能以高概率输出近最优策略；（iii）在表格设置下，PRI保证$\tilde{\mathcal{O}}(\sqrt{K})$级别的遗憾与约束违反，显著优于现有无需模型算法$\tilde{\mathcal{O}}(K^{\frac{4}{5}})$的最佳遗憾界，其中$K$为总回合数。