This paper considers the best policy identification (BPI) problem in online Constrained Markov Decision Processes (CMDPs). We are interested in algorithms that are model-free, have low regret, and identify an optimal policy with a high probability. Existing model-free algorithms for online CMDPs with sublinear regret and constraint violation do not provide any convergence guarantee to an optimal policy and provide only average performance guarantees when a policy is uniformly sampled at random from all previously used policies. In this paper, we develop a new algorithm, named Pruning-Refinement-Identification (PRI), based on a fundamental structural property of CMDPs we discover, called limited stochasticity. The property says for a CMDP with $N$ constraints, there exists an optimal policy with at most $N$ stochastic decisions. The proposed algorithm first identifies at which step and in which state a stochastic decision has to be taken and then fine-tunes the distributions of these stochastic decisions. PRI achieves trio objectives: (i) PRI is a model-free algorithm; and (ii) it outputs a near-optimal policy with a high probability at the end of learning; and (iii) in the tabular setting, PRI guarantees $\tilde{\mathcal{O}}(\sqrt{K})$ regret and constraint violation, which significantly improves the best existing regret bound $\tilde{\mathcal{O}}(K^{\frac{4}{5}})$ under a model-free algorithm, where $K$ is the total number of episodes.

翻译：本文研究在线约束马尔可夫决策过程（CMDPs）中的最佳策略识别问题。我们关注具有低遗憾、高概率识别最优策略的无模型算法。现有的在线CMDPs无模型算法虽能实现次线性遗憾和约束违反，但无法保证收敛到最优策略，且当从所有历史策略中均匀随机采样时仅提供平均性能保证。本文基于发现的CMDPs基本结构性质——有限随机性，提出新算法PRI（剪枝-细化-识别）。该性质指出：对于含$N$个约束的CMDP，存在最多包含$N$个随机决策的最优策略。所提算法首先识别需要做出随机决策的步骤与状态，随后微调这些随机决策的分布。PRI实现三重目标：（i）PRI为无模型算法；（ii）学习结束后能以高概率输出近最优策略；（iii）在表格型环境中，PRI保证$\tilde{\mathcal{O}}(\sqrt{K})$的遗憾和约束违反，相较于现有无模型算法最优结果$\tilde{\mathcal{O}}(K^{\frac{4}{5}})$有显著提升，其中$K$为总回合数。