The Constrained Markov Decision Process (CMDP) formulation allows to solve safety-critical decision making tasks that are subject to constraints. While CMDPs have been extensively studied in the Reinforcement Learning literature, little attention has been given to sampling-based planning algorithms such as MCTS for solving them. Previous approaches perform conservatively with respect to costs as they avoid constraint violations by using Monte Carlo cost estimates that suffer from high variance. We propose Constrained MCTS (C-MCTS), which estimates cost using a safety critic that is trained with Temporal Difference learning in an offline phase prior to agent deployment. The critic limits exploration by pruning unsafe trajectories within MCTS during deployment. C-MCTS satisfies cost constraints but operates closer to the constraint boundary, achieving higher rewards than previous work. As a nice byproduct, the planner is more efficient w.r.t. planning steps. Most importantly, under model mismatch between the planner and the real world, C-MCTS is less susceptible to cost violations than previous work.

翻译：约束马尔可夫决策过程（CMDP）框架能够解决需满足约束条件的安全关键型决策任务。尽管CMDP在强化学习文献中已有广泛研究，但针对其求解的基于采样的规划算法（如MCTS）却鲜受关注。现有方法因采用方差较大的蒙特卡洛成本估计来规避约束违规，导致其保守性过高。本文提出约束MCTS（C-MCTS），该方法在智能体部署前的离线阶段，通过时序差分学习训练安全评判器来估计成本。在部署阶段，该评判器通过剪枝不可行轨迹限制探索范围。C-MCTS既能满足成本约束，又能更接近约束边界运行，较现有方法获得更高回报。此外，该规划器在规划步骤上具有更高效率。尤为重要的是，当规划器与真实世界存在模型失配时，C-MCTS相比现有方法不易发生成本违规。