We study the Constrained Convex Markov Decision Process (MDP), where the goal is to minimize a convex functional of the visitation measure, subject to a convex constraint. Designing algorithms for a constrained convex MDP faces several challenges, including (1) handling the large state space, (2) managing the exploration/exploitation tradeoff, and (3) solving the constrained optimization where the objective and the constraint are both nonlinear functions of the visitation measure. In this work, we present a model-based algorithm, Variational Primal-Dual Policy Optimization (VPDPO), in which Lagrangian and Fenchel duality are implemented to reformulate the original constrained problem into an unconstrained primal-dual optimization. Moreover, the primal variables are updated by model-based value iteration following the principle of Optimism in the Face of Uncertainty (OFU), while the dual variables are updated by gradient ascent. Moreover, by embedding the visitation measure into a finite-dimensional space, we can handle large state spaces by incorporating function approximation. Two notable examples are (1) Kernelized Nonlinear Regulators and (2) Low-rank MDPs. We prove that with an optimistic planning oracle, our algorithm achieves sublinear regret and constraint violation in both cases and can attain the globally optimal policy of the original constrained problem.

翻译：我们研究约束凸马尔可夫决策过程（MDP），其目标是最小化访问测度的凸泛函，同时满足凸约束。为约束凸MDP设计算法面临若干挑战，包括：（1）处理大规模状态空间；（2）管理探索与利用的权衡；（3）求解目标和约束均为访问测度非线性函数的约束优化问题。本文提出一种基于模型的算法——变分原始-对偶策略优化（VPDPO），通过引入拉格朗日对偶与芬切尔对偶，将原始约束问题重新表述为无约束的原始-对偶优化。其中，原始变量采用遵循“面对不确定性保持乐观”（OFU）原则的基于模型的值迭代更新，而对偶变量则通过梯度上升更新。此外，通过将访问测度嵌入有限维空间，我们可利用函数近似处理大规模状态空间。两个典型实例包括：（1）核化非线性调节器与（2）低秩MDP。我们证明，在具备乐观规划预言机的条件下，该算法在以上两种情形中均可实现次线性遗憾与约束违背，并能达到原始约束问题的全局最优策略。