Offline constrained reinforcement learning (RL) aims to learn a policy that maximizes the expected cumulative reward subject to constraints on expected cumulative cost using an existing dataset. In this paper, we propose Primal-Dual-Critic Algorithm (PDCA), a novel algorithm for offline constrained RL with general function approximation. PDCA runs a primal-dual algorithm on the Lagrangian function estimated by critics. The primal player employs a no-regret policy optimization oracle to maximize the Lagrangian estimate and the dual player acts greedily to minimize the Lagrangian estimate. We show that PDCA can successfully find a near saddle point of the Lagrangian, which is nearly optimal for the constrained RL problem. Unlike previous work that requires concentrability and a strong Bellman completeness assumption, PDCA only requires concentrability and realizability assumptions for sample-efficient learning.

翻译：离线约束强化学习旨在利用现有数据集，学习一个在满足预期累积成本约束条件下最大化预期累积奖励的策略。本文提出了一种新颖的离线约束强化学习算法——主-对偶-评论家算法（PDCA），该算法采用通用函数逼近。PDCA在由评论家估计的拉格朗日函数上运行主-对偶算法。主玩家采用无遗憾策略优化预言机以最大化拉格朗日估计值，而对偶玩家则贪婪地最小化该估计值。我们证明，PDCA能够成功找到拉格朗日函数的近似鞍点，该点对于约束强化学习问题几乎是全局最优的。与以往需要可集中性假设和强贝尔曼完备性假设的工作不同，PDCA仅需可集中性和可实现性假设即可实现样本高效学习。