We study a class of constrained reinforcement learning (RL) problems in which multiple constraint specifications are not identified before training. It is challenging to identify appropriate constraint specifications due to the undefined trade-off between the reward maximization objective and the constraint satisfaction, which is ubiquitous in constrained decision-making. To tackle this issue, we propose a new constrained RL approach that searches for policy and constraint specifications together. This method features the adaptation of relaxing the constraint according to a relaxation cost introduced in the learning objective. Since this feature mimics how ecological systems adapt to disruptions by altering operation, our approach is termed as resilient constrained RL. Specifically, we provide a set of sufficient conditions that balance the constraint satisfaction and the reward maximization in notion of resilient equilibrium, propose a tractable formulation of resilient constrained policy optimization that takes this equilibrium as an optimal solution, and advocate two resilient constrained policy search algorithms with non-asymptotic convergence guarantees on the optimality gap and constraint satisfaction. Furthermore, we demonstrate the merits and the effectiveness of our approach in computational experiments.

翻译：我们研究了一类约束强化学习问题，其中多个约束规范在训练前无法预先确定。由于奖励最大化目标与约束满足之间存在未定义的权衡（这在约束决策问题中普遍存在），识别恰当的约束规范具有挑战性。为解决该问题，我们提出了一种联合搜索策略与约束规范的新型约束强化学习方法。该方法的核心特征在于：根据学习目标中引入的松弛成本自适应地放松约束。由于该特征模拟了生态系统通过调整运作方式适应干扰的行为，我们将其命名为弹性约束强化学习。具体而言，我们提出了在弹性均衡概念下平衡约束满足与奖励最大化的一组充分条件，构建了以该均衡为最优解的可处理弹性约束策略优化框架，并提出了两种具有非渐近最优性差距与约束满足收敛保证的弹性约束策略搜索算法。最后，通过计算实验验证了该方法的优势与有效性。