We introduce the Value-at-Risk Constrained Policy Optimization algorithm (VaR-CPO), a sample efficient and conservative method designed to optimize Value-at-Risk (VaR) constrained reinforcement learning (RL) problems. Empirically, we demonstrate that VaR-CPO is capable of safe exploration, achieving zero constraint violations during training in feasible environments, a critical property that baseline methods fail to uphold. To overcome the inherent non-differentiability of the VaR constraint, we employ Cantelli's inequality to obtain a tractable approximation based on the first two moments of the cost return. Additionally, by extending the trust-region framework of the Constrained Policy Optimization (CPO) method, we provide worst-case bounds for both policy improvement and constraint violation during the training process.

翻译：我们提出风险价值约束策略优化算法（VaR-CPO），一种样本高效且保守的方法，专为优化风险价值（VaR）约束下的强化学习（RL）问题而设计。实验表明，VaR-CPO能够实现安全探索，在可行环境中训练期间达到零约束违反，这是基线方法未能满足的关键特性。为克服VaR约束固有的不可微性，我们利用坎泰利不等式，基于成本回报的前两个矩获得可处理的近似。此外，通过扩展约束策略优化（CPO）方法的信任域框架，我们为训练过程中的策略改进和约束违反提供了最坏情况下的界。