Optimizing static risk-averse objectives in Markov decision processes is difficult because they do not admit standard dynamic programming equations common in Reinforcement Learning (RL) algorithms. Dynamic programming decompositions that augment the state space with discrete risk levels have recently gained popularity in the RL community. Prior work has shown that these decompositions are optimal when the risk level is discretized sufficiently. However, we show that these popular decompositions for Conditional-Value-at-Risk (CVaR) and Entropic-Value-at-Risk (EVaR) are inherently suboptimal regardless of the discretization level. In particular, we show that a saddle point property assumed to hold in prior literature may be violated. However, a decomposition does hold for Value-at-Risk and our proof demonstrates how this risk measure differs from CVaR and EVaR. Our findings are significant because risk-averse algorithms are used in high-stake environments, making their correctness much more critical.

翻译：在马尔可夫决策过程中优化静态风险厌恶目标具有挑战性，因为这类目标不满足强化学习算法中常见的标准动态规划方程。近年来，通过添加离散风险水平扩充状态空间的动态规划分解方法在强化学习领域逐渐流行。现有研究表明，当风险水平得到充分离散化时，这些分解方法具有最优性。然而，我们证明针对条件风险价值（CVaR）和熵风险价值（EVaR）的这类流行分解方法存在本质上的次优性，且与离散化精度无关。具体而言，我们指出现有文献中假设成立的鞍点性质可能被违反。但风险价值（VaR）确实存在有效的动态规划分解，我们的证明揭示了该风险度量与CVaR和EVaR的本质差异。本研究成果具有重要意义，因为风险厌恶算法常用于高风险环境，其正确性至关重要。