Modified policy iteration (MPI) also known as optimistic policy iteration is at the core of many reinforcement learning algorithms. It works by combining elements of policy iteration and value iteration. The convergence of MPI has been well studied in the case of discounted and average-cost MDPs. In this work, we consider the exponential cost risk-sensitive MDP formulation, which is known to provide some robustness to model parameters. Although policy iteration and value iteration have been well studied in the context of risk sensitive MDPs, modified policy iteration is relatively unexplored. We provide the first proof that MPI also converges for the risk-sensitive problem in the case of finite state and action spaces. Since the exponential cost formulation deals with the multiplicative Bellman equation, our main contribution is a convergence proof which is quite different than existing results for discounted and risk-neutral average-cost problems. The proof of approximate modified policy iteration for risk sensitive MDPs is also provided in the appendix.

翻译：修正策略迭代（MPI）也称为乐观策略迭代，是许多强化学习算法的核心。它通过结合策略迭代和值迭代的元素来工作。在折扣成本与平均成本MDP情况下，MPI的收敛性已得到充分研究。本文考虑指数代价风险敏感MDP模型，该模型已知能为模型参数提供一定的鲁棒性。尽管策略迭代和值迭代在风险敏感MDP背景下已得到充分研究，但修正策略迭代仍相对未探索。我们首次证明了MPI在有限状态和动作空间的风险敏感问题中同样收敛。由于指数代价模型涉及乘法贝尔曼方程，我们的主要贡献在于一个与现有折扣成本及风险中性平均成本问题的结果截然不同的收敛性证明。附录中还提供了风险敏感MDP的近似修正策略迭代证明。