The optimized certainty equivalent (OCE) is a family of risk measures that cover important examples such as entropic risk, conditional value-at-risk and mean-variance models. In this paper, we propose a new episodic risk-sensitive reinforcement learning formulation based on tabular Markov decision processes with recursive OCEs. We design an efficient learning algorithm for this problem based on value iteration and upper confidence bound. We derive an upper bound on the regret of the proposed algorithm, and also establish a minimax lower bound. Our bounds show that the regret rate achieved by our proposed algorithm has optimal dependence on the number of episodes and the number of actions.

翻译：优化确定性等价（OCE）是一类风险度量，涵盖熵风险、条件风险价值和均值-方差模型等重要实例。本文提出一种基于表格型马尔可夫决策过程与递归OCE的新的回合制风险敏感强化学习框架。我们基于值迭代和上置信界为该问题设计了一种高效学习算法，推导了该算法的遗憾上界，并建立了极小化下界。理论界表明，我们提出的算法所实现的遗憾率在回合数与动作数上具有最优依赖性。