The optimized certainty equivalent (OCE) is a family of risk measures that cover important examples such as entropic risk, conditional value-at-risk and mean-variance models. In this paper, we propose a new episodic risk-sensitive reinforcement learning formulation based on tabular Markov decision processes with recursive OCEs. We design an efficient learning algorithm for this problem based on value iteration and upper confidence bound. We derive an upper bound on the regret of the proposed algorithm, and also establish a minimax lower bound. Our bounds show that the regret rate achieved by our proposed algorithm has optimal dependence on the number of episodes and the number of actions.

翻译：优化确定等价类（OCE）是一类风险度量，涵盖熵风险、条件风险价值及均值-方差模型等重要实例。本文基于表格型马尔可夫决策过程与递归OCE，提出一种新的情节式风险敏感强化学习框架。我们为此问题设计了一种基于值迭代和上置信界的高效学习算法，推导了所提算法遗憾的上界，并建立了极小化最优下界。理论界表明，本算法实现的遗憾率在情节数和动作数上具有最优依赖性。