What are the functionals of the reward that can be computed and optimized exactly in Markov Decision Processes?In the finite-horizon, undiscounted setting, Dynamic Programming (DP) can only handle these operations efficiently for certain classes of statistics. We summarize the characterization of these classes for policy evaluation, and give a new answer for the planning problem. Interestingly, we prove that only generalized means can be optimized exactly, even in the more general framework of Distributional Reinforcement Learning (DistRL).DistRL permits, however, to evaluate other functionals approximately. We provide error bounds on the resulting estimators, and discuss the potential of this approach as well as its limitations.These results contribute to advancing the theory of Markov Decision Processes by examining overall characteristics of the return, and particularly risk-conscious strategies.

翻译：在马尔可夫决策过程中，哪些回报的泛函可以被精确计算和优化？在有限时域、无折扣设定下，动态规划（DP）仅能高效处理特定类别的统计量。我们总结了策略评估中这些类别的表征，并为规划问题给出了新答案。有趣的是，我们证明即便在更一般的分布强化学习（DistRL）框架下，也只有广义均值能被精确优化。然而，DistRL允许近似评估其他泛函。我们给出了所得估计量的误差界，并讨论了该方法的潜力及其局限性。这些结果通过考察回报的整体特征（特别是风险意识策略）推进了马尔可夫决策过程的理论发展。