We address the challenge of quantifying Bayesian uncertainty and incorporating it in offline use cases of finite-state Markov Decision Processes (MDPs) with unknown dynamics. Our approach provides a principled method to disentangle epistemic and aleatoric uncertainty, and a novel technique to find policies that optimise Bayesian posterior expected value without relying on strong assumptions about the MDP's posterior distribution. First, we utilise standard Bayesian reinforcement learning methods to capture the posterior uncertainty in MDP parameters based on available data. We then analytically compute the first two moments of the return distribution across posterior samples and apply the law of total variance to disentangle aleatoric and epistemic uncertainties. To find policies that maximise posterior expected value, we leverage the closed-form expression for value as a function of policy. This allows us to propose a stochastic gradient-based approach for solving the problem. We illustrate the uncertainty quantification and Bayesian posterior value optimisation performance of our agent in simple, interpretable gridworlds and validate it through ground-truth evaluations on synthetic MDPs. Finally, we highlight the real-world impact and computational scalability of our method by applying it to the AI Clinician problem, which recommends treatment for patients in intensive care units and has emerged as a key use case of finite-state MDPs with offline data. We discuss the challenges that arise with Bayesian modelling of larger scale MDPs while demonstrating the potential to apply our methods rooted in Bayesian decision theory into the real world. We make our code available at https://github.com/filippovaldettaro/finite-state-mdps .

翻译：本文针对有限状态马尔可夫决策过程（MDP）在动态未知情况下，离线应用场景中贝叶斯不确定性量化及其融合的挑战展开研究。我们提出了一种原则性方法以分离认知不确定性与偶然不确定性，并发展了一种新技术，用于寻找优化贝叶斯后验期望价值的策略，且无需依赖关于MDP后验分布的强假设。首先，我们利用标准贝叶斯强化学习方法，基于现有数据捕捉MDP参数的后验不确定性。随后，我们解析计算后验样本中回报分布的一阶矩与二阶矩，并应用全方差定律以分离偶然不确定性与认知不确定性。为寻找最大化后验期望价值的策略，我们利用价值作为策略函数的闭式表达式，从而提出一种基于随机梯度的求解方法。我们在简单、可解释的网格世界中展示了智能体的不确定性量化与贝叶斯后验价值优化性能，并通过合成MDP的真实评估验证了其有效性。最后，我们将方法应用于AI临床医生问题——该问题旨在为重症监护病房患者推荐治疗方案，已成为有限状态MDP离线数据应用的关键场景——以凸显本方法的实际影响与计算可扩展性。在展示基于贝叶斯决策理论的方法应用于现实世界潜力的同时，我们也探讨了大规模MDP贝叶斯建模所面临的挑战。代码已公开于 https://github.com/filippovaldettaro/finite-state-mdps。