In restless bandits, a central agent is tasked with optimally distributing limited resources across several bandits (arms), with each arm being a Markov decision process. In this work, we generalize the traditional restless bandits problem with a risk-neutral objective by incorporating risk-awareness, which is particularly important in various real-world applications especially when the decision maker seeks to mitigate downside risks. We establish indexability conditions for the case of a risk-aware objective and provide a solution based on Whittle index for the first time for the planning problem with finite-horizon non-stationary and for infinite-horizon stationary Markov decision processes. In addition, we address the learning problem when the true transition probabilities are unknown by proposing a Thompson sampling approach and show that it achieves bounded regret that scales sublinearly with the number of episodes and quadratically with the number of arms. The efficacy of our method in reducing risk exposure in restless bandits is illustrated through a set of numerical experiments in the contexts of machine replacement and patient scheduling applications under both planning and learning setups.

翻译：在不安定老虎机问题中，中心智能体需要将有限资源最优分配到多个老虎机（臂）上，每个臂均为一个马尔可夫决策过程。本研究通过引入风险感知机制，对传统以风险中性为目标的不安定老虎机问题进行推广，这在众多现实应用中尤为重要，特别是当决策者需要规避下行风险时。我们首次针对风险感知目标建立了可索引性条件，并为有限时域非平稳及无限时域平稳马尔可夫决策过程的规划问题提供了基于Whittle索引的解决方案。此外，我们通过提出一种Thompson采样方法解决了真实转移概率未知时的学习问题，并证明该方法能实现有界遗憾，其随回合数呈次线性增长，随臂数呈二次增长。通过在机器更换和患者调度两类应用场景中，分别针对规划与学习设置进行数值实验，验证了所提方法在降低不安定老虎机风险暴露方面的有效性。