This paper introduces a novel backup strategy for Monte-Carlo Tree Search (MCTS) designed for highly stochastic and partially observable Markov decision processes. We adopt a probabilistic approach, modeling both value and action-value nodes as Gaussian distributions. We introduce a novel backup operator that computes value nodes as the Wasserstein barycenter of their action-value children nodes; thus, propagating the uncertainty of the estimate across the tree to the root node. We study our novel backup operator when using a novel combination of $L^1$-Wasserstein barycenter with $\alpha$-divergence, by drawing a notable connection to the generalized mean backup operator. We complement our probabilistic backup operator with two sampling strategies, based on optimistic selection and Thompson sampling, obtaining our Wasserstein MCTS algorithm. We provide theoretical guarantees of asymptotic convergence to the optimal policy, and an empirical evaluation on several stochastic and partially observable environments, where our approach outperforms well-known related baselines.

翻译：本文提出了一种针对高度随机且部分可观测马尔可夫决策过程的新型蒙特卡洛树搜索（MCTS）回溯策略。我们采用概率建模方法，将价值节点与动作价值节点均表示为高斯分布。我们引入了一个创新的回溯算子，该算子通过计算子动作价值节点的Wasserstein重心来得到价值节点，从而将估计的不确定性沿树传播至根节点。通过建立与广义均值回溯算子的显著关联，我们研究了在$L^1$-Wasserstein重心与$\alpha$-散度新型组合下该回溯算子的特性。为了补充这一概率回溯算子，我们基于乐观选择策略和汤普森采样提出了两种采样方法，最终构建出Wasserstein MCTS算法。我们给出了该算法渐近收敛到最优策略的理论保证，并在多个随机性与部分可观测性环境中进行了实验评估，结果表明本方法显著优于现有的相关基线算法。