Online planning is crucial for high performance in many complex sequential decision-making tasks. Monte Carlo Tree Search (MCTS) employs a principled mechanism for trading off exploration for exploitation for efficient online planning, and it outperforms comparison methods in many discrete decision-making domains such as Go, Chess, and Shogi. Subsequently, extensions of MCTS to continuous domains have been developed. However, the inherent high branching factor and the resulting explosion of the search tree size are limiting the existing methods. To address this problem, we propose Continuous Monte Carlo Graph Search (CMCGS), an extension of MCTS to online planning in environments with continuous state and action spaces. CMCGS takes advantage of the insight that, during planning, sharing the same action policy between several states can yield high performance. To implement this idea, at each time step, CMCGS clusters similar states into a limited number of stochastic action bandit nodes, which produce a layered directed graph instead of an MCTS search tree. Experimental evaluation shows that CMCGS outperforms comparable planning methods in several complex continuous DeepMind Control Suite benchmarks and 2D navigation and exploration tasks with limited sample budgets. Furthermore, CMCGS can be scaled up through parallelization, and it outperforms the Cross-Entropy Method (CEM) in continuous control with learned dynamics models.

翻译：在线规划对于许多复杂序列决策任务的高性能表现至关重要。蒙特卡洛树搜索（MCTS）采用一种原则性机制来权衡探索与利用，从而实现高效的在线规划，并在围棋、国际象棋和将棋等众多离散决策领域中超越了对比方法。随后，MCTS的扩展版本被开发用于连续域。然而，固有的高分支因子及其导致的搜索树规模爆炸性增长限制了现有方法。为解决这一问题，我们提出连续蒙特卡洛图搜索（CMCGS），这是MCTS在连续状态与动作空间环境中进行在线规划的扩展。CMCGS利用了一个关键见解：在规划过程中，多个状态共享相同的动作策略可以实现高性能。为实现这一思想，CMCGS在每一步将相似状态聚类为有限数量的随机动作臂节点，从而生成一个分层有向图，而非MCTS的搜索树。实验评估表明，CMCGS在多个复杂的连续型DeepMind控制套件基准测试以及有限样本预算的二维导航与探索任务中，优于可比规划方法。此外，CMCGS可通过并行化进行扩展，并在使用学习动力学模型的连续控制中超越了交叉熵方法（CEM）。