Model-based reinforcement learning (RL) methods that leverage search are responsible for many milestone breakthroughs in RL. Sequential Monte Carlo (SMC) recently emerged as an alternative to the Monte Carlo Tree Search (MCTS) algorithm which drove these breakthroughs. SMC is easier to parallelize and more suitable to GPU acceleration. However, it also suffers from large variance and path degeneracy which prevent it from scaling well with increased search depth, i.e., increased sequential compute. To address these problems, we introduce Twice Sequential Monte Carlo Tree Search (TSMCTS). Across discrete and continuous environments TSMCTS outperforms the SMC baseline as well as a popular modern version of MCTS as a policy improvement operator, scales favorably with sequential compute, reduces estimator variance and mitigates the effects of path degeneracy while retaining the properties that make SMC natural to parallelize.

翻译：基于模型且利用搜索的强化学习方法推动了强化学习领域的诸多里程碑式突破。序贯蒙特卡洛方法作为驱动这些突破的蒙特卡洛树搜索算法的替代方案近期崭露头角。SMC 更易于并行化，也更适合 GPU 加速。然而，它也存在方差大和路径退化的问题，这阻碍了其随搜索深度（即序贯计算量）增加而有效扩展。为解决这些问题，我们提出了双重序贯蒙特卡洛树搜索。在离散和连续环境中，TSMCTS 作为策略改进算子，其性能优于 SMC 基线方法及一种流行的现代 MCTS 版本，能够随序贯计算量增加而良好扩展，降低估计器方差，缓解路径退化的影响，同时保留了 SMC 固有的易于并行化的特性。