Diffusion models have emerged as a prominent class of generative models, surpassing previous methods regarding sample quality and training stability. Recent works have shown the advantages of diffusion models in improving reinforcement learning (RL) solutions, including as trajectory planners, expressive policy classes, data synthesizers, etc. This survey aims to provide an overview of the advancements in this emerging field and hopes to inspire new avenues of research. First, we examine several challenges encountered by current RL algorithms. Then, we present a taxonomy of existing methods based on the roles played by diffusion models in RL and explore how the existing challenges are addressed. We further outline successful applications of diffusion models in various RL-related tasks while discussing the limitations of current approaches. Finally, we conclude the survey and offer insights into future research directions, focusing on enhancing model performance and applying diffusion models to broader tasks. We are actively maintaining a GitHub repository for papers and other related resources in applying diffusion models in RL: https://github.com/apexrl/Diff4RLSurvey .

翻译：扩散模型已崛起为生成模型中的突出类别，在样本质量和训练稳定性方面超越了先前方法。近期研究展示了扩散模型在改进强化学习（RL）解决方案中的优势，包括作为轨迹规划器、表达性策略类、数据合成器等。本综述旨在概述这一新兴领域的进展，并期望激发新的研究方向。首先，我们考察了当前RL算法面临的若干挑战。随后，我们基于扩散模型在RL中扮演的角色提出了现有方法的分类体系，并探讨了现有挑战是如何被解决的。我们进一步概述了扩散模型在各种RL相关任务中的成功应用，同时讨论了当前方法的局限性。最后，我们总结综述并展望未来研究方向，重点关注提升模型性能及将扩散模型应用于更广泛任务。我们正在积极维护一个GitHub仓库，收录扩散模型在RL中应用的相关论文及其他资源：https://github.com/apexrl/Diff4RLSurvey 。