Diffusion models have emerged as a prominent class of generative models, surpassing previous methods regarding sample quality and training stability. Recent works have shown the advantages of diffusion models in improving reinforcement learning (RL) solutions, including as trajectory planners, expressive policy classes, data synthesizers, etc. This survey aims to provide an overview of the advancements in this emerging field and hopes to inspire new avenues of research. First, we examine several challenges encountered by current RL algorithms. Then, we present a taxonomy of existing methods based on the roles played by diffusion models in RL and explore how the existing challenges are addressed. We further outline successful applications of diffusion models in various RL-related tasks while discussing the limitations of current approaches. Finally, we conclude the survey and offer insights into future research directions, focusing on enhancing model performance and applying diffusion models to broader tasks. We are actively maintaining a GitHub repository for papers and other related resources in applying diffusion models in RL: https://github.com/apexrl/Diff4RLSurvey

翻译：扩散模型已成为一类重要的生成模型，在样本质量和训练稳定性方面超越了先前的方法。近期研究表明，扩散模型在提升强化学习解决方案方面具有优势，包括作为轨迹规划器、表达性策略类别、数据合成器等。本综述旨在概述这一新兴领域的研究进展，并希望启发新的研究方向。首先，我们审视了当前强化学习算法面临的若干挑战。随后，基于扩散模型在强化学习中的作用，我们提出了现有方法的分类体系，并探讨了这些方法如何应对当前挑战。我们进一步总结了扩散模型在各种强化学习相关任务中的成功应用，同时讨论了当前方法的局限性。最后，我们对综述进行总结，并对未来研究方向提出见解，重点关注提升模型性能以及将扩散模型应用于更广泛的任务。我们积极维护一个GitHub仓库（https://github.com/apexrl/Diff4RLSurvey），用于收录扩散模型在强化学习中应用的论文及其他相关资源。