Diffusion models have demonstrated their powerful generative capability in many tasks, with great potential to serve as a paradigm for offline reinforcement learning. However, the quality of the diffusion model is limited by the insufficient diversity of training data, which hinders the performance of planning and the generalizability to new tasks. This paper introduces AdaptDiffuser, an evolutionary planning method with diffusion that can self-evolve to improve the diffusion model hence a better planner, not only for seen tasks but can also adapt to unseen tasks. AdaptDiffuser enables the generation of rich synthetic expert data for goal-conditioned tasks using guidance from reward gradients. It then selects high-quality data via a discriminator to finetune the diffusion model, which improves the generalization ability to unseen tasks. Empirical experiments on two benchmark environments and two carefully designed unseen tasks in KUKA industrial robot arm and Maze2D environments demonstrate the effectiveness of AdaptDiffuser. For example, AdaptDiffuser not only outperforms the previous art Diffuser by 20.8% on Maze2D and 7.5% on MuJoCo locomotion, but also adapts better to new tasks, e.g., KUKA pick-and-place, by 27.9% without requiring additional expert data. More visualization results and demo videos could be found on our project page.

翻译：扩散模型已在众多任务中展现出强大的生成能力，且具备作为离线强化学习范式的巨大潜力。然而，训练数据多样性不足限制了扩散模型的质量，进而阻碍了规划性能及对新任务的泛化能力。本文提出AdaptDiffuser，一种基于扩散模型的自演化规划方法，该方法可通过自我演化提升扩散模型质量，从而获得更优规划器，不仅适用于已见任务，还能自适应未见任务。AdaptDiffuser通过奖励梯度引导，为目标条件任务生成丰富的合成专家数据，并利用判别器筛选高质量数据以微调扩散模型，从而提升对未见任务的泛化能力。在KUKA工业机械臂和Maze2D环境的两个基准任务及两个精心设计的未见任务上的实验证明了AdaptDiffuser的有效性。例如，AdaptDiffuser在Maze2D和MuJoCo运动任务上分别超越先前最优方法Diffuser 20.8%和7.5%，且无需额外专家数据即可更好地适应新任务（如KUKA抓取放置任务），性能提升达27.9%。更多可视化结果及演示视频可参见项目主页。