Diffusion models have demonstrated their powerful generative capability in many tasks, with great potential to serve as a paradigm for offline reinforcement learning. However, the quality of the diffusion model is limited by the insufficient diversity of training data, which hinders the performance of planning and the generalizability to new tasks. This paper introduces AdaptDiffuser, an evolutionary planning method with diffusion that can self-evolve to improve the diffusion model hence a better planner, not only for seen tasks but can also adapt to unseen tasks. AdaptDiffuser enables the generation of rich synthetic expert data for goal-conditioned tasks using guidance from reward gradients. It then selects high-quality data via a discriminator to finetune the diffusion model, which improves the generalization ability to unseen tasks. Empirical experiments on two benchmark environments and two carefully designed unseen tasks in KUKA industrial robot arm and Maze2D environments demonstrate the effectiveness of AdaptDiffuser. For example, AdaptDiffuser not only outperforms the previous art Diffuser by 20.8% on Maze2D and 7.5% on MuJoCo locomotion, but also adapts better to new tasks, e.g., KUKA pick-and-place, by 27.9% without requiring additional expert data.

翻译：扩散模型在许多任务中展现出强大的生成能力，具有成为离线强化学习范式的巨大潜力。然而，扩散模型的质量受限于训练数据多样性不足，这阻碍了规划性能及对新任务的泛化能力。本文提出AdaptDiffuser——一种基于扩散的自演化规划方法，该方法能够自我进化以改进扩散模型，从而成为更优的规划器，不仅适用于已见任务，还能适应未见任务。AdaptDiffuser利用奖励梯度的引导，为目标条件任务生成丰富的合成专家数据，随后通过判别器筛选高质量数据以微调扩散模型，从而提升对未见任务的泛化能力。在KUKA工业机械臂和Maze2D环境中的两个基准环境及两个精心设计的未见任务上的实证实验证明了AdaptDiffuser的有效性。例如，AdaptDiffuser在Maze2D和MuJoCo运动任务上不仅分别以20.8%和7.5%的优势超越先前最佳方案Diffuser，还能更好地适应新任务（如KUKA拾放任务），在无需额外专家数据的情况下性能提升27.9%。