Diffusion models excel at modeling complex and multimodal trajectory distributions for decision-making and control. Reward-gradient guided denoising has been recently proposed to generate trajectories that maximize both a differentiable reward function and the likelihood under the data distribution captured by a diffusion model. Reward-gradient guided denoising requires a differentiable reward function fitted to both clean and noised samples, limiting its applicability as a general trajectory optimizer. In this paper, we propose DiffusionES, a method that combines gradient-free optimization with trajectory denoising to optimize black-box non-differentiable objectives while staying in the data manifold. Diffusion-ES samples trajectories during evolutionary search from a diffusion model and scores them using a black-box reward function. It mutates high-scoring trajectories using a truncated diffusion process that applies a small number of noising and denoising steps, allowing for much more efficient exploration of the solution space. We show that DiffusionES achieves state-of-the-art performance on nuPlan, an established closed-loop planning benchmark for autonomous driving. Diffusion-ES outperforms existing sampling-based planners, reactive deterministic or diffusion-based policies, and reward-gradient guidance. Additionally, we show that unlike prior guidance methods, our method can optimize non-differentiable language-shaped reward functions generated by few-shot LLM prompting. When guided by a human teacher that issues instructions to follow, our method can generate novel, highly complex behaviors, such as aggressive lane weaving, which are not present in the training data. This allows us to solve the hardest nuPlan scenarios which are beyond the capabilities of existing trajectory optimization methods and driving policies.

翻译：扩散模型在建模复杂多模态轨迹分布以用于决策与控制方面表现出色。最近提出的奖励梯度引导去噪方法，可生成能同时最大化可微奖励函数与扩散模型捕捉的数据分布似然性的轨迹。但该方法需要针对干净样本与含噪样本拟合可微奖励函数，这限制了其作为通用轨迹优化器的适用性。本文提出扩散进化策略（Diffusion-ES），通过融合免梯度优化与轨迹去噪技术，在保持数据流形的同时优化黑箱非可微目标函数。Diffusion-ES从扩散模型中采样轨迹进行进化搜索，并使用黑箱奖励函数评估轨迹质量；通过截断扩散过程对高分轨迹施加少量加噪与去噪步长实现变异，从而更高效地探索解空间。实验表明，Diffusion-ES在自动驾驶闭环规划基准nuPlan上取得最优性能，超越现有基于采样的规划器、反应式确定性/扩散策略以及奖励梯度引导方法。此外，不同于先验引导方法，本方法可优化由少样本大语言模型提示生成的非可微语言形奖励函数。在人类教师发出指令引导时，本方法能生成训练数据中不存在的新颖复杂行为（如激进的车道穿插），从而解决远超现有轨迹优化方法与驾驶策略能力的nuPlan最难场景。