Recent progress in Quality Diversity Reinforcement Learning (QD-RL) has enabled learning a collection of behaviorally diverse, high performing policies. However, these methods typically involve storing thousands of policies, which results in high space-complexity and poor scaling to additional behaviors. Condensing the archive into a single model while retaining the performance and coverage of the original collection of policies has proved challenging. In this work, we propose using diffusion models to distill the archive into a single generative model over policy parameters. We show that our method achieves a compression ratio of 13x while recovering 98% of the original rewards and 89% of the original coverage. Further, the conditioning mechanism of diffusion models allows for flexibly selecting and sequencing behaviors, including using language. Project website: https://sites.google.com/view/policydiffusion/home

翻译：近期质量多样性强化学习（QD-RL）的进展使得学习一组行为多样化且高性能的策略成为可能。然而，这些方法通常需要存储数千个策略，导致空间复杂度高，且难以扩展到额外行为。在保持原始策略集合的性能和覆盖范围的同时，将存档浓缩为单个模型一直具有挑战性。在本工作中，我们提出使用扩散模型将存档蒸馏为策略参数上的单一生成模型。我们展示了该方法在恢复98%原始奖励和89%原始覆盖范围的同时，实现了13倍的压缩比。此外，扩散模型的条件机制允许灵活选择和排序行为，包括使用语言。项目网站：https://sites.google.com/view/policydiffusion/home