Enhancing the diversity of policies is beneficial for robustness, exploration, and transfer in reinforcement learning (RL). In this paper, we aim to seek diverse policies in an under-explored setting, namely RL tasks with structured action spaces with the two properties of composability and local dependencies. The complex action structure, non-uniform reward landscape, and subtle hyperparameter tuning due to the properties of structured actions prevent existing approaches from scaling well. We propose a simple and effective RL method, Diverse Policy Optimization (DPO), to model the policies in structured action space as the energy-based models (EBM) by following the probabilistic RL framework. A recently proposed novel and powerful generative model, GFlowNet, is introduced as the efficient, diverse EBM-based policy sampler. DPO follows a joint optimization framework: the outer layer uses the diverse policies sampled by the GFlowNet to update the EBM-based policies, which supports the GFlowNet training in the inner layer. Experiments on ATSC and Battle benchmarks demonstrate that DPO can efficiently discover surprisingly diverse policies in challenging scenarios and substantially outperform existing state-of-the-art methods.

翻译：提升策略多样性有益于强化学习中的鲁棒性、探索性和迁移能力。本文旨在探索一个尚未充分研究的领域——具有可组合性和局部依赖性的结构化动作空间的强化学习任务。由于结构化动作的特性，复杂的动作结构、非均匀的奖励景观以及微妙的超参数调优导致现有方法难以有效扩展。我们提出一种简单有效的强化学习方法——多样策略优化（DPO），通过遵循概率强化学习框架，将结构化动作空间中的策略建模为基于能量的模型（EBM）。最新提出的强大生成模型GFlowNet被引入作为基于EBM的高效多样策略采样器。DPO遵循联合优化框架：外层利用GFlowNet采样的多样策略更新基于EBM的策略，支持内层GFlowNet的训练。在ATSC和Battle基准测试上的实验表明，DPO能够在具有挑战性的场景中高效发现令人惊讶的多样策略，并显著优于现有最先进方法。