Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for training neural policies to solve complex control tasks. However, these policies tend to be overfit to the exact specifications of the task and environment they were trained on, and thus do not perform well when conditions deviate slightly or when composed hierarchically to solve even more complex tasks. Recent work has shown that training a mixture of policies, as opposed to a single one, that are driven to explore different regions of the state-action space can address this shortcoming by generating a diverse set of behaviors, referred to as skills, that can be collectively used to great effect in adaptation tasks or for hierarchical planning. This is typically realized by including a diversity term - often derived from information theory - in the objective function optimized by RL. However these approaches often require careful hyperparameter tuning to be effective. In this work, we demonstrate that less widely-used neuroevolution methods, specifically Quality Diversity (QD), are a competitive alternative to information-theory-augmented RL for skill discovery. Through an extensive empirical evaluation comparing eight state-of-the-art algorithms (four flagship algorithms from each line of work) on the basis of (i) metrics directly evaluating the skills' diversity, (ii) the skills' performance on adaptation tasks, and (iii) the skills' performance when used as primitives for hierarchical planning; QD methods are found to provide equal, and sometimes improved, performance whilst being less sensitive to hyperparameters and more scalable. As no single method is found to provide near-optimal performance across all environments, there is a rich scope for further research which we support by proposing future directions and providing optimized open-source implementations.

翻译：深度强化学习(RL)已成为训练神经策略以解决复杂控制任务的强大范式。然而，这些策略往往对训练所用任务和环境的精确规范存在过拟合，因此在条件发生细微偏离或以层次化方式组合以解决更复杂任务时表现不佳。近期研究表明，训练策略混合体（而非单一策略），并驱动其探索状态-动作空间的不同区域，可通过生成多样化的行为集（称为技能）来弥补这一缺陷，这些行为集在自适应任务或层次化规划中能发挥显著效果。这一目标通常通过RL优化的目标函数中引入多样性项（通常源于信息论）来实现，但这些方法往往需要细致的超参数调优才能有效。在本工作中，我们证明较少被采用的神经进化方法——特别是质量多样性(QD)——是技能发现中信息论增强型RL的竞争性替代方案。通过开展涵盖八种最先进算法（每类工作中的四种旗舰算法）的大规模实证评估，基于(i)直接评估技能多样性的指标、(ii)技能在自适应任务中的表现，以及(iii)技能作为层次规划原语时的性能，我们发现QD方法能提供等同甚至更优的性能，同时对超参数的敏感性更低且可扩展性更强。由于未发现任何单一方法能在所有环境中提供接近最优的性能，该领域存在广泛的研究空间——我们通过提出未来方向并提供优化的开源实现来支持这一研究。