Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for training neural policies to solve complex control tasks. However, these policies tend to be overfit to the exact specifications of the task and environment they were trained on, and thus do not perform well when conditions deviate slightly or when composed hierarchically to solve even more complex tasks. Recent work has shown that training a mixture of policies, as opposed to a single one, that are driven to explore different regions of the state-action space can address this shortcoming by generating a diverse set of behaviors, referred to as skills, that can be collectively used to great effect in adaptation tasks or for hierarchical planning. This is typically realized by including a diversity term - often derived from information theory - in the objective function optimized by RL. However these approaches often require careful hyperparameter tuning to be effective. In this work, we demonstrate that less widely-used neuroevolution methods, specifically Quality Diversity (QD), are a competitive alternative to information-theory-augmented RL for skill discovery. Through an extensive empirical evaluation comparing eight state-of-the-art algorithms (four flagship algorithms from each line of work) on the basis of (i) metrics directly evaluating the skills' diversity, (ii) the skills' performance on adaptation tasks, and (iii) the skills' performance when used as primitives for hierarchical planning; QD methods are found to provide equal, and sometimes improved, performance whilst being less sensitive to hyperparameters and more scalable. As no single method is found to provide near-optimal performance across all environments, there is a rich scope for further research which we support by proposing future directions and providing optimized open-source implementations.

翻译：深度强化学习已成为训练神经策略以解决复杂控制任务的有力范式。然而，这些策略往往过度拟合于其训练任务和环境的精确规范，因此当条件轻微偏离或为处理更复杂任务而进行层次化组合时，其表现不佳。近期研究表明，训练一组混合策略（而非单一策略）并驱动其探索状态-动作空间的不同区域，可通过生成多样化的行为（称为技能）来弥补这一不足，这些技能在自适应任务或层次规划中可被联合利用并产生显著效果。这通常通过在强化学习优化的目标函数中包含多样性项（常源自信息论）来实现。然而，这些方法通常需要精细的超参数调优才能有效。在本工作中，我们证明较少使用的神经进化方法，特别是质量多样性（QD），是信息论增强型强化学习在技能发现中的竞争性替代方案。通过广泛实证评估，基于（i）直接衡量技能多样性的指标、（ii）技能在自适应任务中的表现，以及（iii）技能作为层次规划基元时的表现，对八种最新算法（每个研究方向各四种旗舰算法）进行比较，发现QD方法能提供相当甚至更优的性能，同时对超参数敏感性更低且更具可扩展性。由于未发现任何单一方法能在所有环境中提供接近最优的性能，因此该领域存在丰富的研究空间，我们通过提出未来方向并提供优化的开源实现来支持进一步研究。