Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for training neural policies to solve complex control tasks. However, these policies tend to be overfit to the exact specifications of the task and environment they were trained on, and thus do not perform well when conditions deviate slightly or when composed hierarchically to solve even more complex tasks. Recent work has shown that training a mixture of policies, as opposed to a single one, that are driven to explore different regions of the state-action space can address this shortcoming by generating a diverse set of behaviors, referred to as skills, that can be collectively used to great effect in adaptation tasks or for hierarchical planning. This is typically realized by including a diversity term - often derived from information theory - in the objective function optimized by RL. However these approaches often require careful hyperparameter tuning to be effective. In this work, we demonstrate that less widely-used neuroevolution methods, specifically Quality Diversity (QD), are a competitive alternative to information-theory-augmented RL for skill discovery. Through an extensive empirical evaluation comparing eight state-of-the-art algorithms (four flagship algorithms from each line of work) on the basis of (i) metrics directly evaluating the skills' diversity, (ii) the skills' performance on adaptation tasks, and (iii) the skills' performance when used as primitives for hierarchical planning; QD methods are found to provide equal, and sometimes improved, performance whilst being less sensitive to hyperparameters and more scalable. As no single method is found to provide near-optimal performance across all environments, there is a rich scope for further research which we support by proposing future directions and providing optimized open-source implementations.

翻译：深度强化学习（RL）已成为训练神经策略以解决复杂控制任务的有效范式。然而，这些策略往往过度适配于其训练环境和任务的具体规范，因此在条件发生微小变化或需通过层次化组合解决更复杂任务时表现不佳。近期研究表明，训练一组混合策略（而非单一策略）并驱动其探索状态-动作空间的不同区域，可通过生成多样化行为（称为技能）来弥补这一缺陷。这些技能在自适应任务或层次化规划中可被协同利用以产生显著效果。这一目标通常通过在RL优化的目标函数中引入多样性项（通常源自信息论）来实现，但此类方法往往需要精细的超参数调优才能有效。在本工作中，我们证明：较少被使用的神经进化方法——特别是质量多样性（QD）——是信息论增强型RL在技能发现中的竞争性替代方案。通过基于以下三方面对八种最先进算法（来自两个研究方向各四种代表性算法）进行广泛实证评估：（i）直接评估技能多样性的指标，（ii）技能在自适应任务中的表现，（iii）技能作为层次化规划基元时的表现；我们发现QD方法能提供相等甚至更优的性能，同时对超参数敏感性更低且更具可扩展性。由于没有任何单一方法能在所有环境中提供接近最优的性能，本研究为后续探索留下了丰富空间——我们通过提出未来研究方向并提供优化的开源实现来支持这一探索。