Quality Diversity (QD) has emerged as a powerful alternative optimization paradigm that aims at generating large and diverse collections of solutions, notably with its flagship algorithm MAP-ELITES (ME) which evolves solutions through mutations and crossovers. While very effective for some unstructured problems, early ME implementations relied exclusively on random search to evolve the population of solutions, rendering them notoriously sample-inefficient for high-dimensional problems, such as when evolving neural networks. Follow-up works considered exploiting gradient information to guide the search in order to address these shortcomings through techniques borrowed from either Black-Box Optimization (BBO) or Reinforcement Learning (RL). While mixing RL techniques with ME unlocked state-of-the-art performance for robotics control problems that require a good amount of exploration, it also plagued these ME variants with limitations common among RL algorithms that ME was free of, such as hyperparameter sensitivity, high stochasticity as well as training instability, including when the population size increases as some components are shared across the population in recent approaches. Furthermore, existing approaches mixing ME with RL tend to be tied to a specific RL algorithm, which effectively prevents their use on problems where the corresponding RL algorithm fails. To address these shortcomings, we introduce a flexible framework that allows the use of any RL algorithm and alleviates the aforementioned limitations by evolving populations of agents (whose definition include hyperparameters and all learnable parameters) instead of just policies. We demonstrate the benefits brought about by our framework through extensive numerical experiments on a number of robotics control problems, some of which with deceptive rewards, taken from the QD-RL literature.

翻译：质量多样性（QD）已成为一种强大的替代优化范式，旨在生成大规模且多样化的解决方案集合，其代表性算法MAP-ELITES（ME）通过变异与交叉操作进化解决方案。虽然该方法对某些非结构化问题效果显著，但早期ME实现完全依赖随机搜索来进化解决方案群体，导致其在处理高维问题（如进化神经网络）时样本效率极低。后续研究尝试利用梯度信息引导搜索，通过引入黑盒优化（BBO）或强化学习（RL）技术来克服这些缺陷。然而，将RL技术与ME结合虽在需要充分探索的机器人控制问题上展现出最优性能，却也导致这些ME变体受到RL算法常见局限性的困扰——而原始ME本无需面对此类问题，包括超参数敏感性、高随机性及训练不稳定性，尤其当群体规模增大时，由于近期方法中群体共享部分组件，这些问题更为突出。此外，现有融合ME与RL的方法往往绑定特定RL算法，这实际上阻碍了将其应用于对应RL算法失效的问题场景。为解决上述缺陷，我们提出一个灵活框架，允许使用任意RL算法，并通过进化包含超参数与所有可学习参数的智能体群体（而非仅策略）来缓解前述局限性。通过QD-RL文献中多个机器人控制问题（含部分存在欺骗性奖励的任务）的广泛数值实验，我们验证了该框架带来的显著优势。