Quality Diversity (QD) has emerged as a powerful alternative optimization paradigm that aims at generating large and diverse collections of solutions, notably with its flagship algorithm MAP-ELITES (ME) which evolves solutions through mutations and crossovers. While very effective for some unstructured problems, early ME implementations relied exclusively on random search to evolve the population of solutions, rendering them notoriously sample-inefficient for high-dimensional problems, such as when evolving neural networks. Follow-up works considered exploiting gradient information to guide the search in order to address these shortcomings through techniques borrowed from either Black-Box Optimization (BBO) or Reinforcement Learning (RL). While mixing RL techniques with ME unlocked state-of-the-art performance for robotics control problems that require a good amount of exploration, it also plagued these ME variants with limitations common among RL algorithms that ME was free of, such as hyperparameter sensitivity, high stochasticity as well as training instability, including when the population size increases as some components are shared across the population in recent approaches. Furthermore, existing approaches mixing ME with RL tend to be tied to a specific RL algorithm, which effectively prevents their use on problems where the corresponding RL algorithm fails. To address these shortcomings, we introduce a flexible framework that allows the use of any RL algorithm and alleviates the aforementioned limitations by evolving populations of agents (whose definition include hyperparameters and all learnable parameters) instead of just policies. We demonstrate the benefits brought about by our framework through extensive numerical experiments on a number of robotics control problems, some of which with deceptive rewards, taken from the QD-RL literature.

翻译：质量多样性（QD）已成为一种强大的替代优化范式，旨在生成大规模多样化的解集合，其代表性算法MAP-ELITES（ME）通过变异和交叉进化解集合。尽管该算法对某些非结构化问题非常有效，但早期ME实现完全依赖随机搜索来进化解种群，导致其在处理高维问题（如进化神经网络）时样本效率极低。后续研究考虑利用梯度信息引导搜索，通过借鉴黑盒优化（BBO）或强化学习（RL）技术来弥补这些缺陷。虽然将RL技术与ME结合在需要大量探索的机器人控制问题上取得了最先进的性能，但也使这些ME变体陷入RL算法常见的局限性——这是ME原本所没有的，例如超参数敏感性、高随机性以及训练不稳定性，尤其是在近年来一些方法将部分组件跨种群共享导致种群规模增大时。此外，现有混合ME与RL的方法往往与特定RL算法绑定，这从根本上阻碍了其在对应RL算法失效的问题上的应用。为解决这些缺陷，我们提出一种灵活框架，允许使用任意RL算法，并通过进化智能体种群（其定义包含超参数和所有可学习参数）而非仅进化策略来缓解上述局限。通过在QD-RL文献中多个机器人控制问题（部分包含欺骗性奖励）上的大量数值实验，我们展示了该框架带来的优势。