Recently, evolutionary reinforcement learning has obtained much attention in various domains. Maintaining a population of actors, evolutionary reinforcement learning utilises the collected experiences to improve the behaviour policy through efficient exploration. However, the poor scalability of genetic operators limits the efficiency of optimising high-dimensional neural networks. To address this issue, this paper proposes a novel cooperative coevolutionary reinforcement learning (CoERL) algorithm. Inspired by cooperative coevolution, CoERL periodically and adaptively decomposes the policy optimisation problem into multiple subproblems and evolves a population of neural networks for each of the subproblems. Instead of using genetic operators, CoERL directly searches for partial gradients to update the policy. Updating policy with partial gradients maintains consistency between the behaviour spaces of parents and offspring across generations. The experiences collected by the population are then used to improve the entire policy, which enhances the sampling efficiency. Experiments on six benchmark locomotion tasks demonstrate that CoERL outperforms seven state-of-the-art algorithms and baselines. Ablation study verifies the unique contribution of CoERL's core ingredients.

翻译：近期，演化强化学习在多个领域受到广泛关注。通过维持智能体种群，演化强化学习利用收集的经验进行高效探索，从而改进行为策略。然而，遗传算子的可扩展性较差，限制了高维神经网络优化的效率。针对这一问题，本文提出一种新型协同进化强化学习（CoERL）算法。受协同进化思想启发，CoERL周期性地自适应地将策略优化问题分解为多个子问题，并为每个子问题进化一个神经网络种群。CoERL不使用遗传算子，而是直接搜索部分梯度以更新策略。通过部分梯度更新策略，保持了跨代父代与子代行为空间的一致性。种群收集的经验随后用于改进整体策略，从而提升采样效率。在六个基准运动控制任务上的实验表明，CoERL优于七种最先进算法及基线方法。消融研究验证了CoERL核心组件的独特贡献。