In recent years, deep reinforcement learning (RL) has shown its effectiveness in solving complex continuous control tasks like locomotion and dexterous manipulation. However, this comes at the cost of an enormous amount of experience required for training, exacerbated by the sensitivity of learning efficiency and the policy performance to hyperparameter selection, which often requires numerous trials of time-consuming experiments. This work introduces a Population-Based Reinforcement Learning (PBRL) approach that exploits a GPU-accelerated physics simulator to enhance the exploration capabilities of RL by concurrently training multiple policies in parallel. The PBRL framework is applied to three state-of-the-art RL algorithms - PPO, SAC, and DDPG - dynamically adjusting hyperparameters based on the performance of learning agents. The experiments are performed on four challenging tasks in Isaac Gym - Anymal Terrain, Shadow Hand, Humanoid, Franka Nut Pick - by analyzing the effect of population size and mutation mechanisms for hyperparameters. The results demonstrate that PBRL agents outperform non-evolutionary baseline agents across tasks essential for humanoid robots, such as bipedal locomotion, manipulation, and grasping in unstructured environments. The trained agents are finally deployed in the real world for the Franka Nut Pick manipulation task. To our knowledge, this is the first sim-to-real attempt for successfully deploying PBRL agents on real hardware. Code and videos of the learned policies are available on our project website (https://sites.google.com/view/pbrl).

翻译：近年来，深度强化学习在解决复杂连续控制任务（如运动控制和灵巧操作）方面展现出显著成效。然而，这种成功是以训练所需的海量经验为代价的，且学习效率与策略性能对超参数选择的敏感性进一步加剧了这一问题，通常需要进行大量耗时的实验尝试。本研究提出一种群体强化学习方法，通过利用GPU加速的物理仿真器并行训练多个策略，从而增强强化学习的探索能力。该PBRL框架应用于三种先进强化学习算法——PPO、SAC和DDPG，能够根据智能体的学习表现动态调整超参数。通过在Isaac Gym中的四个挑战性任务（Anymal地形行走、Shadow Hand灵巧操作、Humanoid双足运动、Franka坚果抓取）上分析群体规模与超参数突变机制的影响进行实验验证。结果表明，在人形机器人关键任务（如非结构化环境中的双足运动、操作与抓取）中，PBRL智能体均优于非进化的基线智能体。最终，训练完成的智能体在真实世界中成功部署于Franka坚果抓取操作任务。据我们所知，这是首次成功将PBRL智能体部署至真实硬件的仿真到现实尝试。学习策略的代码与演示视频已在项目网站（https://sites.google.com/view/pbrl）公开。