Hyperparameter optimization plays a key role in the machine learning domain. Its significance is especially pronounced in reinforcement learning (RL), where agents continuously interact with and adapt to their environments, requiring dynamic adjustments in their learning trajectories. To cater to this dynamicity, the Population-Based Training (PBT) was introduced, leveraging the collective intelligence of a population of agents learning simultaneously. However, PBT tends to favor high-performing agents, potentially neglecting the explorative potential of agents on the brink of significant advancements. To mitigate the limitations of PBT, we present the Generalized Population-Based Training (GPBT), a refined framework designed for enhanced granularity and flexibility in hyperparameter adaptation. Complementing GPBT, we further introduce Pairwise Learning (PL). Instead of merely focusing on elite agents, PL employs a comprehensive pairwise strategy to identify performance differentials and provide holistic guidance to underperforming agents. By integrating the capabilities of GPBT and PL, our approach significantly improves upon traditional PBT in terms of adaptability and computational efficiency. Rigorous empirical evaluations across a range of RL benchmarks confirm that our approach consistently outperforms not only the conventional PBT but also its Bayesian-optimized variant.

翻译：超参数优化在机器学习领域扮演着关键角色，其在强化学习（RL）中的重要性尤为突出——智能体需持续与环境交互并适应环境变化，相应地要求其学习轨迹具备动态调整能力。为满足这种动态性需求，种群训练（PBT）应运而生，该技术通过利用同时学习的多个智能体的集体智慧实现优化。然而，PBT倾向于关注高性能智能体，可能忽略那些处于突破边缘的智能体的探索潜力。为弥补PBT的局限性，我们提出广义种群训练（GPBT），这是一个旨在实现超参数自适应中更高粒度和灵活性的改进框架。作为GPBT的补充，我们进一步引入成对学习（PL）。不同于仅关注精英智能体，PL采用全面的成对策略来识别性能差异，并为表现欠佳的智能体提供全局性指导。通过整合GPBT与PL的能力，我们的方法在适应性和计算效率方面显著优于传统PBT。在多个RL基准测试上进行的严格实证评估证实，我们的方法不仅持续超越传统PBT，而且优于其贝叶斯优化变体。