The exploration problem is one of the main challenges in deep reinforcement learning (RL). Recent promising works tried to handle the problem with population-based methods, which collect samples with diverse behaviors derived from a population of different exploratory policies. Adaptive policy selection has been adopted for behavior control. However, the behavior selection space is largely limited by the predefined policy population, which further limits behavior diversity. In this paper, we propose a general framework called Learnable Behavioral Control (LBC) to address the limitation, which a) enables a significantly enlarged behavior selection space via formulating a hybrid behavior mapping from all policies; b) constructs a unified learnable process for behavior selection. We introduce LBC into distributed off-policy actor-critic methods and achieve behavior control via optimizing the selection of the behavior mappings with bandit-based meta-controllers. Our agents have achieved 10077.52% mean human normalized score and surpassed 24 human world records within 1B training frames in the Arcade Learning Environment, which demonstrates our significant state-of-the-art (SOTA) performance without degrading the sample efficiency.

翻译：探索问题是深度强化学习（RL）中的主要挑战之一。近期有前景的研究尝试通过基于种群的方法处理该问题，这些方法从由不同探索策略构成的种群中收集具有多样行为的样本。自适应策略选择已被用于行为控制。然而，行为选择空间在很大程度上受限于预定义的策略种群，这进一步限制了行为的多样性。在本文中，我们提出一个名为可学习行为控制（LBC）的通用框架来解决这一限制，该框架：a) 通过从所有策略构建混合行为映射，显著扩大行为选择空间；b) 构建统一的可学习过程用于行为选择。我们将LBC引入分布式离策略actor-critic方法中，并通过基于赌博机的元控制器优化行为映射的选择，实现行为控制。我们的代理在Arcade学习环境中取得了10077.52%的平均人类标准化分数，并在10亿训练帧内超越了24项人类世界纪录，这展示了我们在不降低样本效率情况下的显著最新（SOTA）性能。