Many environments contain numerous available niches of variable value, each associated with a different local optimum in the space of behaviors (policy space). In such situations it is often difficult to design a learning process capable of evading distraction by poor local optima long enough to stumble upon the best available niche. In this work we propose a generic reinforcement learning (RL) algorithm that performs better than baseline deep Q-learning algorithms in such environments with multiple variably-valued niches. The algorithm we propose consists of two parts: an agent architecture and a learning rule. The agent architecture contains multiple sub-policies. The learning rule is inspired by fitness sharing in evolutionary computation and applied in reinforcement learning using Value-Decomposition-Networks in a novel manner for a single-agent's internal population. It can concretely be understood as adding an extra loss term where one policy's experience is also used to update all the other policies in a manner that decreases their value estimates for the visited states. In particular, when one sub-policy visits a particular state frequently this decreases the value predicted for other sub-policies for going to that state. Further, we introduce an artificial chemistry inspired platform where it is easy to create tasks with multiple rewarding strategies utilizing different resources (i.e. multiple niches). We show that agents trained this way can escape poor-but-attractive local optima to instead converge to harder-to-discover higher value strategies in both the artificial chemistry environments and in simpler illustrative environments.

翻译：许多环境包含大量价值各异的小生境，每个小生境与行为空间（策略空间）中不同的局部最优解相关联。在这种情况下，设计一个能够避开劣质局部最优解的干扰，最终偶然发现最佳小生境的学习过程往往非常困难。本文提出了一种通用的强化学习算法，在具有多个不同价值小生境的环境中，其性能优于基准深度Q学习算法。该算法由两部分组成：智能体架构和学习规则。智能体架构包含多个子策略。学习规则受进化计算中适应度共享思想的启发，并通过价值分解网络以新颖的方式应用于单个智能体内部群体的强化学习中。具体可以理解为增加一个额外的损失项，其中一项策略的经验也被用于更新所有其他策略，从而降低它们对已访问状态的价值估计。特别是，当某个子策略频繁访问特定状态时，这会降低其他子策略访问该状态的预测价值。此外，我们引入了一个受人工化学启发的平台，在该平台上可以轻松创建具有多种利用不同资源（即多个小生境）的奖励策略的任务。我们表明，通过这种方式训练的智能体能够摆脱劣质但具吸引力的局部最优解，转而收敛到更难发现的更高价值策略，无论是在人工化学环境还是在更简单的说明性环境中均如此。