Lifelong learning aims to create AI systems that continuously and incrementally learn during a lifetime, similar to biological learning. Attempts so far have met problems, including catastrophic forgetting, interference among tasks, and the inability to exploit previous knowledge. While considerable research has focused on learning multiple supervised classification tasks that involve changes in the input distribution, lifelong reinforcement learning (LRL) must deal with variations in the state and transition distributions, and in the reward functions. Modulating masks with a fixed backbone network, recently developed for classification, are particularly suitable to deal with such a large spectrum of task variations. In this paper, we adapted modulating masks to work with deep LRL, specifically PPO and IMPALA agents. The comparison with LRL baselines in both discrete and continuous RL tasks shows superior performance. We further investigated the use of a linear combination of previously learned masks to exploit previous knowledge when learning new tasks: not only is learning faster, the algorithm solves tasks that we could not otherwise solve from scratch due to extremely sparse rewards. The results suggest that RL with modulating masks is a promising approach to lifelong learning, to the composition of knowledge to learn increasingly complex tasks, and to knowledge reuse for efficient and faster learning.

翻译：终身学习旨在创建类似生物学习过程的AI系统，使其能够在整个生命周期中持续增量地学习。迄今为止的相关尝试仍面临诸多难题，包括灾难性遗忘、任务间干扰以及无法利用先前知识等问题。虽然已有大量研究聚焦于输入分布变化下的多监督分类任务学习，但终身强化学习还必须应对状态分布、转移分布以及奖励函数的变化。最新针对分类任务开发的固定骨干网络调制掩码技术，特别适合处理如此广泛的任务变化。本文将调制掩码适配到深度终身强化学习中，具体应用于PPO和IMPALA智能体。与终身强化学习基线方法在离散和连续强化学习任务中的对比表明，该方法具有更优性能。我们进一步研究了通过线性组合先前学习的掩码来利用历史知识的方法：不仅学习速度更快，该算法还能解决因极度稀疏奖励而无法从零开始学习的任务。研究结果表明，基于调制掩码的强化学习是终身学习的有效途径，可用于知识组合以学习日益复杂的任务，并通过知识复现实现高效快速的学习。