Deep reinforcement learning (DRL) frameworks are increasingly used to solve high-dimensional continuous-control tasks in robotics. However, due to the lack of sample efficiency, applying DRL for online learning is still practically infeasible in the robotics domain. One reason is that DRL agents do not leverage the solution of previous tasks for new tasks. Recent work on multi-tasking DRL agents based on successor features has proven to be quite promising in increasing sample efficiency. In this work, we present a new approach that unifies two prior multi-task RL frameworks, SF-GPI and value composition, for the continuous control domain. We exploit compositional properties of successor features to compose a policy distribution from a set of primitives without training any new policy. Lastly, to demonstrate the multi-tasking mechanism, we present a new benchmark for multi-task continuous control environment based on Raisim. This also facilitates large-scale parallelization to accelerate the experiments. Our experimental results in the Pointmass environment show that our multi-task agent has single task performance on par with soft actor critic (SAC) and the agent can successfully transfer to new unseen tasks where SAC fails. We provide our code as open-source at https://github.com/robot-perception-group/concurrent_composition for the benefit of the community.

翻译：深度强化学习（DRL）框架越来越多地被用于解决机器人学中的高维连续控制任务。然而，由于样本效率低下，将DRL应用于在线学习在机器人领域仍然实际上不可行。原因之一是DRL智能体未能利用先前任务的解决方案来应对新任务。基于后继特征的多任务DRL智能体近期研究在提升样本效率方面展现出巨大潜力。本文提出了一种新方法，统一了两种先前的多任务强化学习框架——SF-GPI与价值组合——在连续控制领域的应用。我们利用后继特征的组合性质，无需训练任何新策略即可从一组基元策略中组合出策略分布。最后，为展示多任务机制，我们基于Raisim提出了一个新的多任务连续控制环境基准。该环境还支持大规模并行化以加速实验。我们在Pointmass环境中的实验结果表明：我们的多任务智能体在单任务性能上与软演员-评论家（SAC）算法相当，且能成功迁移至SAC无法解决的新颖未见任务。为惠及社区，我们将代码开源在https://github.com/robot-perception-group/concurrent_composition。