Compositional reinforcement learning is a promising approach for training policies to perform complex long-horizon tasks. Typically, a high-level task is decomposed into a sequence of subtasks and a separate policy is trained to perform each subtask. In this paper, we focus on the problem of training subtask policies in a way that they can be used to perform any task; here, a task is given by a sequence of subtasks. We aim to maximize the worst-case performance over all tasks as opposed to the average-case performance. We formulate the problem as a two agent zero-sum game in which the adversary picks the sequence of subtasks. We propose two RL algorithms to solve this game: one is an adaptation of existing multi-agent RL algorithms to our setting and the other is an asynchronous version which enables parallel training of subtask policies. We evaluate our approach on two multi-task environments with continuous states and actions and demonstrate that our algorithms outperform state-of-the-art baselines.

翻译：组合强化学习是一种有前景的方法，用于训练策略以执行复杂的长时间跨度任务。通常，高层任务被分解为一系列子任务，并训练独立的策略来执行每个子任务。本文聚焦于训练子任务策略的问题，使其能够用于执行任意任务；这里，任务由一系列子任务序列给出。我们旨在最大化所有任务中的最差性能，而非平均性能。我们将该问题建模为一个双智能体零和博弈，其中对手选择子任务序列。我们提出了两种强化学习算法来解决这一博弈：一种是对现有多智能体强化学习算法在本场景下的适配，另一种是异步版本，能够实现子任务策略的并行训练。我们在两个具有连续状态和动作的多任务环境中评估了我们的方法，并证明我们的算法优于当前最先进的基线方法。