Compositional reinforcement learning is a promising approach for training policies to perform complex long-horizon tasks. Typically, a high-level task is decomposed into a sequence of subtasks and a separate policy is trained to perform each subtask. In this paper, we focus on the problem of training subtask policies in a way that they can be used to perform any task; here, a task is given by a sequence of subtasks. We aim to maximize the worst-case performance over all tasks as opposed to the average-case performance. We formulate the problem as a two agent zero-sum game in which the adversary picks the sequence of subtasks. We propose two RL algorithms to solve this game: one is an adaptation of existing multi-agent RL algorithms to our setting and the other is an asynchronous version which enables parallel training of subtask policies. We evaluate our approach on two multi-task environments with continuous states and actions and demonstrate that our algorithms outperform state-of-the-art baselines.

翻译：组合强化学习是一种有前景的方法，可用于训练策略以执行复杂的长期任务。通常，高层任务被分解为一系列子任务，并为每个子任务训练独立的策略。本文聚焦于训练子任务策略的方式，使其能够执行任意任务；此处的任务由一系列子任务序列构成。我们旨在最大化所有任务中的最差性能，而非平均性能。我们将该问题建模为双智能体零和博弈，其中对抗方选取子任务序列。我们提出两种强化学习算法求解该博弈：一种是对现有多智能体强化学习算法的适应性改进，另一种是异步版本，支持子任务策略的并行训练。我们在两个具有连续状态和动作的多任务环境中评估了该方法，结果表明我们的算法优于当前最先进的基线模型。