In multi-task reinforcement learning (RL) under Markov decision processes (MDPs), the presence of shared latent structures among multiple MDPs has been shown to yield significant benefits to the sample efficiency compared to single-task RL. In this paper, we investigate whether such a benefit can extend to more general sequential decision making problems, such as partially observable MDPs (POMDPs) and more general predictive state representations (PSRs). The main challenge here is that the large and complex model space makes it hard to identify what types of common latent structure of multi-task PSRs can reduce the model complexity and improve sample efficiency. To this end, we posit a joint model class for tasks and use the notion of $\eta$-bracketing number to quantify its complexity; this number also serves as a general metric to capture the similarity of tasks and thus determines the benefit of multi-task over single-task RL. We first study upstream multi-task learning over PSRs, in which all tasks share the same observation and action spaces. We propose a provably efficient algorithm UMT-PSR for finding near-optimal policies for all PSRs, and demonstrate that the advantage of multi-task learning manifests if the joint model class of PSRs has a smaller $\eta$-bracketing number compared to that of individual single-task learning. We also provide several example multi-task PSRs with small $\eta$-bracketing numbers, which reap the benefits of multi-task learning. We further investigate downstream learning, in which the agent needs to learn a new target task that shares some commonalities with the upstream tasks via a similarity constraint. By exploiting the learned PSRs from the upstream, we develop a sample-efficient algorithm that provably finds a near-optimal policy.

翻译：在马尔可夫决策过程（MDP）的多任务强化学习（RL）中，多个MDP间共享潜在结构已被证明相比单任务RL能显著提升样本效率。本文研究这种优势是否能推广至更一般的序贯决策问题，如部分可观测MDP（POMDP）和更具一般性的预测状态表示（PSR）。关键挑战在于：庞大复杂的模型空间使得识别多任务PSR中何种共享潜在结构能降低模型复杂度并提升样本效率变得困难。为此，我们提出任务的联合模型类，并利用η-包围数概念量化其复杂度；该指标同时作为任务相似度的通用度量，决定了多任务RL相比单任务RL的优势。我们首先研究PSR上的上游多任务学习，其中所有任务共享相同的观测空间和动作空间。我们提出可证高效的UMT-PSR算法来为所有PSR寻找近优策略，并证明：当PSR联合模型类的η-包围数小于单任务学习的对应值时，多任务学习优势得以体现。我们还给出多个具有小η-包围数的多任务PSR示例，从而获得多任务学习的收益。进一步研究下游学习：智能体需通过相似性约束，学习与上游任务存在共性的新目标任务。通过利用上游学得的PSR，我们开发出可证能找到近优策略的样本高效算法。