In this paper, we address the following problem: Given an offline demonstration dataset from an imperfect expert, what is the best way to leverage it to bootstrap online learning performance in MDPs. We first propose an Informed Posterior Sampling-based RL (iPSRL) algorithm that uses the offline dataset, and information about the expert's behavioral policy used to generate the offline dataset. Its cumulative Bayesian regret goes down to zero exponentially fast in N, the offline dataset size if the expert is competent enough. Since this algorithm is computationally impractical, we then propose the iRLSVI algorithm that can be seen as a combination of the RLSVI algorithm for online RL, and imitation learning. Our empirical results show that the proposed iRLSVI algorithm is able to achieve significant reduction in regret as compared to two baselines: no offline data, and offline dataset but used without information about the generative policy. Our algorithm bridges online RL and imitation learning for the first time.

翻译：本文研究以下问题：给定来自非完美专家的离线演示数据集，如何最好地利用它来启动马尔可夫决策过程中的在线学习性能。我们首先提出一种基于信息后验采样的强化学习（iPSRL）算法，该算法利用离线数据集以及用于生成该数据集的专家行为策略信息。如果专家足够胜任，其累积贝叶斯遗憾将以指数速度下降至零，下降速度取决于离线数据集大小N。由于该算法计算上不可行，我们随后提出iRLSVI算法，可视为在线强化学习RLSVI算法与模仿学习的结合。我们的实验结果表明，与两种基线方法（不使用离线数据、使用离线数据集但未利用生成策略信息）相比，所提出的iRLSVI算法能够显著降低遗憾。我们的算法首次在在线强化学习与模仿学习之间建立了桥梁。