One of the key challenges of Reinforcement Learning (RL) is the ability of agents to generalise their learned policy to unseen settings. Moreover, training RL agents requires large numbers of interactions with the environment. Motivated by the recent success of Offline RL and Imitation Learning (IL), we conduct a study to investigate whether agents can leverage offline data in the form of trajectories to improve the sample-efficiency in procedurally generated environments. We consider two settings of using IL from offline data for RL: (1) pre-training a policy before online RL training and (2) concurrently training a policy with online RL and IL from offline data. We analyse the impact of the quality (optimality of trajectories) and diversity (number of trajectories and covered level) of available offline trajectories on the effectiveness of both approaches. Across four well-known sparse reward tasks in the MiniGrid environment, we find that using IL for pre-training and concurrently during online RL training both consistently improve the sample-efficiency while converging to optimal policies. Furthermore, we show that pre-training a policy from as few as two trajectories can make the difference between learning an optimal policy at the end of online training and not learning at all. Our findings motivate the widespread adoption of IL for pre-training and concurrent IL in procedurally generated environments whenever offline trajectories are available or can be generated.

翻译：强化学习（RL）的关键挑战之一在于智能体能否将所学策略泛化至未见环境。此外，训练RL智能体需要与环境进行大量交互。受离线RL与模仿学习（IL）近期成功的启发，我们开展了一项研究，旨在探究智能体是否能够利用轨迹形式的离线数据，提升在程序化生成环境中的样本效率。我们考虑了两种利用离线数据进行IL的RL应用场景：（1）在线RL训练前进行策略预训练；（2）在线RL与离线IL并行训练策略。我们分析了可用离线轨迹的质量（轨迹最优性）与多样性（轨迹数量及覆盖层级）对两种方法有效性的影响。在MiniGrid环境中四项知名的稀疏奖励任务上，我们发现采用IL进行预训练及与在线RL并行训练，均能在收敛至最优策略的同时持续提升样本效率。此外，研究表明，仅使用两条轨迹进行策略预训练，即可决定在线训练结束时能否学习到最优策略，抑或完全无法学习。我们的研究结果呼吁，在程序化生成环境中，只要存在或可生成离线轨迹，就应广泛采用IL进行预训练与并行训练。