We consider a hybrid reinforcement learning setting (Hybrid RL), in which an agent has access to an offline dataset and the ability to collect experience via real-world online interaction. The framework mitigates the challenges that arise in both pure offline and online RL settings, allowing for the design of simple and highly effective algorithms, in both theory and practice. We demonstrate these advantages by adapting the classical Q learning/iteration algorithm to the hybrid setting, which we call Hybrid Q-Learning or Hy-Q. In our theoretical results, we prove that the algorithm is both computationally and statistically efficient whenever the offline dataset supports a high-quality policy and the environment has bounded bilinear rank. Notably, we require no assumptions on the coverage provided by the initial distribution, in contrast with guarantees for policy gradient/iteration methods. In our experimental results, we show that Hy-Q with neural network function approximation outperforms state-of-the-art online, offline, and hybrid RL baselines on challenging benchmarks, including Montezuma's Revenge.

翻译：我们考虑一种混合强化学习设置（Hybrid RL），其中智能体既能访问离线数据集，又能通过真实世界在线交互收集经验。该框架缓解了纯离线与纯在线强化学习设置中出现的挑战，使得在理论与实践层面均可设计出简单且高效的算法。我们通过将经典Q学习/迭代算法适配至混合设置（称为Hybrid Q-Learning或Hy-Q）来展示这些优势。在理论结果中，我们证明：只要离线数据集支持高质量策略且环境具有有界双线性秩，该算法便兼具计算效率与统计效率。值得注意的是，与策略梯度/迭代方法的保证条件不同，我们无需对初始分布的覆盖范围做任何假设。在实验结果中，我们表明采用神经网络函数近似的Hy-Q在具有挑战性的基准测试（包括《蒙特祖玛的复仇》）上，性能超越了当前最先进的在线、离线及混合强化学习基线方法。