Hybrid RL is the setting where an RL agent has access to both offline data and online data by interacting with the real-world environment. In this work, we propose a new hybrid RL algorithm that combines an on-policy actor-critic method with offline data. On-policy methods such as policy gradient and natural policy gradient (NPG) have shown to be more robust to model misspecification, though sometimes it may not be as sample efficient as methods that rely on off-policy learning. On the other hand, offline methods that depend on off-policy training often require strong assumptions in theory and are less stable to train in practice. Our new approach integrates a procedure of off-policy training on the offline data into an on-policy NPG framework. We show that our approach, in theory, can obtain a best-of-both-worlds type of result -- it achieves the state-of-art theoretical guarantees of offline RL when offline RL-specific assumptions hold, while at the same time maintaining the theoretical guarantees of on-policy NPG regardless of the offline RL assumptions' validity. Experimentally, in challenging rich-observation environments, we show that our approach outperforms a state-of-the-art hybrid RL baseline which only relies on off-policy policy optimization, demonstrating the empirical benefit of combining on-policy and off-policy learning. Our code is publicly available at https://github.com/YifeiZhou02/HNPG.

翻译：混合强化学习是指强化学习代理能够同时通过真实环境交互获取在线数据和离线数据的设定。本文提出一种新的混合强化学习算法，将在线策略的演员-评论家方法与离线数据相结合。已有研究表明，策略梯度和自然策略梯度等在线策略方法对模型误定具有更强的鲁棒性，尽管其样本效率可能低于依赖离线策略学习的方法。另一方面，依赖离线策略训练的离线方法在理论上需要强假设，且实际训练中稳定性较差。我们的新方法将离线数据上的离线策略训练过程整合进在线策略的自然策略梯度框架中。理论分析表明，该方法能获得"两全其美"的结果：当离线强化学习的特定假设成立时，能达到当前离线强化学习的最优理论保证；同时无论这些假设是否成立，都能保持在线自然策略梯度的理论保证。在具有挑战性的高观测复杂性环境中，实验结果表明该方法优于仅依赖离线策略策略优化的最先进混合强化学习基线，这验证了结合在线与离线策略学习的实际效益。我们的代码已开源在 https://github.com/YifeiZhou02/HNPG。