Offline reinforcement learning (RL) allows for the training of competent agents from offline datasets without any interaction with the environment. Online finetuning of such offline models can further improve performance. But how should we ideally finetune agents obtained from offline RL training? While offline RL algorithms can in principle be used for finetuning, in practice, their online performance improves slowly. In contrast, we show that it is possible to use standard online off-policy algorithms for faster improvement. However, we find this approach may suffer from policy collapse, where the policy undergoes severe performance deterioration during initial online learning. We investigate the issue of policy collapse and how it relates to data diversity, algorithm choices and online replay distribution. Based on these insights, we propose a conservative policy optimization procedure that can achieve stable and sample-efficient online learning from offline pretraining.

翻译：离线强化学习使得从离线数据集中训练出有能力的智能体成为可能，而无需与环境进行任何交互。对此类离线模型进行在线微调可以进一步提升性能。但理想情况下，我们应该如何微调通过离线强化学习训练得到的智能体？虽然离线强化学习算法原则上可用于微调，但在实践中，其在线性能提升缓慢。相比之下，我们展示了使用标准在线异策略算法可以实现更快的改进。然而，我们发现这种方法可能会遭遇策略崩溃问题，即在初始在线学习阶段，策略性能严重恶化。我们研究了策略崩溃问题及其与数据多样性、算法选择和在线回放分布的关系。基于这些见解，我们提出了一种保守的策略优化方法，该方法能够在离线预训练的基础上实现稳定且样本高效的在线学习。