A compelling use case of offline reinforcement learning (RL) is to obtain a policy initialization from existing datasets followed by fast online fine-tuning with limited interaction. However, existing offline RL methods tend to behave poorly during fine-tuning. In this paper, we study the fine-tuning problem in the context of conservative offline RL methods and we devise an approach for learning an effective initialization from offline data that also enables fast online fine-tuning capabilities. Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also ensuring that the learned Q-values are at a reasonable scale. We refer to this property as calibration, and define it formally as providing a lower bound on the true value function of the learned policy and an upper bound on the value of some other (suboptimal) reference policy, which may simply be the behavior policy. We show that a conservative offline RL algorithm that also learns a calibrated value function leads to effective online fine-tuning, enabling us to take the benefits of offline initializations in online fine-tuning. In practice, Cal-QL can be implemented on top of the conservative Q learning (CQL) for offline RL within a one-line code change. Empirically, Cal-QL outperforms state-of-the-art methods on 9/11 fine-tuning benchmark tasks that we study in this paper. Code and video are available at https://nakamotoo.github.io/projects/Cal-QL

翻译：摘要：离线强化学习的一个引人注目的应用场景是从现有数据集中获取策略初始化，随后通过有限交互进行快速在线微调。然而，现有离线强化学习方法在微调期间往往表现不佳。本文在保守离线强化学习方法的背景下研究了微调问题，并提出了一种从离线数据中学习有效初始化的方法，该方法同时具备快速在线微调能力。我们的方法——校准Q学习（Cal-QL）——通过以下方式实现这一目标：学习一个保守的值函数初始化，该初始化低估了从离线数据学得的策略的价值，同时确保学得的Q值处于合理量级。我们将此性质称为校准性，并正式定义其为：对学得策略的真实值函数提供下界，并对其他（次优）参考策略（可简单为行为策略）的价值提供上界。我们证明，一种同时学习校准值函数的保守离线强化学习算法能够实现有效的在线微调，从而让我们在在线微调中充分利用离线初始化的优势。实践中，Cal-QL可在保守Q学习（CQL）离线强化学习算法基础上通过单行代码修改实现。实验表明，在我们研究的11个微调基准任务中，Cal-QL在9个任务上优于现有最先进方法。代码与视频见 https://nakamotoo.github.io/projects/Cal-QL