The technique of Reinforcement Learning from Human Feedback (RLHF) is a commonly employed method to improve pre-trained Language Models (LM), enhancing their ability to conform to human preferences. Nevertheless, the current RLHF-based LMs necessitate full retraining each time novel queries or feedback are introduced, which becomes a challenging task because human preferences can vary between different domains or tasks. Retraining LMs poses practical difficulties in many real-world situations due to the significant time and computational resources required, along with concerns related to data privacy. To address this limitation, we propose a new method called Continual Optimal Policy Fitting (COPF), in which we estimate a series of optimal policies using the Monte Carlo method, and then continually fit the policy sequence with the function regularization. COPF involves a single learning phase and doesn't necessitate complex reinforcement learning. Importantly, it shares the capability with RLHF to learn from unlabeled data, making it flexible for continual preference learning. Our experimental results show that COPF outperforms strong Continuous learning (CL) baselines when it comes to consistently aligning with human preferences on different tasks and domains.

翻译：基于人类反馈的强化学习（RLHF）技术是一种提升预训练语言模型（LM）的常用方法，旨在增强其遵循人类偏好的能力。然而，当前基于RLHF的语言模型在每次引入新的查询或反馈时都需要完全重新训练，由于人类偏好在不同领域或任务间可能发生变化，这成为一项挑战性任务。在许多实际场景中，重新训练语言模型会因显著的时间与计算资源需求以及数据隐私问题而面临实践困难。为解决这一局限，我们提出了一种名为“持续最优策略拟合”（COPF）的新方法：该方法通过蒙特卡洛方法估计一系列最优策略，并利用函数正则化持续拟合策略序列。COPF仅需单阶段学习过程，无需复杂的强化学习。重要的是，它具备与RLHF相同的从未标注数据中学习的能力，从而为持续偏好学习提供了灵活性。实验结果表明，在不同任务和领域上持续对齐人类偏好时，COPF的性能优于强持续学习（CL）基线方法。