Reinforcement Learning (RL) has been used to finetune Large Language Models (LLMs) using a reward model trained from preference data, to better align with human judgment. The recently introduced direct alignment methods, which are often simpler, more stable, and computationally lighter, can more directly achieve this. However, these approaches cannot optimize arbitrary rewards, and the preference-based ones are not the only rewards of interest for LLMs (eg., unit tests for code generation or textual entailment for summarization, among others). RL-finetuning is usually done with a variation of policy gradient, which calls for on-policy or near-on-policy samples, requiring costly generations. We introduce Contrastive Policy Gradient, or CoPG, a simple and mathematically principled new RL algorithm that can estimate the optimal policy even from off-policy data. It can be seen as an off-policy policy gradient approach that does not rely on important sampling techniques and highlights the importance of using (the right) state baseline. We show this approach to generalize the direct alignment method IPO (identity preference optimization) and classic policy gradient. We experiment with the proposed CoPG on a toy bandit problem to illustrate its properties, as well as for finetuning LLMs on a summarization task, using a learned reward function considered as ground truth for the purpose of the experiments.

翻译：强化学习（RL）已被用于通过基于偏好数据训练得到的奖励模型来微调大型语言模型（LLMs），以更好地与人类判断对齐。最近引入的直接对齐方法通常更简单、更稳定且计算量更轻，能够更直接地实现这一目标。然而，这些方法无法优化任意奖励，且基于偏好的奖励并非LLMs唯一关注的奖励类型（例如，代码生成的单元测试或摘要的文本蕴含等）。RL微调通常采用策略梯度的变体实现，这需要同策略或近似同策略的样本，导致生成成本高昂。本文提出对比策略梯度（CoPG），这是一种简单且数学原理严谨的新型RL算法，即使仅使用离策略数据也能估计最优策略。它可以被视为一种不依赖于重要性采样技术、并强调使用（正确的）状态基线重要性的离策略策略梯度方法。我们证明该方法可推广直接对齐方法IPO（身份偏好优化）和经典策略梯度。我们通过一个玩具赌博机问题展示了CoPG的特性，并利用在实验中视为真实奖励的已学习奖励函数，在摘要任务上对LLMs进行微调实验。