Generative foundation models are susceptible to implicit biases that can arise from extensive unsupervised training data. Such biases can produce suboptimal samples, skewed outcomes, and unfairness, with potentially significant repercussions. Consequently, aligning these models with human ethics and preferences is an essential step toward ensuring their responsible and effective deployment in real-world applications. Prior research has primarily employed Reinforcement Learning from Human Feedback (RLHF) as a means of addressing this problem, wherein generative models are fine-tuned using RL algorithms guided by a human-feedback-informed reward model. However, the inefficiencies and instabilities associated with RL algorithms frequently present substantial obstacles to the successful alignment of generative models, necessitating the development of a more robust and streamlined approach. To this end, we introduce a new framework, Reward rAnked FineTuning (RAFT), designed to align generative models more effectively. Utilizing a reward model and a sufficient number of samples, our approach selects the high-quality samples, discarding those that exhibit undesired behavior, and subsequently assembles a streaming dataset. This dataset serves as the basis for aligning the generative model and can be employed under both offline and online settings. Notably, the sample generation process within RAFT is gradient-free, rendering it compatible with black-box generators. Through extensive experiments, we demonstrate that our proposed algorithm exhibits strong performance in the context of both large language models and diffusion models.

翻译：生成式基础模型容易受到来自海量无监督训练数据中隐含偏差的影响。这些偏差可能导致次优样本、偏斜结果和不公平性，并可能产生显著后果。因此，将这些模型与人类伦理和偏好对齐，是确保其在现实应用中负责任且有效部署的关键步骤。先前研究主要采用基于人类反馈的强化学习（Reinforcement Learning from Human Feedback, RLHF）来解决这一问题，即通过由人类反馈信息训练的奖励模型引导的强化学习算法对生成模型进行微调。然而，与强化学习算法相关的低效性和不稳定性常常为生成模型的成功对齐带来重大障碍，亟需开发一种更具鲁棒性且更简化的方法。为此，我们提出了一种新框架——奖励排序微调（Reward rAnked FineTuning, RAFT），旨在更有效地对齐生成模型。该方法利用奖励模型和充足样本，筛选出高质量样本，丢弃具有不良行为的样本，并随后构建流式数据集。该数据集作为对齐生成模型的基础，可在离线与在线两种设置下使用。值得注意的是，RAFT中的样本生成过程无需梯度，使其兼容黑盒生成器。通过大量实验，我们证明所提出的算法在大语言模型和扩散模型场景下均表现出强劲性能。