Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM simultaneously. In this paper, we introduce Proxy-RLHF, which decouples the generation and alignment processes of LLMs, achieving alignment with human values at a much lower computational cost. We start with a novel Markov Decision Process (MDP) designed for the alignment process and employ Reinforcement Learning (RL) to train a streamlined proxy model that oversees the token generation of the LLM, without altering the LLM itself. Experiments show that our method achieves a comparable level of alignment with only 1\% of the training parameters of other methods.

翻译：基于人类反馈的强化学习（RLHF）是确保大型语言模型（LLMs）与人类价值观对齐的主流方法。然而，现有RLHF方法需要高昂的计算成本，其主要原因之一是RLHF同时将生成任务和对齐任务分配给LLM执行。本文提出Proxy-RLHF，该方法将LLM的生成过程与对齐过程解耦，以更低的计算成本实现与人类价值观的对齐。我们首先设计了一个用于对齐过程的新型马尔可夫决策过程（MDP），并采用强化学习（RL）训练一个精简的代理模型，该模型在不改变LLM本身的前提下监督其token生成。实验表明，本方法仅需其他方法1%的训练参数即可达到相当的对齐水平。