Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).

翻译：大语言模型的细粒度控制仍是一项重大挑战，限制了其对多样化用户需求的适应性。尽管基于人类反馈的强化学习（RLHF）在模型对齐方面展现出潜力，但其依赖标量奖励的特性往往难以捕捉真实应用中用户的多样化偏好。为突破这一局限，我们提出了方向性偏好对齐（DPA）框架。与标量奖励的RLHF不同，DPA采用多目标奖励建模来表征多样化偏好配置。此外，DPA将用户偏好建模为奖励空间中的方向（即单位向量），从而实现用户依赖的偏好控制。我们的方法包含两个阶段：首先训练多目标奖励模型，随后使用偏好条件化的拒绝采样微调（RSF）变体对LLM进行微调——该RSF方法源自Llama 2采用的RLHF技术。该方法能在多个奖励目标之间实现更优的性能权衡。相较于标量奖励的RLHF，DPA为用户提供了对大模型生成的直观控制能力：他们可通过算术运算指定期望的权衡关系（例如，在降低冗长度的同时提升有用性）。我们还在Mistral-7B上通过真实对齐实验验证了DPA的有效性。本方法可在保持与直接偏好优化（DPO）等强基线模型相当性能的同时，对有用性与冗长度之间的权衡实现直观的算术控制。