Large foundation models pretrained on raw web-scale data are not readily deployable without additional step of extensive alignment to human preferences. Such alignment is typically done by collecting large amounts of pairwise comparisons from humans ("Do you prefer output A or B?") and learning a reward model or a policy with the Bradley-Terry-Luce (BTL) model as a proxy for a human's underlying implicit preferences. These methods generally suffer from assuming a universal preference shared by all humans, which lacks the flexibility of adapting to plurality of opinions and preferences. In this work, we propose PAL, a framework to model human preference complementary to existing pretraining strategies, which incorporates plurality from the ground up. We propose using the ideal point model as a lens to view alignment using preference comparisons. Together with our novel reformulation and using mixture modeling, our framework captures the plurality of population preferences while simultaneously learning a common preference latent space across different preferences, which can few-shot generalize to new, unseen users. Our approach enables us to use the penultimate-layer representation of large foundation models and simple MLP layers to learn reward functions that are on-par with the existing large state-of-the-art reward models, thereby enhancing efficiency of reward modeling significantly. We show that PAL achieves competitive reward model accuracy compared to strong baselines on 1) Language models with Summary dataset ; 2) Image Generative models with Pick-a-Pic dataset ; 3) A new semisynthetic heterogeneous dataset generated using Anthropic Personas. Finally, our experiments also highlight the shortcoming of current preference datasets that are created using rigid rubrics which wash away heterogeneity, and call for more nuanced data collection approaches.

翻译：在原始网络规模数据上预训练的大型基础模型，若不经额外的广泛对齐步骤以适应人类偏好，则难以直接部署。此类对齐通常通过收集大量来自人类的成对比较（“您更偏好输出A还是B？”）并借助Bradley-Terry-Luce（BTL）模型作为人类潜在隐式偏好的代理，来学习奖励模型或策略。这些方法普遍存在一个缺陷，即假设所有人类共享一种统一偏好，缺乏适应多元观点与偏好的灵活性。本研究中，我们提出PAL框架，作为对现有预训练策略的补充，用于建模人类偏好，并从底层设计即纳入多元性。我们提出以理想点模型作为视角，通过偏好比较进行对齐。结合我们新颖的重构方法与混合建模技术，本框架能够捕捉群体偏好的多元性，同时学习跨不同偏好的共同潜在偏好空间，从而能够对新出现的、未见过的用户进行少样本泛化。我们的方法使得能够利用大型基础模型的倒数第二层表示与简单的MLP层，学习出与现有大型先进奖励模型性能相当的奖励函数，从而显著提升奖励建模的效率。实验表明，PAL在以下任务中均取得了与强基线模型相竞争的奖励模型准确率：1）基于摘要数据集的语言模型；2）基于Pick-a-Pic数据集的图像生成模型；3）使用Anthropic Personas生成的新型半合成异构数据集。最后，我们的实验也揭示了当前偏好数据集的不足——这些数据集通常基于僵化的评分标准创建，从而抹消了异质性，因此我们呼吁采用更细致的数据收集方法。