As language models (LMs) become more capable, it is increasingly important to align them with human preferences. However, the dominant paradigm for training Preference Models (PMs) for that purpose suffers from fundamental limitations, such as lack of transparency and scalability, along with susceptibility to overfitting the preference dataset. We propose Compositional Preference Models (CPMs), a novel PM framework that decomposes one global preference assessment into several interpretable features, obtains scalar scores for these features from a prompted LM, and aggregates these scores using a logistic regression classifier. Through these simple steps, CPMs allow to control which properties of the preference data are used to train the preference model and to build it based on features that are believed to underlie the human preference judgment. Our experiments show that CPMs not only improve generalization and are more robust to overoptimization than standard PMs, but also that best-of-n samples obtained using CPMs tend to be preferred over samples obtained using conventional PMs. Overall, our approach demonstrates the benefits of endowing PMs with priors about which features determine human preferences while relying on LM capabilities to extract those features in a scalable and robust way.

翻译：随着语言模型（LM）能力不断增强，使其与人类偏好对齐变得愈发重要。然而，当前用于训练偏好模型（PM）的主流范式存在根本性局限，例如缺乏透明度和可扩展性，且易对偏好数据集过拟合。我们提出组合偏好模型（CPM），这是一种全新的PM框架，将全局偏好评估分解为若干可解释特征，通过提示语言模型获取这些特征的标量得分，并利用逻辑回归分类器聚合得分。通过这一简洁流程，CPM可控制偏好数据中用于训练偏好模型的属性，并基于被认为构成人类偏好判断基础的特征构建模型。实验表明，CPM不仅相比标准PM具有更强的泛化能力和对过度优化的鲁棒性，且通过CPM获得的Best-of-N样本往往优于传统PM所得样本。总体而言，本方法证明了将先验知识注入PM（即哪些特征决定人类偏好），同时依赖LM能力以可扩展且稳健的方式提取这些特征，所带来的显著优势。