Deep Reinforcement Learning is widely used for aligning Large Language Models (LLM) with human preference. However, the conventional reward modelling is predominantly dependent on human annotations provided by a select cohort of individuals. Such dependence may unintentionally result in skewed models that reflect the inclinations of these annotators, thereby failing to adequately represent the wider population's expectations. We propose the Distributional Preference Reward Model (DPRM), a simple yet effective framework to align large language models with diverse human preferences. To this end, we characterize multiple preferences by a categorical distribution and introduce a Bayesian updater to accommodate shifted or new preferences. On top of that, we design an optimal-transportation-based loss to calibrate DPRM to align with the preference distribution. Finally, the expected reward is utilized to fine-tune an LLM policy to generate responses favoured by the population. Our experiments show that DPRM significantly enhances the alignment of LLMs with population preference, yielding more accurate, unbiased, and contextually appropriate responses.

翻译：深度强化学习被广泛应用于将大型语言模型与人类偏好对齐。然而，传统的奖励建模主要依赖于由特定人群提供的人工标注。这种依赖性可能导致模型无意中产生偏差，仅反映标注者的倾向，从而无法充分代表更广泛人群的期望。我们提出了分布偏好奖励模型，这是一个简单而有效的框架，用于将大型语言模型与多样化的人类偏好对齐。为此，我们通过分类分布来刻画多种偏好，并引入贝叶斯更新器以适应变化或新的偏好。在此基础上，我们设计了一种基于最优传输的损失函数来校准分布偏好奖励模型，使其与偏好分布对齐。最后，利用期望奖励对大型语言模型策略进行微调，以生成受人群偏好的响应。实验表明，分布偏好奖励模型显著增强了大型语言模型与人群偏好的对齐，产生了更准确、无偏见且上下文恰当的响应。