Deep Reinforcement Learning is widely used for aligning Large Language Models (LLM) with human preference. However, the conventional reward modelling has predominantly depended on human annotations provided by a select cohort of individuals. Such dependence may unintentionally result in models that are skewed to reflect the inclinations of these annotators, thereby failing to represent the expectations of the wider population adequately. In this paper, we introduce the Distributional Preference Reward Model (DPRM), a simple yet effective framework to align large language models with a diverse set of human preferences. To this end, we characterize the preferences by a beta distribution, which can dynamically adapt to fluctuations in preference trends. On top of that, we design an optimal-transportation-based loss to calibrate DPRM to align with the preference distribution. Finally, the expected reward is utilized to fine-tune an LLM policy to generate responses favoured by the population. Our experiments show that DPRM significantly enhances the alignment of LLMs with population preference, yielding more accurate, unbiased, and contextually appropriate responses.

翻译：深度强化学习被广泛用于将大型语言模型（LLM）与人类偏好对齐。然而，传统的奖励建模主要依赖于由少数个体提供的人工标注。这种依赖可能无意中导致模型偏向反映这些标注者的倾向，从而无法充分代表更广泛群体的期望。本文提出分布偏好奖励模型（DPRM），这是一种简单而有效的框架，旨在将大型语言模型与多样的人类偏好对齐。为此，我们通过贝塔分布来刻画偏好，该分布能够动态适应偏好趋势的波动。在此基础上，我们设计了一种基于最优输运的损失函数来校准DPRM，使其与偏好分布对齐。最后，利用期望奖励对LLM策略进行微调，以生成受群体偏好的响应。实验表明，DPRM显著增强了LLM与群体偏好的一致性，生成了更准确、无偏且上下文恰当的响应。