Deep Reinforcement Learning is widely used for aligning Large Language Models (LLM) with human preference. However, the conventional reward modelling has predominantly depended on human annotations provided by a select cohort of individuals. Such dependence may unintentionally result in models that are skewed to reflect the inclinations of these annotators, thereby failing to represent the expectations of the wider population adequately. In this paper, we introduce the Distributional Preference Reward Model (DPRM), a simple yet effective framework to align large language models with a diverse set of human preferences. To this end, we characterize the preferences by a beta distribution, which can dynamically adapt to fluctuations in preference trends. On top of that, we design an optimal-transportation-based loss to calibrate DPRM to align with the preference distribution. Finally, the expected reward is utilized to fine-tune an LLM policy to generate responses favoured by the population. Our experiments show that DPRM significantly enhances the alignment of LLMs with population preference, yielding more accurate, unbiased, and contextually appropriate responses.

翻译：深度强化学习被广泛用于将大型语言模型（LLM）与人类偏好对齐。然而，传统的奖励建模主要依赖少数个体提供的人工标注，这种依赖可能导致模型无意中偏向反映这些标注者的倾向，从而无法充分代表更广泛群体的期望。本文提出分布偏好奖励模型（DPRM），这是一种简单而有效的框架，旨在将大型语言模型与多样化的人类偏好对齐。为此，我们采用贝塔分布来表征偏好，该分布能够动态适应偏好趋势的波动。在此基础上，我们设计了一种基于最优传输的损失函数来校准DPRM，使其与偏好分布对齐。最后，利用期望奖励对LLM策略进行微调，以生成符合群体偏好的响应。实验表明，DPRM显著增强了LLM与群体偏好的一致性，生成的响应更加准确、无偏且上下文恰当。