In the quest to advance human-centric natural language generation (NLG) systems, ensuring alignment between NLG models and human preferences is crucial. For this alignment, current popular methods leverage a reinforcement learning (RL) approach with a reward model trained on feedback from humans. However, inherent disagreements due to the subjective nature of human preferences pose a significant challenge for training the reward model, resulting in a deterioration of the NLG performance. To tackle this issue, previous approaches typically rely on majority voting or averaging to consolidate multiple inconsistent preferences into a merged one. Although straightforward to understand and execute, such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences. To address this challenge, this paper proposes a novel approach, which employs a Bayesian framework to account for the distribution of disagreements among human preferences as training a preference model, and names it as d-PM. Besides, considering the RL strategy's inefficient and complex training process over the training efficiency, we further propose utilizing the contrastive learning strategy to train the NLG model with the preference scores derived from the d-PM model. Extensive experiments on two human-centric NLG tasks, i.e., emotional support conversation and integrity "Rule-of-Thumb" generation, show that our method consistently exceeds previous SOTA models in both automatic and human evaluations.

翻译：在推进以人为中心的自然语言生成（NLG）系统发展过程中，确保NLG模型与人类偏好对齐至关重要。当前主流方法采用基于人类反馈训练的奖励模型驱动强化学习（RL）策略实现这一对齐。然而，人类偏好固有的主观性导致内在分歧，这对奖励模型训练构成显著挑战，进而损害NLG性能。为解决该问题，现有方法通常依赖多数投票或均值合并策略，将多个不一致偏好整合为统一表征。尽管这些方法易于理解与实现，但其无法捕捉人类偏好中细微的分歧程度差异，往往仅代表特定个体子集，因而难以定量揭示人类偏好的普适性。针对这一挑战，本文提出基于贝叶斯框架的创新方法（命名为d-PM），在训练偏好模型时对人类偏好的分歧分布进行建模。此外，针对RL策略在训练效率方面存在的复杂低效问题，我们进一步提出利用对比学习策略，基于d-PM模型输出的偏好分数训练NLG模型。在情感支持对话与完整性"经验法则"生成两项以人为中心的NLG任务上的大量实验表明，本方法在自动评估与人工评估中均持续超越现有最优模型。