In the quest to advance human-centric natural language generation (NLG) systems, ensuring alignment between NLG models and human preferences is crucial. For this alignment, current popular methods leverage a reinforcement learning (RL) approach with a reward model trained on feedback from humans. However, inherent disagreements due to the subjective nature of human preferences pose a significant challenge for training the reward model, resulting in a deterioration of the NLG performance. To tackle this issue, previous approaches typically rely on majority voting or averaging to consolidate multiple inconsistent preferences into a merged one. Although straightforward to understand and execute, such methods suffer from an inability to capture the nuanced degrees of disaggregation among humans and may only represent a specialized subset of individuals, thereby lacking the ability to quantitatively disclose the universality of human preferences. To address this challenge, this paper proposes a novel approach, which employs a Bayesian framework to account for the distribution of disagreements among human preferences as training a preference model, and names it as d-PM. Besides, considering the RL strategy's inefficient and complex training process over the training efficiency, we further propose utilizing the contrastive learning strategy to train the NLG model with the preference scores derived from the d-PM model. Extensive experiments on two human-centric NLG tasks, i.e., emotional support conversation and integrity "Rule-of-Thumb" generation, show that our method consistently exceeds previous SOTA models in both automatic and human evaluations.

翻译：在推进以人为中心的自然语言生成（NLG）系统时，确保NLG模型与人类偏好之间的校准至关重要。当前主流方法采用基于人类反馈训练奖励模型的强化学习（RL）方法来实现这种校准。然而，人类偏好具有主观性，固有的分歧对训练奖励模型构成了重大挑战，导致NLG性能下降。为解决此问题，现有方法通常依赖多数投票或平均投票将多个不一致的偏好整合为单一偏好。尽管这些方法易于理解和执行，但它们难以捕捉人类偏好间细微的分歧程度，且可能仅代表特定个体子集，缺乏定量揭示人类偏好普遍性的能力。为此，本文提出一种新方法，在训练偏好模型时采用贝叶斯框架来表征人类偏好中的分歧分布，命名为d-PM。此外，针对RL策略在训练效率上存在的低效和复杂性问题，我们进一步提出利用对比学习策略，基于d-PM模型生成的偏好得分来训练NLG模型。在两项以人为中心的NLG任务（情感支持对话与完整性"经验法则"生成）上的大量实验表明，我们的方法在自动评估和人工评估中均持续超越之前的最优模型（SOTA）。