Reinforcement Learning from Human Feedback (RLHF) can be used to capture complex and nuanced properties of text generation quality. As a result, the task of text summarization has been identified as a good candidate for this process. In this paper, we explore how preference agreement impacts the efficacy of RLHF for summarization. We show that sampling human preferences to include a range of annotator agreement results in (1) higher accuracy reward models and (2) alters the characteristics of quality captured. We additionally show improvements in downstream generation when using a reward model trained with a range of preference agreements. Our contributions have implications for the design of synthetic datasets as well as the importance of considering quality differentials in comparison-based data.

翻译：基于人类反馈的强化学习（RLHF）能够捕捉文本生成质量的复杂细微特性，因此文本摘要任务被视为该方法的理想应用场景。本文探究偏好一致性如何影响RLHF在摘要生成中的有效性。研究表明，通过采样包含不同标注者一致性程度的人类偏好数据，可带来两方面的效果：（1）构建更高精度的奖励模型；（2）改变所捕获质量特征的表现形式。进一步实验证明，采用融合多种偏好一致性程度训练的奖励模型，能够提升下游文本生成质量。本研究的发现对于设计合成数据集具有指导意义，同时揭示了在基于比较的数据中考虑质量差异性的重要性。