Learning from preference-based feedback has recently gained considerable traction as a promising approach to align generative models with human interests. Instead of relying on numerical rewards, the generative models are trained using reinforcement learning with human feedback (RLHF). These approaches first solicit feedback from human labelers typically in the form of pairwise comparisons between two possible actions, then estimate a reward model using these comparisons, and finally employ a policy based on the estimated reward model. An adversarial attack in any step of the above pipeline might reveal private and sensitive information of human labelers. In this work, we adopt the notion of label differential privacy (DP) and focus on the problem of reward estimation from preference-based feedback while protecting privacy of each individual labelers. Specifically, we consider the parametric Bradley-Terry-Luce (BTL) model for such pairwise comparison feedback involving a latent reward parameter $\theta^* \in \mathbb{R}^d$. Within a standard minimax estimation framework, we provide tight upper and lower bounds on the error in estimating $\theta^*$ under both local and central models of DP. We show, for a given privacy budget $\epsilon$ and number of samples $n$, that the additional cost to ensure label-DP under local model is $\Theta \big(\frac{1}{ e^\epsilon-1}\sqrt{\frac{d}{n}}\big)$, while it is $\Theta\big(\frac{\text{poly}(d)}{\epsilon n} \big)$ under the weaker central model. We perform simulations on synthetic data that corroborate these theoretical results.

翻译：基于偏好的反馈学习近来作为使生成模型与人类兴趣对齐的一种有前途方法备受关注。生成模型并非依赖数值奖励，而是通过基于人类反馈的强化学习（RLHF）进行训练。这些方法首先以成对比较形式（即在两个可能行为间进行选择）征求人类标注者的反馈，随后利用这些比较估计奖励模型，最后基于估计的奖励模型制定策略。上述流程中任何步骤遭遇对抗性攻击都可能导致人类标注者的隐私和敏感信息泄露。本研究采用标签差分隐私（DP）概念，聚焦于保护每位标注者隐私前提下，从基于偏好的反馈中估计奖励的问题。具体而言，我们采用参数化Bradley-Terry-Luce（BTL）模型处理此类涉及潜在奖励参数$\theta^* \in \mathbb{R}^d$的成对比较反馈。在标准极小极大估计框架内，我们给出了DP局部模型和中心模型下估计$\theta^*$误差的严格上下界。研究表明，对于给定隐私预算$\epsilon$和样本数量$n$，局部模型下确保标签差分隐私的额外代价为$\Theta \big(\frac{1}{ e^\epsilon-1}\sqrt{\frac{d}{n}}\big)$，而较弱的中心模型下则为$\Theta\big(\frac{\text{poly}(d)}{\epsilon n} \big)$。我们在合成数据上进行了仿真实验，结果验证了这些理论结论。