Learning from human preference data is becoming a useful tool, from fine-tuning large language models to training reinforcement learning agents. However, in most scenarios, the model is trained on the average preference of all human evaluators, which, under large variations of preferences, can be unfair to minority groups. In this work, we consider fairness in dueling bandits, a standard framework for online learning from preference data. We assume that each user has a (potentially distinct) Condorcet winner, which is an arm preferred to every other arm. Using these user-specific Condorcet winners as reference points, we evaluate and score arms according to their performance relative to the corresponding winner. To promote fairness across heterogeneous users, we adopt the well-established Nash Social Welfare objective, which maximizes the product of user utilities, thereby inherently penalizing inequality and preventing the marginalization of any single user. Within this framework, we construct a hard instance to establish a regret lower bound of $Ω(T^{2/3}\min(K,D)^\frac{1}{3})$ for a time horizon $T$, $K$ arms, and $D$ users, which, to the best of our knowledge, is the first result quantifying the cost of fairness in dueling bandits with heterogeneous preferences. We then present the Fair-Explore-Then-Commit and Fair-$ε$-Greedy algorithms with a Condorcet winner identification phase. We further derive their regret upper bounds that match the lower-bound dependence on $T$ up to logarithmic factors.

翻译：从人类偏好数据中学习正成为一项实用工具，从微调大型语言模型到训练强化学习智能体。然而在多数场景中，模型基于所有人类评估者的平均偏好进行训练，当偏好存在较大差异时，这可能会对少数群体产生不公。本研究考虑对决式老虎机中的公平性问题——这是从偏好数据中进行在线学习的标准框架。我们假设每位用户拥有（可能不同的）孔多塞赢家，即优于所有其他臂的臂。以这些用户特定的孔多塞赢家为基准，我们根据各臂相对于对应赢家的表现进行评估与评分。为促进异质性用户间的公平性，我们采用成熟的纳什社会福利目标函数，该函数通过最大化用户效用的乘积来固有地惩罚不平等性并防止任何单一个体被边缘化。在此框架下，我们构建了一个困难实例，建立时间范围为$T$、臂数为$K$、用户数为$D$时的遗憾下界$\Omega(T^{2/3}\min(K,D)^\frac{1}{3})$——据我们所知，这是首个量化异质性偏好对决式老虎机中公平性代价的结果。随后我们提出包含孔多塞赢家识别阶段的Fair-Explore-Then-Commit和Fair-$ε$-Greedy算法，并推导出与$T$相关的下界匹配（对数因子内）的遗憾上界。