Learning from human feedback plays an important role in aligning generative models, such as large language models (LLM). However, the effectiveness of this approach can be influenced by adversaries, who may intentionally provide misleading preferences to manipulate the output in an undesirable or harmful direction. To tackle this challenge, we study a specific model within this problem domain--contextual dueling bandits with adversarial feedback, where the true preference label can be flipped by an adversary. We propose an algorithm namely robust contextual dueling bandits (RCDB), which is based on uncertainty-weighted maximum likelihood estimation. Our algorithm achieves an $\tilde O(d\sqrt{T}/\kappa+dC/\kappa)$ regret bound, where $T$ is the number of rounds, $d$ is the dimension of the context, $\kappa$ is the lower bound of the derivative of the link function, and $ 0 \le C \le T$ is the total number of adversarial feedback. We also prove a lower bound to show that our regret bound is nearly optimal, both in scenarios with and without ($C=0$) adversarial feedback. Our work is the first to achieve nearly minimax optimal regret for dueling bandits in the presence of adversarial preference feedback. Additionally, for the sigmoid link function, we develop a novel algorithm that takes into account the effect of local derivatives into maximum likelihood estimation (MLE) analysis through a refined method for estimating the link function's derivative. This method helps us to eliminate the $\kappa$ dependence in the leading term with respect to $T$, which reduces the exponential dependence on the parameter radius $B$ to a polynomial dependence.

翻译：从人类反馈中学习在调整生成模型（例如大型语言模型LLM）方面发挥着重要作用。然而，这种方法的效果可能受到对抗者的影响，他们可能故意提供误导性偏好，以将输出操纵至不良或有害的方向。为应对这一挑战，我们研究了该问题域内的一个特定模型——具有对抗反馈的上下文对决赌博机，其中真实的偏好标签可能被对抗者翻转。我们提出了一种名为鲁棒上下文对决赌博机（RCDB）的算法，该算法基于不确定性加权的最大似然估计。我们的算法实现了$\tilde O(d\sqrt{T}/\kappa+dC/\kappa)$的遗憾界，其中$T$为轮数，$d$为上下文维度，$\kappa$为链接函数导数的下界，$0 \le C \le T$为对抗反馈的总数。我们还证明了一个下界，以表明无论在存在或不存在（$C=0$）对抗反馈的情况下，我们的遗憾界都是近乎最优的。我们的工作是首个在对决赌博机存在对抗偏好反馈的情况下实现近乎极小极大最优遗憾的研究。此外，针对sigmoid链接函数，我们开发了一种新颖算法，通过一种改进的链接函数导数估计方法，将局部导数的影响纳入最大似然估计（MLE）分析。该方法帮助我们消除了关于$T$的主导项中对$\kappa$的依赖，从而将关于参数半径$B$的指数依赖降低为多项式依赖。