Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm in artificial intelligence to align large models with human preferences. In this paper, we propose a novel statistical framework to simultaneously conduct the online decision-making and statistical inference on the optimal model using human preference data based on dynamic contextual information. Our approach introduces an efficient decision strategy that achieves both the optimal regret bound and the asymptotic distribution of the estimators. A key challenge in RLHF is handling the dependent online human preference outcomes with dynamic contexts. To address this, in the methodological aspect, we propose a two-stage algorithm starting with $ε$-greedy followed by exploitations; in the theoretical aspect, we tailor anti-concentration inequalities and matrix martingale concentration techniques to derive the uniform estimation rate and asymptotic normality of the estimators using dependent samples from both stages. Extensive simulation results demonstrate that our method outperforms state-of-the-art strategies. We apply the proposed framework to analyze the human preference data for ranking large language models on the Massive Multitask Language Understanding dataset, yielding insightful results on the performance of different large language models for medical anatomy knowledge.

翻译：基于人类反馈的强化学习（RLHF）已成为人工智能中使大模型与人类偏好对齐的关键范式。本文提出一种新颖的统计框架，基于动态上下文信息，利用人类偏好数据同时进行在线决策与最优模型的统计推断。该方法引入了一种高效决策策略，在实现最优遗憾界的同时获得估计量的渐近分布。RLHF的核心挑战在于处理具有动态上下文的在线人类偏好依赖结果。为应对这一挑战，方法层面，我们提出两阶段算法：以ε-贪婪策略启动，随后转为利用；理论层面，我们定制反集中不等式与矩阵鞅集中技术，推导依赖样本（来自两阶段）的均匀估计率及估计量渐近正态性。大量模拟实验表明，所提方法优于现有最优策略。我们将该框架应用于大规模多任务语言理解数据集上对大语言模型进行排序的人类偏好数据，揭示了不同大语言模型在医学解剖知识方面的性能洞见。