Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedback, yet existing exploratory bonus methods to incentivize exploration often fail to realize optimism. We provide a theoretical analysis showing that current formulations, under KL or $α$-divergence regularization, unintentionally bias exploration toward high-probability regions of the reference model, thereby reinforcing conservative behavior instead of promoting discovery of uncertain regions. To address this pitfall, we introduce the General Exploratory Bonus (GEB), a novel theoretical framework that provably satisfies the optimism principle. GEB counteracts divergence-induced bias via reference-dependent reward regulation and unifies prior heuristic bonuses as special cases, while extending naturally across the full $α$-divergence family. Empirically, GEB consistently outperforms baselines on alignment tasks across multiple divergence settings and large language model backbones. These results demonstrate that GEB offers both a principled and practical solution for optimistic exploration in RLHF.

翻译：乐观探索对于提升基于人类反馈的强化学习样本效率至关重要，然而现有的激励探索的奖励方法往往无法实现真正的乐观性。我们通过理论分析表明，当前基于KL散度或$α$-散度正则化的方法会无意中将探索偏向参考模型的高概率区域，从而强化了保守行为而非促进对不确定区域的发现。为解决这一缺陷，我们提出了通用探索奖励（GEB）这一新颖的理论框架，该框架可证明满足乐观性原则。GEB通过参考依赖的奖励调节机制抵消散度诱导的偏差，并将先前的启发式奖励方法统一为特例，同时自然地扩展至完整的$α$-散度族。实证研究表明，在多种散度设置和大语言模型基座上，GEB在对齐任务中始终优于基线方法。这些结果证明，GEB为RLHF中的乐观探索提供了兼具理论原则性与实践可行性的解决方案。