This work investigates the offline formulation of the contextual bandit problem, where the goal is to leverage past interactions collected under a behavior policy to evaluate, select, and learn new, potentially better-performing, policies. Motivated by critical applications, we move beyond point estimators. Instead, we adopt the principle of pessimism where we construct upper bounds that assess a policy's worst-case performance, enabling us to confidently select and learn improved policies. Precisely, we introduce novel, fully empirical concentration bounds for a broad class of importance weighting risk estimators. These bounds are general enough to cover most existing estimators and pave the way for the development of new ones. In particular, our pursuit of the tightest bound within this class motivates a novel estimator (LS), that logarithmically smooths large importance weights. The bound for LS is provably tighter than all its competitors, and naturally results in improved policy selection and learning strategies. Extensive policy evaluation, selection, and learning experiments highlight the versatility and favorable performance of LS.

翻译：本研究探讨了上下文赌博机问题的离线形式，其目标在于利用行为策略收集的历史交互数据来评估、选择并学习新的、可能具有更优性能的策略。受关键应用场景的驱动，我们超越了点估计的范畴，转而采用悲观主义原则，构建用于评估策略最坏情况性能的上界，从而使我们能够可靠地选择和习得改进后的策略。具体而言，我们针对一大类重要性加权风险估计器提出了新颖的、完全基于经验的集中界。这些界具有足够的通用性，能够覆盖大多数现有估计器，并为开发新型估计器开辟了道路。特别地，为寻求该类中最紧致的界，我们提出了一种新颖的估计器（LS），该估计器通过对较大的重要性权重进行对数平滑处理。理论证明LS的界较所有现有方法更为紧致，并自然地催生了改进的策略选择与学习策略。大量的策略评估、选择与学习实验验证了LS的通用性及其优越的性能表现。