Evaluating the performance of an ongoing policy plays a vital role in many areas such as medicine and economics, to provide crucial instruction on the early-stop of the online experiment and timely feedback from the environment. Policy evaluation in online learning thus attracts increasing attention by inferring the mean outcome of the optimal policy (i.e., the value) in real-time. Yet, such a problem is particularly challenging due to the dependent data generated in the online environment, the unknown optimal policy, and the complex exploration and exploitation trade-off in the adaptive experiment. In this paper, we aim to overcome these difficulties in policy evaluation for online learning. We explicitly derive the probability of exploration that quantifies the probability of exploring the non-optimal actions under commonly used bandit algorithms. We use this probability to conduct valid inference on the online conditional mean estimator under each action and develop the doubly robust interval estimation (DREAM) method to infer the value under the estimated optimal policy in online learning. The proposed value estimator provides double protection on the consistency and is asymptotically normal with a Wald-type confidence interval provided. Extensive simulations and real data applications are conducted to demonstrate the empirical validity of the proposed DREAM method.

翻译：评估正在运行策略的性能在医学和经济学等多个领域具有重要作用，可为在线实验的提前终止提供关键指导，并获取环境的及时反馈。在线学习中的策略评估通过实时推断最优策略的平均结果（即价值函数）而日益受到关注。然而，由于在线环境中产生的相依数据、未知的最优策略，以及自适应实验中复杂的探索与利用权衡问题，这类问题极具挑战性。本文旨在克服在线学习策略评估中的这些困难。我们明确推导了常见赌博机算法下用于量化探索非最优动作概率的探索概率，并据此对每个动作下的在线条件均值估计量进行有效推断，进而提出双重稳健区间估计方法（DREAM），以推断在线学习中估计最优策略下的价值函数。所提出的价值估计量在一致性上具有双重保护，且渐近服从正态分布，并提供Wald型置信区间。通过大量模拟和实际数据应用，验证了所提出DREAM方法的实证有效性。