Multi armed bandit (MAB) algorithms have been increasingly used to complement or integrate with A/B tests and randomized clinical trials in e-commerce, healthcare, and policymaking. Recent developments incorporate possible delayed feedback. While existing MAB literature often focuses on maximizing the expected cumulative reward outcomes (or, equivalently, regret minimization), few efforts have been devoted to establish valid statistical inference approaches to quantify the uncertainty of learned policies. We attempt to fill this gap by providing a unified statistical inference framework for policy evaluation where a target policy is allowed to differ from the data collecting policy, and our framework allows delay to be associated with the treatment arms. We present an adaptively weighted estimator that on one hand incorporates the arm-dependent delaying mechanism to achieve consistency, and on the other hand mitigates the variance inflation across stages due to vanishing sampling probability. In particular, our estimator does not critically depend on the ability to estimate the unknown delay mechanism. Under appropriate conditions, we prove that our estimator converges to a normal distribution as the number of time points goes to infinity, which provides guarantees for large-sample statistical inference. We illustrate the finite-sample performance of our approach through Monte Carlo experiments.

翻译：多臂老虎机（MAB）算法已被日益广泛地应用于电子商务、医疗健康及政策制定等领域，用于补充或整合A/B测试与随机临床试验。近期研究进展中引入了可能的延迟反馈。尽管现有MAB文献通常聚焦于最大化期望累积奖励结果（或等效于最小化遗憾值），但鲜有研究致力于建立有效的统计推断方法以量化学习策略的不确定性。我们试图弥补这一空白，提出一套统一的策略评估统计推断框架，允许目标策略与数据采集策略存在差异，且框架支持延迟与实验组别相关联。我们提出了一种自适应加权估计量：一方面通过引入与臂相关的延迟机制保证估计一致性，另一方面缓解因采样概率递减导致的跨阶段方差膨胀。特别地，该估计量不严重依赖于对未知延迟机制的估计能力。在适当条件下，我们证明了当时点数量趋于无穷时，该估计量收敛于正态分布，为大样本统计推断提供了理论保障。通过蒙特卡洛实验展示了我们方法在有限样本下的表现。