Standard bandit algorithms that assume continual reallocation of measurement effort are challenging to implement due to delayed feedback and infrastructural/organizational difficulties. Motivated by practical instances involving a handful of reallocation epochs in which outcomes are measured in batches, we develop a new adaptive experimentation framework that can flexibly handle any batch size. Our main observation is that normal approximations, which are universal in statistical inference, can also guide the design of scalable adaptive designs. By deriving an asymptotic sequential experiment, we formulate a dynamic program that can leverage prior information on average rewards. We propose a simple iterative planning method, Residual Horizon Optimization, which selects sampling allocations by optimizing a planning objective with stochastic gradient descent. Our method significantly improves statistical power over standard adaptive policies, even when compared to Bayesian bandit algorithms (e.g., Thompson sampling) that require full distributional knowledge of individual rewards. Overall, we expand the scope of adaptive experimentation to settings which are difficult for standard adaptive policies, including problems with a small number of reallocation epochs, low signal-to-noise ratio, and unknown reward distributions.

翻译：标准多臂赌博机算法假设测量资源可连续重新分配，但因反馈延迟及基础设施/组织层面的困难而难以实际实施。受实践中少数重分配阶段（以批次测量结果）的启发，我们开发了一种能灵活处理任意批次大小的新型自适应实验框架。我们的核心发现是：统计推断中通用的正态近似方法同样可指导可扩展自适应实验的设计。通过推导渐近序贯实验，我们构建了一个能利用平均奖励先验信息的动态规划模型。我们提出了一种简单的迭代规划方法——残差水平优化，该方法通过随机梯度下降优化规划目标来选择采样分配。相较于需要完整奖励分布先验知识的贝叶斯多臂赌博机算法（如汤普森采样），我们的方法在统计功效上实现了显著提升。总体而言，我们将自适应实验的适用范围扩展至标准自适应策略难以处理的场景，包括重分配阶段数较少、信噪比低以及奖励分布未知的问题。