Many real-world experimental design problems (a) evaluate multiple experimental conditions in parallel and (b) replicate each condition multiple times due to large and heteroscedastic observation noise. Given a fixed total budget, this naturally induces a trade-off between evaluating more unique conditions while replicating each of them fewer times vs. evaluating fewer unique conditions and replicating each more times. Moreover, in these problems, practitioners may be risk-averse and hence prefer an input with both good average performance and small variability. To tackle both challenges, we propose the Batch Thompson Sampling for Replicable Experimental Design (BTS-RED) framework, which encompasses three algorithms. Our BTS-RED-Known and BTS-RED-Unknown algorithms, for, respectively, known and unknown noise variance, choose the number of replications adaptively rather than deterministically such that an input with a larger noise variance is replicated more times. As a result, despite the noise heteroscedasticity, both algorithms enjoy a theoretical guarantee and are asymptotically no-regret. Our Mean-Var-BTS-RED algorithm aims at risk-averse optimization and is also asymptotically no-regret. We also show the effectiveness of our algorithms in two practical real-world applications: precision agriculture and AutoML.

翻译：许多现实中的实验设计问题（a）需并行评估多种实验条件，且（b）因观测噪声大且异方差，需对每种条件进行多次重复。在固定总预算下，这自然引发了一个权衡：是评估更多独特条件但减少重复次数，还是评估更少独特条件但增加重复次数。此外，实践者可能具有风险规避倾向，更偏好平均性能优异且变异性小的输入。为应对这两项挑战，我们提出了可复制实验设计的批量汤普森采样框架（BTS-RED），该框架包含三种算法。针对噪声方差已知与未知的情形，我们的BTS-RED-Known与BTS-RED-Unknown算法能够自适应地（而非确定性地）选择重复次数，使得噪声方差较大的输入获得更多重复。因此，尽管存在噪声异方差性，两种算法均具备理论保障，且渐近地实现无遗憾。我们的Mean-Var-BTS-RED算法面向风险规避优化，同样具有渐近无遗憾性质。我们在两个实际应用（精准农业与AutoML）中验证了算法的有效性。