The selection of the assumed effect size (AES) critically determines the duration of an experiment, and hence its accuracy and efficiency. Traditionally, experimenters determine AES based on domain knowledge. However, this method becomes impractical for online experimentation services managing numerous experiments, and a more automated approach is hence of great demand. We initiate the study of data-driven AES selection in for online experimentation services by introducing two solutions. The first employs a three-layer Gaussian Mixture Model considering the heteroskedasticity across experiments, and it seeks to estimate the true expected effect size among positive experiments. The second method, grounded in utility theory, aims to determine the optimal effect size by striking a balance between the experiment's cost and the precision of decision-making. Through comparisons with baseline methods using both simulated and real data, we showcase the superior performance of the proposed approaches.

翻译：在线实验中，假定效应量（AES）的选择决定了实验持续时间，进而影响实验的精度与效率。传统上，实验者根据领域知识确定AES。然而，对于管理大量实验的在线实验服务来说，该方法缺乏可扩展性，因此亟需更自动化的解决方案。我们首次提出两种数据驱动的AES选择方法，以满足在线实验服务的需求。第一种方法采用考虑实验间异方差性的三层高斯混合模型，旨在估计阳性实验中真实期望效应量；第二种方法基于效用理论，通过权衡实验成本与决策精度来确定最优效应量。通过与基线方法在模拟和真实数据上的对比，我们验证了所提方法的优越性能。