The selection of the assumed effect size (AES) critically determines the duration of an experiment, and hence its accuracy and efficiency. Traditionally, experimenters determine AES based on domain knowledge. However, this method becomes impractical for online experimentation services managing numerous experiments, and a more automated approach is hence of great demand. We initiate the study of data-driven AES selection in for online experimentation services by introducing two solutions. The first employs a three-layer Gaussian Mixture Model considering the heteroskedasticity across experiments, and it seeks to estimate the true expected effect size among positive experiments. The second method, grounded in utility theory, aims to determine the optimal effect size by striking a balance between the experiment's cost and the precision of decision-making. Through comparisons with baseline methods using both simulated and real data, we showcase the superior performance of the proposed approaches.

翻译：实验假定效应量（AES）的选择关键决定了实验时长，从而影响其准确性和效率。传统上，实验人员基于领域知识确定AES。然而，这种方法对于管理大量实验的在线实验服务而言难以实施，因此亟需更自动化的解决方案。我们通过提出两种方法，率先开展了面向在线实验服务的数据驱动型AES选择研究。第一种方法采用考虑实验间异方差性的三层高斯混合模型，旨在估计阳性实验中真实期望效应量。第二种方法基于效用理论，通过权衡实验成本与决策精度来确定最优效应量。通过使用模拟数据和真实数据与基线方法进行比较，我们展示了所提出方法的优越性能。