Online controlled experiments play a crucial role in enabling data-driven decisions across a wide range of companies. Variance reduction is an effective technique to improve the sensitivity of experiments, achieving higher statistical power while using fewer samples and shorter experimental periods. However, typical variance reduction methods (e.g., regression-adjusted estimators) are built upon the intuitional assumption of Gaussian distributions and cannot properly characterize the real business metrics with heavy-tailed distributions. Furthermore, outliers diminish the correlation between pre-experiment covariates and outcome metrics, greatly limiting the effectiveness of variance reduction. In this paper, we develop a novel framework that integrates the Student's t-distribution with machine learning tools to fit heavy-tailed metrics and construct a robust average treatment effect estimator in online controlled experiments, which we call STATE. By adopting a variational EM method to optimize the loglikehood function, we can infer a robust solution that greatly eliminates the negative impact of outliers and achieves significant variance reduction. Moreover, we extend the STATE method from count metrics to ratio metrics by utilizing linear transformation that preserves unbiased estimation, whose variance reduction is more complex but less investigated in existing works. Finally, both simulations on synthetic data and long-term empirical results on Meituan experiment platform demonstrate the effectiveness of our method. Compared with the state-of-the-art estimators (CUPAC/MLRATE), STATE achieves over 50% variance reduction, indicating it can reach the same statistical power with only half of the observations, or half the experimental duration.

翻译：在线对照实验在推动各类公司实现数据驱动决策方面发挥着至关重要的作用。方差缩减是一种提升实验灵敏度的有效技术，能够在使用更少样本和更短实验周期的同时，获得更高的统计功效。然而，典型的方差缩减方法（例如，回归调整估计器）建立在正态分布的直观假设之上，无法恰当刻画具有重尾分布的真实业务指标。此外，异常值削弱了实验前协变量与结果指标之间的相关性，极大地限制了方差缩减的效果。本文提出了一种新颖的框架，该框架将学生t分布与机器学习工具相结合，以拟合重尾指标并构建在线对照实验中稳健的平均处理效应估计器，我们称之为STATE。通过采用变分EM算法来优化对数似然函数，我们能够推断出一个稳健的解，该解极大地消除了异常值的负面影响，并实现了显著的方差缩减。此外，我们通过利用保持无偏估计的线性变换，将STATE方法从计数指标扩展到比率指标，其方差缩减更为复杂，但在现有工作中研究较少。最后，在合成数据上的模拟以及在美团实验平台上的长期实证结果均证明了我们方法的有效性。与最先进的估计器（CUPAC/MLRATE）相比，STATE实现了超过50%的方差缩减，这表明其仅需一半的观测值或一半的实验时长即可达到相同的统计功效。