Accurate estimation of treatment effects in online A/B testing is challenging with zero-inflated and skewed metrics. Traditional tests, like Welch's t-test, often lack sensitivity with heavy-tailed data due to their reliance on means, as opposed to e.g., percentiles. The Controlled Experiments Using Pre-experiment Data (CUPED) technique improves sensitivity by reducing variance, yet that variance reduction is insufficient for highly skewed metrics. Alternatively, Yuen's t-test uses trimmed means to robustly handle outliers and skewness. This paper introduces a method that combines the variance reduction of CUPED with the robustness of Yuen's t-test to enhance hypothesis testing sensitivity. Our novel approach integrates trimmed data in a principled manner, offering a framework that balances variance reduction with robust location measures. We demonstrate improved detection of significant effects with smaller sample sizes, enabling quicker experimental decisions without sacrificing statistical power. This work broadens the utility of controlled experiments in environments characterized by highly skewed or high-variance data.

翻译：在线A/B测试中，对于零膨胀和偏态指标，治疗效应的准确估计具有挑战性。传统检验方法（如韦尔奇t检验）由于依赖均值而非百分位数等统计量，在处理重尾数据时往往灵敏度不足。利用实验前数据的受控实验（CUPED）技术通过降低方差来提升灵敏度，但对于高度偏态的指标，其方差缩减效果仍显不足。另一方面，Yuen的t检验采用截尾均值来稳健处理异常值和偏态。本文提出一种将CUPED的方差缩减能力与Yuen t检验的稳健性相结合的方法，以增强假设检验的灵敏度。我们的创新方法以原则性方式整合截尾数据，提供了一个在方差缩减与稳健位置度量之间取得平衡的框架。实验证明，该方法能以更小的样本量提升显著效应的检测能力，从而在不牺牲统计功效的前提下加速实验决策。这项工作拓展了受控实验在高度偏态或高方差数据环境中的适用性。