A/B-tests are a cornerstone of experimental design on the web, with wide-ranging applications and use-cases. The statistical $t$-test comparing differences in means is the most commonly used method for assessing treatment effects, often justified through the Central Limit Theorem (CLT). The CLT ascertains that, as the sample size grows, the sampling distribution of the Average Treatment Effect converges to normality, making the $t$-test valid for sufficiently large sample sizes. When outcome measures are skewed or non-normal, quantifying what "sufficiently large" entails is not straightforward. To ensure that confidence intervals maintain proper coverage and that $p$-values accurately reflect the false positive rate, it is critical to validate this normality assumption. We propose a practical method to test this, by analysing repeatedly resampled A/A-tests. When the normality assumption holds, the resulting $p$-value distribution should be uniform, and this property can be tested using the Kolmogorov-Smirnov test. This provides an efficient and effective way to empirically assess whether the $t$-test's assumptions are met, and the A/B-test is valid. We demonstrate our methodology and highlight how it helps to identify scenarios prone to inflated Type-I errors. Our approach provides a practical framework to ensure and improve the reliability and robustness of A/B-testing practices.

翻译：A/B测试是网络实验设计的基石，具有广泛的应用场景。基于均值差异比较的统计t检验是评估处理效应最常用的方法，其理论依据通常来自中心极限定理（CLT）。CLT确保随着样本量增加，平均处理效应的抽样分布逐渐趋近正态分布，使得t检验在足够大的样本量下具有有效性。当结果度量指标呈偏态或非正态分布时，“足够大”的具体样本量标准并不明确。为确保置信区间保持正确的覆盖范围且p值能准确反映假阳性率，验证正态性假设至关重要。我们提出一种通过重复重采样A/A测试进行分析的实用检验方法。当正态性假设成立时，所得p值分布应服从均匀分布，这一特性可通过Kolmogorov-Smirnov检验进行验证。该方法为经验性评估t检验假设是否满足及A/B测试是否有效提供了高效实用的途径。我们展示了该方法的具体应用，并重点说明其如何帮助识别容易产生I类错误膨胀的场景。本方法为保障和提升A/B测试实践的可靠性与稳健性提供了实用框架。