This paper examines the use of Monte Carlo simulations to understand statistical concepts in A/B testing and Randomized Controlled Trials (RCTs). We discuss the applicability of simulations in understanding false positive rates and estimate statistical power, implementing variance reduction techniques and examining the effects of early stopping. By comparing frequentist and Bayesian approaches, we illustrate how simulations can clarify the relationship between p-values and posterior probabilities, and the validity of such approximations. The study also references how Monte Carlo simulations can be used to understand network effects in RCTs on social networks. Our findings show that Monte Carlo simulations are an effective tool for experimenters to deepen their understanding and ensure their results are statistically valid and practically meaningful.

翻译：本文探讨了利用蒙特卡洛模拟来理解A/B测试与随机对照试验中的统计概念。我们讨论了模拟在理解假阳性率、估计统计功效方面的适用性，并实现了方差缩减技术，同时检验了早期停止的影响。通过比较频率学派与贝叶斯学派的方法，我们阐明了模拟如何能够澄清p值与后验概率之间的关系，以及此类近似的有效性。本研究还涉及了如何运用蒙特卡洛模拟来理解社交网络上随机对照试验中的网络效应。我们的研究结果表明，蒙特卡洛模拟是实验者加深理解、确保其结果的统计有效性及实际意义的一种有效工具。