A B testing serves as the gold standard for large scale, data driven decision making in online businesses. To mitigate metric variability and enhance testing sensitivity, control variates and regression adjustment have emerged as prominent variance reduction techniques, leveraging pre experiment data to improve estimator performance. Over the past decade, these methods have spawned numerous derivatives, yet their theoretical connections and comparative properties remain underexplored. In this paper, we conduct a comprehensive analysis of their statistical properties, establish a formal bridge between the two frameworks in practical implementations, and extend the investigation from design based to model-based frameworks. Through simulation studies and real world experiments at ByteDance, we validate our theoretical insights across both frameworks. Our work aims to provide rigorous guidance for practitioners in online controlled experiments, addressing critical considerations of internal and external validity. The recommended method control variates with group specific coefficient estimates has been fully implemented and deployed on ByteDance's experimental platform.

翻译：A/B测试是在线业务中大规模数据驱动决策的黄金标准。为降低指标变异性并提升测试灵敏度，控制变量与回归调整已成为主流的方差缩减技术，它们利用实验前数据以改进估计量性能。过去十年间，这些方法衍生出众多变体，但其理论联系与比较特性仍未得到充分探索。本文系统分析了它们的统计特性，在实际应用中建立了两个框架间的正式桥梁，并将研究从基于设计的框架拓展至基于模型的框架。通过模拟研究及字节跳动的真实实验，我们在两种框架下验证了理论见解。本工作旨在为在线对照实验的实践者提供严谨指导，以应对内部效度与外部效度的关键考量。所推荐的具有组别特定系数估计的控制变量方法已在字节跳动实验平台上全面实施并部署。