The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this paper we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry.

翻译：20世纪90年代末，基于互联网的服务与产品的兴起为在线企业带来了前所未有的大规模数据驱动决策机遇。过去二十年间，爱彼迎（Airbnb）、阿里巴巴、亚马逊（Amazon）、百度、缤客（Booking）、谷歌旗下的Alphabet、领英（LinkedIn）、来福车（Lyft）、Meta旗下的Facebook、微软（Microsoft）、网飞（Netflix）、推特（Twitter）、优步（Uber）及Yandex等机构投入了巨大资源开展在线控制实验（OCE），以评估创新对其客户及业务的影响。大规模实施OCE面临着诸多需跨领域解决方案的挑战。本文综述了需要新统计方法论应对的挑战，重点探讨了在线实验的实践文化及其统计学文献，将现有方法论置于相关统计学脉络中，并提供了OCE应用的示例说明。本研究旨在提升学术统计学家对这些新研究机遇的认知，从而促进学术界与在线产业界的合作。