The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this paper we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry.

翻译：互联网服务和产品在1990年代后期的崛起给在线企业带来了前所未有的机会,让它们参与大规模的数据驱动决策。在过去二十年里,Airbnb、Alibaba、Amazon、Baidu、Booking、Alphabet's Google、LinkedIn、Lyft、Meta的Facebook、微软、Netflix、Twitter、Uber和Yandex等组织为在线控制实验(OCEs)投资了巨大的资源,以评估创新对其客户和企业的影响。大规模运行OCE提出了许多挑战,需要许多领域的解决方案。在本文件中,我们审查了需要新的统计方法来应对的挑战。特别是,我们讨论了在线实验的做法和文化及其统计文献,将现行方法置于相关的统计线外,并提供OCE应用的示例。我们的目标是提高学术统计人员对这些新研究机会的认识,以加强学术界和在线产业之间的合作。