The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this paper we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry.

翻译：20世纪90年代末互联网服务与产品的兴起，为在线企业开展大规模数据驱动决策提供了前所未有的机遇。过去二十年间，Airbnb、阿里巴巴、亚马逊、百度、Booking、谷歌、领英、Lyft、Meta旗下Facebook、微软、Netflix、Twitter、优步及Yandex等机构投入大量资源开展在线受控实验（OCEs），以评估创新对其客户及业务的影响。大规模运行OCEs带来了一系列需要多领域协同解决的挑战。本文聚焦于需要新统计方法应对的挑战，重点论述了在线实验的实践与文化及其统计文献体系，将现有方法纳入相关统计脉络，并提供OCEs应用的典型案例。本研究旨在提升学术界统计学者对这些新兴研究机遇的认知，从而促进学术界与在线产业界的深度合作。