The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this paper we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry.

翻译：20世纪90年代末互联网服务和产品的兴起，为在线企业开展大规模数据驱动决策提供了前所未有的机遇。过去二十年间，Airbnb、阿里巴巴、亚马逊、百度、Booking、谷歌（Alphabet旗下）、领英（LinkedIn）、Lyft、Meta旗下Facebook、微软、Netflix、推特（Twitter）、优步（Uber）及Yandex等组织投入大量资源开展在线受控实验（OCE），以评估创新对其客户和业务的影响。大规模实施OCE引发了诸多需要多领域解决方案的挑战。本文重点探讨需要新统计方法论应对的挑战，具体讨论在线实验的实践规范与组织文化及其统计学文献，将现有方法论纳入相关统计体系脉络，并提供OCE应用的示范案例。旨在提升学术统计学家对这些新兴研究机遇的认知，促进学术界与在线产业界的协同合作。