Digital experimentation and measurement (DEM) capabilities -- the knowledge and tools necessary to run experiments with digital products, services, or experiences and measure their impact -- are fast becoming part of the standard toolkit of digital/data-driven organisations in guiding business decisions. Many large technology companies report having mature DEM capabilities, and several businesses have been established purely to manage experiments for others. Given the growing evidence that data-driven organisations tend to outperform their non-data-driven counterparts, there has never been a greater need for organisations to build/acquire DEM capabilities to thrive in the current digital era. This thesis presents several novel approaches to statistical and data challenges for organisations building DEM capabilities. We focus on the fundamentals associated with building DEM capabilities, which lead to a richer understanding of the underlying assumptions and thus enable us to develop more appropriate capabilities. We address why one should engage in DEM by quantifying the benefits and risks of acquiring DEM capabilities. This is done using a ranking under lower uncertainty model, enabling one to construct a business case. We also examine what ingredients are necessary to run digital experiments. In addition to clarifying the existing literature around statistical tests, datasets, and methods in experimental design and causal inference, we construct an additional dataset and detailed case studies on applying state-of-the-art methods. Finally, we investigate when a digital experiment design would outperform another, leading to an evaluation framework that compares competing designs' data efficiency.

翻译：数字实验与测量（DEM）能力——即运行数字产品、服务或体验实验并衡量其影响所需的知识与工具——正迅速成为数字/数据驱动型组织在指导商业决策时的标准工具集。多家大型科技企业已具备成熟的DEM能力，更有企业纯粹为管理他人实验而成立。鉴于越来越多的证据表明数据驱动型组织往往优于非数据驱动型同行，当前数字时代组织要实现蓬勃发展，对构建/获取DEM能力的需求前所未有地迫切。本论文针对组织构建DEM能力时面临的统计与数据挑战提出了若干创新方法。我们聚焦于构建DEM能力相关的基础问题，这有助于深入理解底层假设，从而开发更适配的能力体系。通过量化获取DEM能力的收益与风险，我们论证了为何应当参与DEM——采用低不确定性排序模型即可构建商业案例论证。我们还考察了运行数字实验所需的基本要素。在厘清现有文献关于统计检验、数据集、实验设计与因果推断方法的基础上，我们构建了额外数据集，并提供了应用前沿方法的详细案例研究。最终，我们探究了数字实验设计何时能优于其他方案，由此建立了比较竞争性设计方案数据效率的评估框架。