Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers' decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to "peek" at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-arm bandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.

翻译：随机实验已成为企业评估新产品或服务性能的标准方法。除了增强管理决策，实验通过限制接受创新客户的比例来降低风险。由于许多实验针对的是依次到达的客户，一个潜在的解决方案是允许管理者在获得新数据时“偷看”结果，并在结果具有统计显著性时停止测试。不幸的是，偷看会使标准统计分析的统计保证失效，并导致无法控制的一类错误。本文提供了有效的基于设计的置信序列，即一系列置信区间，在弱假设条件下对多种序贯实验具有时间上统一的一类错误保证。具体而言，我们关注定义在研究参与者上的有限样本估计量，作为企业所承受风险的直接度量。我们提出的置信序列对大量实验类别均有效，包括多臂老虎机、时间序列和面板实验。我们进一步提供了一种结合建模假设和协变量的方差缩减技术。最后，我们通过模拟研究和Netflix的三个实际应用展示了所提出方法的有效性。结果表明，利用我们的置信序列，有害实验在仅观测到少量单元后即可终止；例如，Netflix在其注册页面上对3万名潜在客户开展的一项实验，本可通过我们的方法在第一天、观测数未达100个时就予以终止。