Online Controlled Experiments (OCEs) are the gold standard in evaluating the effectiveness of changes to websites. An important type of OCE evaluates different personalization strategies, which present challenges in low test power and lack of full control in group assignment. We argue that getting the right experiment setup -- the allocation of users to treatment/analysis groups -- should take precedence of post-hoc variance reduction techniques in order to enable the scaling of the number of experiments. We present an evaluation framework that, along with a few simple rule of thumbs, allow experimenters to quickly compare which experiment setup will lead to the highest probability of detecting a treatment effect under their particular circumstance.

翻译：在线受控实验（OCEs）是评估网站变更效果的黄金标准。一类重要的OCE评估不同的个性化策略，这些策略面临测试功效低和组别分配缺乏完全控制等挑战。我们认为，正确配置实验方案——即用户分配到处理组/分析组的分配方式——应优先于事后方差缩减技术，以便支持实验数量的规模化扩展。我们提出一个评估框架，结合若干简单经验法则，使实验人员能够快速比较在特定情境下哪种实验方案能带来检测处理效应的最高概率。