Adaptive experimental design (AED) methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. However, the behavior and guarantees of such methods are not well-understood beyond idealized stationary settings. This paper shares lessons learned regarding the challenges of naively using AED systems in industrial settings where non-stationarity is prevalent, while also providing perspectives on the proper objectives and system specifications in such settings. We developed an AED framework for counterfactual inference based on these experiences, and tested it in a commercial environment.

翻译：适应性实验设计（AED）方法在工业界日益被用作提升测试效率或降低实验成本的工具，相较于传统A/B/N测试方法具有显著优势。然而，在理想化平稳环境之外，这类方法的行为表现和理论保证尚未得到充分理解。本文分享了在非平稳性普遍存在的工业场景中简单套用AED系统所面临的挑战教训，同时提供了此类场景下合理目标设定与系统规范的见解。我们基于这些实践经验开发了用于反事实推断的AED框架，并在商业环境中完成了测试验证。