Adaptive experimental design (AED) methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. However, the behavior and guarantees of such methods are not well-understood beyond idealized stationary settings. This paper shares lessons learned regarding the challenges of naively using AED systems in industrial settings where non-stationarity is prevalent, while also providing perspectives on the proper objectives and system specifications in such settings. We developed an AED framework for counterfactual inference based on these experiences, and tested it in a commercial environment.

翻译：自适应实验设计（AED）方法在工业界日益被用作提升测试效率或降低实验成本的工具，相比传统的A/B/N测试方法更具优势。然而，在理想化的平稳环境之外，这类方法的行为特性和理论保障尚未得到充分理解。本文分享了在非平稳性普遍存在的工业环境中，简单套用AED系统所遇到的挑战与经验教训，同时针对此类场景下合理的目标设定与系统规范提供了见解。基于这些实践经验，我们开发了用于反事实推断的AED框架，并在商业环境中进行了测试。