We introduce AdCraft, a novel benchmark environment for the Reinforcement Learning (RL) community distinguished by its stochastic and non-stationary properties. The environment simulates bidding and budgeting dynamics within Search Engine Marketing (SEM), a digital marketing technique utilizing paid advertising to enhance the visibility of websites on search engine results pages (SERPs). The performance of SEM advertisement campaigns depends on several factors, including keyword selection, ad design, bid management, budget adjustments, and performance monitoring. Deep RL recently emerged as a potential strategy to optimize campaign profitability within the complex and dynamic landscape of SEM but it requires substantial data, which may be costly or infeasible to acquire in practice. Our customizable environment enables practitioners to assess and enhance the robustness of RL algorithms pertinent to SEM bid and budget management without such costs. Through a series of experiments within the environment, we demonstrate the challenges imposed by sparsity and non-stationarity on agent convergence and performance. We hope these challenges further encourage discourse and development around effective strategies for managing real-world uncertainties.

翻译：我们提出了AdCraft，一个面向强化学习（RL）社区的新型基准环境，其显著特征在于随机性和非平稳性。该环境模拟了搜索引擎营销（SEM）中的竞价与预算调整动态过程——SEM是一种利用付费广告提升网站在搜索引擎结果页（SERPs）可见性的数字营销技术。SEM广告活动的绩效取决于关键词选择、广告设计、竞价管理、预算调整及效果监测等多个因素。深度强化学习近年来被视为在复杂动态的SEM环境中优化广告活动盈利能力的潜在策略，但其需要大量数据，而实践中获取这些数据可能成本高昂或难以实现。我们提出的可定制环境使从业者能够评估和增强与SEM竞价及预算管理相关的RL算法鲁棒性，且无需承担此类成本。通过在环境中的系列实验，我们证明了稀疏性与非平稳性对智能体收敛性和性能带来的挑战。我们期待这些挑战能进一步推动关于管理现实世界不确定性的有效策略的讨论与发展。