In online advertising, advertisers participate in ad auctions to acquire ad opportunities, often by utilizing auto-bidding tools provided by demand-side platforms (DSPs). The current auto-bidding algorithms typically employ reinforcement learning (RL). However, due to safety concerns, most RL-based auto-bidding policies are trained in simulation, leading to a performance degradation when deployed in online environments. To narrow this gap, we can deploy multiple auto-bidding agents in parallel to collect a large interaction dataset. Offline RL algorithms can then be utilized to train a new policy. The trained policy can subsequently be deployed for further data collection, resulting in an iterative training framework, which we refer to as iterative offline RL. In this work, we identify the performance bottleneck of this iterative offline RL framework, which originates from the ineffective exploration and exploitation caused by the inherent conservatism of offline RL algorithms. To overcome this bottleneck, we propose Trajectory-wise Exploration and Exploitation (TEE), which introduces a novel data collecting and data utilization method for iterative offline RL from a trajectory perspective. Furthermore, to ensure the safety of online exploration while preserving the dataset quality for TEE, we propose Safe Exploration by Adaptive Action Selection (SEAS). Both offline experiments and real-world experiments on Alibaba display advertising platform demonstrate the effectiveness of our proposed method.

翻译：在在线广告中，广告主通过需求方平台（DSP）提供的自动出价工具参与广告拍卖以获取广告机会。当前自动出价算法通常采用强化学习（RL）方法。然而出于安全考虑，大多数基于RL的自动出价策略在仿真环境中训练，导致在线部署时性能下降。为缩小这一差距，我们可并行部署多个自动出价代理以收集大规模交互数据集，进而利用离线RL算法训练新策略。训练后的策略可重新部署用于后续数据收集，形成迭代训练框架（称为迭代离线RL）。本文揭示了该迭代离线RL框架的性能瓶颈源于离线RL算法固有的保守性导致的无效探索与利用。为突破此瓶颈，我们提出轨迹式探索与利用（TEE），该方法从轨迹视角为迭代离线RL引入新颖的数据收集与利用机制。进一步地，为保障在线探索的安全性并维护TEE所需的数据集质量，我们提出自适应动作选择的安全探索（SEAS）。在阿里巴巴展示广告平台上的离线实验与真实场景实验均验证了所提方法的有效性。