We study reserve price optimization in multi-phase second price auctions, where the seller's prior actions affect the bidders' later valuations through a Markov Decision Process (MDP). Compared to the bandit setting in existing works, the setting in ours involves three challenges. First, from the seller's perspective, we need to efficiently explore the environment in the presence of potentially untruthful bidders who aim to manipulate the seller's policy. Second, we want to minimize the seller's revenue regret when the market noise distribution is unknown. Third, the seller's per-step revenue is an unknown, nonlinear random variable, and cannot even be directly observed from the environment but realized values. We propose a mechanism addressing all three challenges. To address the first challenge, we use a combination of a new technique named "buffer periods" and inspirations from Reinforcement Learning (RL) with low switching cost to limit bidders' surplus from untruthful bidding, thereby incentivizing approximately truthful bidding. The second one is tackled by a novel algorithm that removes the need for pure exploration when the market noise distribution is unknown. The third challenge is resolved by an extension of LSVI-UCB, where we use the auction's underlying structure to control the uncertainty of the revenue function. The three techniques culminate in the Contextual-LSVI-UCB-Buffer (CLUB) algorithm which achieves $\tilde{O}(H^{5/2}\sqrt{K})$ revenue regret, where $K$ is the number of episodes and $H$ is the length of each episode, when the market noise is known and $\tilde{O}(H^{3}\sqrt{K})$ revenue regret when the noise is unknown with no assumptions on bidders' truthfulness.

翻译：我们研究了多阶段次价拍卖中的保留价优化问题，其中卖家的前期行动通过马尔可夫决策过程（MDP）影响竞拍者后期的估值。与现有研究中的赌博机设置相比，我们的设置面临三个挑战。首先，从卖家的视角出发，我们需要在存在可能不诚实的竞拍者（其目标是操纵卖家策略）的情况下高效探索环境。其次，我们希望在市场噪声分布未知时最小化卖家的收益遗憾。第三，卖家的单步收益是一个未知的非线性随机变量，甚至无法直接从环境中观测，而只能通过实现值获得。我们提出了一种同时解决这三个挑战的机制。针对第一个挑战，我们结合使用名为"缓冲期"的新技术和来自低切换成本强化学习（RL）的启发，以限制不诚实竞价带来的竞拍者剩余，从而激励近似诚实的竞价行为。第二个挑战通过一种新颖算法解决，该算法在市场噪声分布未知时消除了纯探索的需求。第三个挑战通过LSVI-UCB的扩展方案解决，我们利用拍卖的底层结构来控制收益函数的不确定性。这三种技术最终形成了Contextual-LSVI-UCB-Buffer（CLUB）算法，该算法在市场噪声已知时实现$\tilde{O}(H^{5/2}\sqrt{K})$的收益遗憾，在噪声未知且不对竞拍者诚实性做任何假设时实现$\tilde{O}(H^{3}\sqrt{K})$的收益遗憾，其中$K$为回合数，$H$为每回合长度。