Online auction scenarios, such as bidding searches on advertising platforms, often require bidders to participate repeatedly in auctions for identical or similar items. Most previous studies have only considered the process by which the seller learns the prior-dependent optimal mechanism in a repeated auction. However, in this paper, we define a multiagent reinforcement learning environment in which strategic bidders and the seller learn their strategies simultaneously and design an automatic bidding algorithm that updates the strategy of bidders through online interactions. We propose Bid Net to replace the linear shading function as a representation of the strategic bidders' strategy, which effectively improves the utility of strategy learned by bidders. We apply and revise the opponent modeling methods to design the PG (pseudo-gradient) algorithm, which allows bidders to learn optimal bidding strategies with predictions of the other agents' strategy transition. We prove that when a bidder uses the PG algorithm, it can learn the best response to static opponents. When all bidders adopt the PG algorithm, the system will converge to the equilibrium of the game induced by the auction. In experiments with diverse environmental settings and varying opponent strategies, the PG algorithm maximizes the utility of bidders. We hope that this article will inspire research on automatic bidding strategies for strategic bidders.

翻译：在线拍卖场景，例如广告平台上的竞价搜索，通常要求投标者反复参与相同或类似物品的拍卖。以往研究多关注卖家在重复拍卖中学习先验依赖的最优机制的过程。然而，本文定义了一个多智能体强化学习环境，其中战略投标者和卖家同时学习各自策略，并设计了一种通过在线交互更新投标者策略的自动出价算法。我们提出Bid Net，用以替代线性遮蔽函数作为战略投标者策略的表示，从而有效提升投标者所学策略的效用。我们应用并改进了对手建模方法，设计了PG（伪梯度）算法，使投标者能够通过预测其他智能体的策略转移来学习最优出价策略。我们证明，当投标者使用PG算法时，能够学习到针对静态对手的最优反应。当所有投标者均采用PG算法时，系统将收敛至拍卖所诱导博弈的均衡。在多种环境设置及不同对手策略的实验中，PG算法最大化了投标者的效用。我们希望本文能激发对战略投标者自动出价策略的研究。