Motivated by a number of real-world applications from domains like healthcare and sustainable transportation, in this paper we study a scenario of repeated principal-agent games within a multi-armed bandit (MAB) framework, where: the principal gives a different incentive for each bandit arm, the agent picks a bandit arm to maximize its own expected reward plus incentive, and the principal observes which arm is chosen and receives a reward (different than that of the agent) for the chosen arm. Designing policies for the principal is challenging because the principal cannot directly observe the reward that the agent receives for their chosen actions, and so the principal cannot directly learn the expected reward using existing estimation techniques. As a result, the problem of designing policies for this scenario, as well as similar ones, remains mostly unexplored. In this paper, we construct a policy that achieves a low regret (i.e., square-root regret up to a log factor) in this scenario for the case where the agent has perfect-knowledge about its own expected rewards for each bandit arm. We design our policy by first constructing an estimator for the agent's expected reward for each bandit arm. Since our estimator uses as data the sequence of incentives offered and subsequently chosen arms, the principal's estimation can be regarded as an analogy of online inverse optimization in MAB's. Next we construct a policy that we prove achieves a low regret by deriving finite-sample concentration bounds for our estimator. We conclude with numerical simulations demonstrating the applicability of our policy to real-life setting from collaborative transportation planning.

翻译：受医疗健康和可持续交通等领域的实际应用启发，本文研究了多臂赌博机框架下重复性委托-代理博弈场景。其中：委托人对每个赌博机臂提供差异化激励，代理人为最大化自身期望奖励与激励之和而选择臂，委托人则观察被选中的臂并获取与代理人不同的奖励。为委托人设计策略面临挑战，因为委托人无法直接观测代理人选择动作所获奖励，因而无法利用现有估计技术直接学习期望奖励。这使得此类场景及相似场景下的策略设计问题仍主要处于未探索状态。本文针对代理人对各臂期望奖励具有完美知识的情形，构造了一种低遗憾策略（即对数因子下的平方根遗憾）。我们首先为代理人各臂期望奖励构建估计器。由于该估计器以激励序列及后续选臂数据作为输入，委托人的估计可视为多臂赌博机框架下在线逆向优化的类比。随后通过推导估计器的有限样本集中界，我们证明了所构造策略可实现低遗憾。最后通过数值仿真验证了该策略在协同运输规划实际场景中的适用性。