Motivated by a number of real-world applications from domains like healthcare and sustainable transportation, in this paper we study a scenario of repeated principal-agent games within a multi-armed bandit (MAB) framework, where: the principal gives a different incentive for each bandit arm, the agent picks a bandit arm to maximize its own expected reward plus incentive, and the principal observes which arm is chosen and receives a reward (different than that of the agent) for the chosen arm. Designing policies for the principal is challenging because the principal cannot directly observe the reward that the agent receives for their chosen actions, and so the principal cannot directly learn the expected reward using existing estimation techniques. As a result, the problem of designing policies for this scenario, as well as similar ones, remains mostly unexplored. In this paper, we construct a policy that achieves a low regret (i.e., square-root regret up to a log factor) in this scenario for the case where the agent has perfect-knowledge about its own expected rewards for each bandit arm. We design our policy by first constructing an estimator for the agent's expected reward for each bandit arm. Since our estimator uses as data the sequence of incentives offered and subsequently chosen arms, the principal's estimation can be regarded as an analogy of online inverse optimization in MAB's. Next we construct a policy that we prove achieves a low regret by deriving finite-sample concentration bounds for our estimator. We conclude with numerical simulations demonstrating the applicability of our policy to real-life setting from collaborative transportation planning.

翻译：受医疗保健和可持续交通等领域的实际应用启发，本文研究多臂老虎机（MAB）框架下的重复性委托-代理博弈场景：委托人针对每个老虎机臂提供不同激励，代理人选择最大化自身期望收益与激励之和的臂，委托人观测到被选中的臂并获取该臂的收益（与代理人收益不同）。由于委托人无法直接观测代理人因选择动作所获收益，现有估计技术无法直接学习期望收益，导致此类场景及类似场景的策略设计问题尚未得到充分探索。本文针对代理人对其各臂期望收益具有完全知识的情形，构建了一种低遗憾（即对数因子下的平方根遗憾）策略。该策略通过首先构建代理人各臂期望收益的估计器实现——该估计器以激励序列及后续被选中的臂作为数据，使得委托人的估计可类比为多臂老虎机中的在线逆优化。随后，我们通过推导估计器的有限样本集中界，证明所构建策略可实现低遗憾。最终通过数值仿真验证了该策略在协同运输规划实际场景中的适用性。