Stochastic multi-armed bandits are a sequential-decision-making framework, where, at each interaction step, the learner selects an arm and observes a stochastic reward. Within the context of best-arm identification (BAI) problems, the goal of the agent lies in finding the optimal arm, i.e., the one with highest expected reward, as accurately and efficiently as possible. Nevertheless, the sequential interaction protocol of classical BAI problems, where the agent has complete control over the arm being pulled at each round, does not effectively model several decision-making problems of interest (e.g., off-policy learning, partially controllable environments, and human feedback). For this reason, in this work, we propose a novel strict generalization of the classical BAI problem that we refer to as best-arm identification under mediators' feedback (BAI-MF). More specifically, we consider the scenario in which the learner has access to a set of mediators, each of which selects the arms on the agent's behalf according to a stochastic and possibly unknown policy. The mediator, then, communicates back to the agent the pulled arm together with the observed reward. In this setting, the agent's goal lies in sequentially choosing which mediator to query to identify with high probability the optimal arm while minimizing the identification time, i.e., the sample complexity. To this end, we first derive and analyze a statistical lower bound on the sample complexity specific to our general mediator feedback scenario. Then, we propose a sequential decision-making strategy for discovering the best arm under the assumption that the mediators' policies are known to the learner. As our theory verifies, this algorithm matches the lower bound both almost surely and in expectation. Finally, we extend these results to cases where the mediators' policies are unknown to the learner obtaining comparable results.

翻译：随机多臂赌博机是一种序贯决策框架，在每个交互步骤中，学习器选择一个臂并观察随机奖励。在最优臂识别（BAI）问题的背景下，智能体的目标在于尽可能准确高效地找到最优臂（即具有最高期望奖励的臂）。然而，经典BAI问题中智能体对每轮拉动的臂拥有完全控制权的序贯交互协议，并不能有效建模多个重要的决策问题（例如离策略学习、部分可控环境和人类反馈）。为此，本文提出经典BAI问题的严格泛化新形式，称为"中介者反馈下的最优臂识别"（BAI-MF）。具体而言，我们考虑如下场景：学习器可访问一组中介者，每个中介者根据某种随机且可能未知的策略代表智能体选择臂，随后将所拉动的臂与观测到的奖励反馈给智能体。在此设定下，智能体的目标在于序贯选择待查询的中介者，以高概率识别最优臂，同时最小化识别时间（即样本复杂度）。为此，我们首先推导并分析了适用于通用中介反馈场景的样本复杂度统计下界，随后提出一种在中介者策略已知于学习器的假设下发现最优臂的序贯决策策略。理论验证表明，该算法在几乎必然意义和期望意义上均能达到下界。最后，我们将这些结果推广至中介者策略未知于学习器的场景，并获得了可比较的结果。