We study the nonstationary stochastic Multi-Armed Bandit (MAB) problem in which the distribution of rewards associated with each arm are assumed to be time-varying and the total variation in the expected rewards is subject to a variation budget. The regret of a policy is defined by the difference in the expected cumulative rewards obtained using the policy and using an oracle that selects the arm with the maximum mean reward at each time. We characterize the performance of the proposed policies in terms of the worst-case regret, which is the supremum of the regret over the set of reward distribution sequences satisfying the variation budget. We extend Upper-Confidence Bound (UCB)-based policies with three different approaches, namely, periodic resetting, sliding observation window and discount factor and show that they are order-optimal with respect to the minimax regret, i.e., the minimum worst-case regret achieved by any policy. We also relax the sub-Gaussian assumption on reward distributions and develop robust versions the proposed polices that can handle heavy-tailed reward distributions and maintain their performance guarantees.

翻译：我们研究的是非静止的多武装盗匪(MAB)问题,在这一问题中,与每只手臂有关的奖赏的分配假定是时间变化性的,预期的奖赏的全面变化取决于不同的预算预算,对政策的遗憾在于利用政策以及利用每次以最高平均奖赏选择手臂的甲骨文来选择手臂的预期累积奖赏的差别。我们从最坏的悔恨的角度来描述拟议政策的执行情况,即对符合变差预算的一套奖赏分配顺序的遗憾的夸夸大。我们以三种不同的方法,即定期重新制定、滑动观察窗口和折扣因素,扩展基于上层戒严的政策,并表明这些政策对于最优于微鼠头悔,即任何政策的最小最坏的悔过。我们还放松了对奖赏分配的子-Gaussian假设,并发展了能够处理重尾奖赏分配和保持其业绩保障的拟议警察的健全版本。